WO2024228231A1 - 制御システム、ロボットシステム、制御方法及びコンピュータプログラム - Google Patents

制御システム、ロボットシステム、制御方法及びコンピュータプログラム Download PDF

Info

Publication number
WO2024228231A1
WO2024228231A1 PCT/JP2023/017026 JP2023017026W WO2024228231A1 WO 2024228231 A1 WO2024228231 A1 WO 2024228231A1 JP 2023017026 W JP2023017026 W JP 2023017026W WO 2024228231 A1 WO2024228231 A1 WO 2024228231A1
Authority
WO
WIPO (PCT)
Prior art keywords
target object
processing device
movement process
robot
processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2023/017026
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
晃平 三村
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nikon Corp
Original Assignee
Nikon Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nikon Corp filed Critical Nikon Corp
Priority to CN202380100006.3A priority Critical patent/CN121464022A/zh
Priority to KR1020257039892A priority patent/KR20260005965A/ko
Priority to PCT/JP2023/017026 priority patent/WO2024228231A1/ja
Priority to JP2025518059A priority patent/JPWO2024228231A1/ja
Priority to EP23935746.0A priority patent/EP4706910A1/en
Publication of WO2024228231A1 publication Critical patent/WO2024228231A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
    • B25J19/02—Sensing devices
    • B25J19/021—Optical sensing devices
    • B25J19/023—Optical sensing devices including video camera means
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1602—Program controls characterised by the control system, structure, architecture
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1656—Program controls characterised by programming, planning systems for manipulators
    • B25J9/1664—Program controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1694—Program controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras

Definitions

  • the present invention relates to the technical fields of, for example, a control system capable of generating a control signal for controlling a robot, a robot system, a control method, and a computer program.
  • Patent Document 1 An example of a control device that controls a robot to which a processing device such as an end effector is attached is described in Patent Document 1. Such a control device is required to control the robot so as to move the processing device efficiently.
  • a control system for generating a control signal for controlling a robot, the robot being equipped with a processing device for performing processing on a target object and an imaging device for generating at least image data of the target object by capturing an image of the target object, the robot moving the processing device and the imaging device, the control system including a control device for generating and outputting the control signal, the control device performing a second movement process after a first movement process, the first movement process including a process for controlling the robot to move the processing device without using the image data, and the second movement process including a process for generating and outputting the control signal for controlling the robot to move the processing device based on a calculation result of the position and orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object after the first movement process.
  • a control system that generates a control signal for controlling a robot, the robot being equipped with a processing device that processes a target object and an imaging device that captures an image of the target object to generate at least image data of the target object, the robot moves the processing device and the imaging device, the control system includes a second control device that performs a first movement process to control the robot to move the processing device in accordance with movement instruction information using a first control device, and then generates and outputs the control signal for controlling the robot to move the processing device based on the calculation results of the position and orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object, the second control device being a control device different from the first control device.
  • a robot system includes the control system provided in the first or second aspect and the robot.
  • a control method for generating a control signal for controlling a robot the robot being equipped with a processing device for performing processing on a target object and an imaging device for capturing an image of the target object and outputting at least image data of the target object, the robot moving the processing device and the imaging device, the control method including performing a second movement process after a first movement process, the first movement process including a process of controlling the robot to move the processing device without using the image data, and performing the second movement process including generating and outputting the control signal for controlling the robot to move the processing device based on a calculation result of the position and orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object after the first movement process.
  • a control method for generating a control signal for controlling a robot the robot being equipped with a processing device for performing processing on a target object and an imaging device for capturing an image of the target object and outputting at least image data of the target object, the robot moving the processing device and the imaging device, and the control method including performing a first movement process by a first control device to control the robot to move the processing device in accordance with movement instruction information, and then performing a second movement process by using a second control device different from the first control device to generate and output the control signal for controlling the robot to move the processing device based on the calculation results of the position and orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object.
  • a computer program is provided that causes a computer to execute the control method provided by the fourth or fifth aspect.
  • FIG. 1 is a block diagram showing the configuration of a robot system according to the present embodiment.
  • FIG. 2 is a side view showing the appearance of the robot of this embodiment.
  • Each of Figs. 3(a) to 3(d) is a cross-sectional view showing a first example of the operation of the robot controlled by the robot control process including the rough movement process and the fine movement process.
  • Each of FIGS. 4(a) to 4(d) is a cross-sectional view showing a second example of the operation of the robot controlled by the robot control process including the rough movement process and the fine movement process.
  • Each of Figs. 5(a) to 5(d) is a cross-sectional view showing a third example of the operation of the robot controlled by the robot control process including the rough movement process and the fine movement process.
  • FIGS. 6(a) to 6(c) are cross-sectional view showing a third example of the operation of the robot controlled by the robot control process including the rough movement process and the fine movement process.
  • FIG. 7 is a block diagram showing the configuration of the control device of this embodiment.
  • FIG. 8 is a block diagram showing the configuration of a robot control device according to this embodiment.
  • FIG. 9 is a flowchart showing the flow of the robot control process.
  • FIG. 10 is a flowchart showing the flow of the fine movement process.
  • FIG. 11 shows a schematic diagram of the matching process.
  • FIG. 12 is a flowchart showing the flow of the robot control process in the first modified example.
  • FIG. 13 shows a first example of a robot control process in which rough movement processes and fine movement processes are alternately repeated.
  • FIG. 14 shows a second example of a robot control process in which rough movement processes and fine movement processes are alternately repeated.
  • FIG. 15 shows the edge model data.
  • FIG. 16 shows model data indicating an object model and an end effector model.
  • FIG. 17 shows an image that includes multiple target objects detected by the matching process, together with model data showing object models and an end effector model that are aligned with each other.
  • FIG. 18 is a flowchart showing the flow of the robot control process in the fifth modified example.
  • FIG. 19 is a flowchart showing the flow of the robot control process in the sixth modified example.
  • FIG. 20 is a flowchart showing the flow of the robot control process in the seventh modified example.
  • FIG. 21 is a side view showing the appearance of the robot in the seventh modified example.
  • FIG. 22 is a flowchart showing the flow of fine movement processing in the eighth modified example.
  • Fig. 1 is a block diagram showing the overall configuration of the robot system SYS.
  • the robot system SYS includes a robot 1, an imaging unit 2, a control device 3, a robot control device 4, and an end effector 5.
  • the robot 1 is a device capable of performing a predetermined process on a target object OBJ.
  • An example of the robot 1 is shown in FIG. 2.
  • FIG. 2 is a side view showing the external appearance of the robot 1.
  • the robot 1 includes, for example, a base 11 and a robot arm 12.
  • the robot 1 may also include the robot control device 4 described above.
  • a device including the base 11, the robot arm 12, and the robot control device 4 may be referred to as the robot 1.
  • the base 11 is a component that serves as the foundation of the robot 1.
  • the base 11 is placed on a support surface S, such as a floor surface.
  • the base 11 may be fixed to the support surface S.
  • the base 11 may be movable relative to the support surface S.
  • the base 11 may be capable of self-propelling on the support surface S.
  • the base 11 may be installed on an automatic guided vehicle (AGV).
  • AGV automatic guided vehicle
  • FIG. 2 shows an example in which the base 11 is fixed to the support surface S.
  • the robot arm 12 is attached to the base 11.
  • the robot arm 12 is a device in which multiple links 121 are connected via joints 122.
  • An actuator is built into the joint 122.
  • the link 121 may be rotatable around an axis defined by the joint 122 by the actuator built into the joint 122.
  • At least one link 121 may be expandable and contractible along the direction in which the link 121 extends.
  • the device including the device in which multiple links 121 are connected via joints 122 and the base 11 may be referred to as the robot arm 12.
  • the end effector 5 is attached to the robot arm 12. That is, the end effector 5 is attached to the robot 1. In the example shown in FIG. 2, the end effector 5 is attached to the tip of the robot arm 12.
  • the end effector 5 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the end effector 5. That is, the robot 1 moves the end effector 5.
  • the end effector 5 is a device that performs a predetermined process (in other words, a predetermined operation) on the target object OBJ.
  • the end effector 5 that performs a predetermined process on the target object OBJ may also be called a processing device.
  • the end effector 5 may perform a holding process to hold the target object OBJ as an example of the predetermined process.
  • the end effector 5 may be considered to be performing a holding process on the target object OBJ that the end effector 5 is to hold.
  • An end effector 5 capable of performing a holding process may be referred to as a holding device.
  • holding the target object OBJ may include, as an example, gripping the target object OBJ.
  • holding the target object OBJ may include gripping the target object OBJ using a hand gripper, which is an example of the end effector 5 and will be described later.
  • Holding the target object OBJ may include adsorbing the target object OBJ.
  • holding the target object OBJ may include adsorbing (vacuum adsorption) the target object OBJ using a vacuum gripper, which is an example of the end effector 5 and will be described later.
  • holding the target object OBJ may include adsorbing the target object OBJ using a magnetic adsorption gripper, which is an example of the end effector 5 and will be described later.
  • the end effector 5 may perform a release process (in other words, a release operation) to release (i.e., let go of) the target object OBJ that it is holding.
  • the end effector 5 may be considered to be performing a release process on the target object OBJ that it is holding.
  • An end effector 5 capable of performing a release process may be referred to as a release device.
  • an end effector 5 capable of holding and releasing is a hand clipper.
  • a hand clipper is an end effector 5 that can hold a target object OBJ by physically pinching the target object OBJ using multiple (e.g., two, three, or four) finger members or claw members.
  • Another example of an end effector 5 capable of holding and releasing is a vacuum clipper.
  • a vacuum clipper is an end effector 5 that can hold a target object OBJ by vacuum-adsorbing the target object OBJ.
  • Figure 2 shows an example of an end effector 5 that is a hand clipper.
  • the robot 1 may use the end effector 5 capable of holding and releasing to perform a placement process (in other words, a placement operation) for placing the target object OBJ at a desired position, as an example of a predetermined process.
  • the robot 1 may hold a first target object OBJ using the end effector 5, and then perform a placement process for placing the first target object OBJ held by the end effector 5 at a desired position of a second target object OBJ different from the first target object OBJ.
  • the end effector 5 may be considered to be performing a release process on the second target object OBJ on which the end effector 5 should place the first target object OBJ.
  • the end effector 5 may be considered to be performing a release process on the first target object OBJ that the end effector 5 should release.
  • the robot 1 may use the end effector 5 capable of holding and releasing to perform a fitting process (in other words, a fitting operation) for fitting the first target object OBJ into a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation).
  • the robot 1 may hold the first target object OBJ using the end effector 5, and then perform a fitting process for fitting the first target object OBJ held by the end effector 5 into a second target object OBJ different from the first target object OBJ.
  • the end effector 5 may be considered to be performing a release process on the second target object OBJ into which the end effector 5 should fit the first target object OBJ.
  • the end effector 5 may be considered to be performing a release process on the first target object OBJ that the end effector 5 should release.
  • the fitting process may include a process for fitting (i.e., inserting) the first target object OBJ into a hole formed in the second target object OBJ.
  • the robot 1 may hold the first target object OBJ using the end effector 5, and then perform a fitting process for fitting (i.e., inserting) the first target object OBJ held by the end effector 5 into a hole formed in the second target object OBJ.
  • the end effector 5 may be considered to be performing a release process on the second target object OBJ in which a hole is formed into which the end effector 5 should fit (i.e., insert) the first target object OBJ.
  • the end effector 5 may be considered to be performing a release process on the first target object OBJ that the end effector 5 should release.
  • the robot 1 may use the end effector 5 capable of holding and releasing to perform a pasting process (in other words, a pasting operation) for pasting a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation).
  • a pasting process in other words, a pasting operation
  • the robot 1 may hold the first target object OBJ using the end effector 5, and then perform a pasting process for pasting the first target object OBJ held by the end effector 5 to a second target object OBJ different from the first target object OBJ.
  • the end effector 5 may be considered to be performing a release process on the second target object OBJ to which the end effector 5 should paste the first target object OBJ.
  • the end effector 5 may be considered to be performing a release process on the first target object OBJ to which the end effector 5 should release.
  • the robot 1 may use the end effector 5 capable of holding and releasing to perform an adhesion process (in other words, an adhesion operation) for adhering a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation).
  • an adhesion process in other words, an adhesion operation
  • the robot 1 may hold the first target object OBJ using the end effector 5, and then perform an adhesion process for adhering the first target object OBJ held by the end effector 5 to a second target object OBJ different from the first target object OBJ.
  • the end effector 5 may be considered to be performing a release process on the second target object OBJ to which the end effector 5 should adhere the first target object OBJ.
  • the end effector 5 may be considered to be performing a release process on the first target object OBJ to which the end effector 5 should release.
  • the robot 1 may use the end effector 5 capable of holding and releasing to perform a welding process (in other words, a welding operation) for welding a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation).
  • a welding process in other words, a welding operation
  • the robot 1 may use the end effector 5 to hold the first target object OBJ, and then perform a welding process for welding the first target object OBJ held by the end effector 5 to a second target object OBJ different from the first target object OBJ.
  • the end effector 5 may be considered to be performing a release process on the second target object OBJ to which the end effector 5 is to weld the first target object OBJ.
  • the end effector 5 may be considered to be performing a release process on the first target object OBJ to which the end effector 5 is to release.
  • the robot 1 may use the end effector 5 capable of holding and releasing to perform a screwing process (in other words, a screwing operation) for screwing a first target object OBJ capable of functioning as a screw into a screw hole formed in a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation).
  • a screwing process in other words, a screwing operation
  • the robot 1 may hold the first target object OBJ using the end effector 5, and then perform a screwing process for screwing the first target object OBJ held by the end effector 5 into a second target object OBJ different from the first target object OBJ.
  • the end effector 5 may be considered to be performing a release process on the second target object OBJ to which the end effector 5 is to screw the first target object OBJ. Similarly, the end effector 5 may be considered to be performing a release process on the first target object OBJ to be released by the end effector 5.
  • the end effector 5 may perform a predetermined process on each of the multiple target objects OBJ. That is, the end effector 5 may perform a predetermined process on the multiple target objects OBJ in sequence.
  • the robot 1 may move the end effector 5 to a first position where the end effector 5 can perform a predetermined process on the first target object OBJ, and after the end effector 5 has moved to the first position, the end effector 5 may perform a predetermined process on the first target object OBJ.
  • the robot 1 may move the end effector 5 to a second position where the end effector 5 can perform a predetermined process on a second target object OBJ different from the first target object OBJ, and after the end effector 5 has moved to the second position, the end effector 5 may perform a predetermined process on the second target object OBJ.
  • the end effector 5 may perform a predetermined process on each of the multiple parts of a single target object OBJ. That is, the end effector 5 may perform a predetermined process on the multiple parts of a single target object OBJ in sequence.
  • the robot 1 may move the end effector 5 to a third position where the end effector 5 can perform a predetermined process on the first part of the target object OBJ, and after the end effector 5 has moved to the third position, the end effector 5 may perform a predetermined process on the first part of the target object OBJ.
  • the robot 1 may move the end effector 5 to a fourth position where the end effector 5 can perform a predetermined process on the second part of the target object OBJ, and after the end effector 5 has moved to the fourth position, the end effector 5 may perform a predetermined process on the second part of the target object OBJ.
  • the robot 1 may repeat a fitting process of fitting the first target object OBJ into each hole of a second target object OBJ in which multiple holes are formed.
  • a portion of the target object OBJ on which the end effector 5 performs a predetermined process may be referred to as a target portion or target member.
  • each portion (each member) of the second target object OBJ in which each hole is formed may be referred to as a target portion or target member.
  • each part when the target object OBJ is composed of multiple parts, each part may be referred to as a target portion or target member.
  • each member may be considered as a target portion or target member.
  • the target object OBJ on which the end effector 5 performs a predetermined process may include a workpiece W, as shown in FIG. 2.
  • the workpiece W may include, for example, parts or members used to manufacture a desired product.
  • the workpiece W may include, for example, parts or members to be processed to manufacture a desired product.
  • the workpiece W may include, for example, parts or members to be transported to manufacture a desired product.
  • the workpiece W may include, for example, parts or members that are moving due to transportation to manufacture a desired product.
  • the workpiece W may include, for example, parts or members that are moving to manufacture a desired product.
  • the target object OBJ on which the end effector 5 performs a predetermined process may include a placement device T on which the workpiece W is placed, as shown in FIG. 2.
  • An example of the placement device T is a pallet.
  • the placement device T may be disposed on the support surface S.
  • the placement device T may be fixed to the support surface S.
  • at least a part of the placement device T may be movable relative to the support surface S.
  • the placement device T may be capable of self-propelling on the support surface S.
  • a self-propelled placement device T may be called an automatic guided vehicle (AGV).
  • the placement device T may be a belt conveyor.
  • a belt that is part of the belt conveyor and on which the workpiece W is placed may move relative to the support surface S.
  • the placement device T may be capable of flying above the support surface S. That is, the placement device T may be movable relative to the support surface S by flying above the support surface S.
  • the placement device T may be referred to as an unmanned aerial vehicle.
  • the placement device T may be another robot arm that is different from the robot arm 12 and has an end effector attached thereto that can hold the workpiece W. In this case, the end effector attached to the other robot arm may move relative to the support surface S.
  • FIG. 2 shows an example in which the placement device T is capable of self-propelling on the support surface S.
  • the placement device T When at least the placement device T is movable relative to the support surface S, the workpiece W placed on the placement device T also moves relative to the support surface S as the placement device T moves. For this reason, the placement device T may be capable of functioning as a moving device that moves the workpiece W.
  • the placement device T may be capable of functioning as a transport device that transports the workpiece W.
  • the transport device may be a device that moves the workpiece W. Note that the transport device may be referred to as a moving device.
  • the first placement device T corresponding to the pallet on which the workpiece W is placed may be placed on a second placement device T that is movable relative to the support surface S.
  • the device including the first placement device T and the second placement device T may be referred to as the placement device T.
  • the target object OBJ does not have to include the placement device T.
  • the workpiece W may be placed on the support surface S.
  • the above-mentioned holding process may include a process of holding the workpiece W placed on the placement device T, which is stationary or moving.
  • the above-mentioned holding process may include a process of holding the workpiece W placed on the support surface S.
  • the above-mentioned release process may include a process of releasing the workpiece W held by the end effector 5 in order to place the workpiece W held by the end effector 5 at a desired position on the placement device T, which is stationary or moving.
  • the above-mentioned release process may include a process of releasing the workpiece W held by the end effector 5 in order to place the workpiece W held by the end effector 5 at a desired position on the support surface S.
  • the above-mentioned release process may include a process of releasing the first workpiece W held by the end effector 5 in order to fit the first workpiece W held by the end effector 5 into a second workpiece W placed on the placement device T, which is stationary or moving.
  • the above-mentioned release process may include a process of releasing the first workpiece W held by the end effector 5 in order to fit the first workpiece W held by the end effector 5 into a second workpiece W placed on the support surface S.
  • the above-mentioned release process may include a process of releasing the first workpiece W held by the end effector 5 in order to fit (i.e., insert) the first workpiece W held by the end effector 5 into a hole formed in the second workpiece W placed on a stationary or moving placement device T.
  • the above-mentioned release process may include a process of releasing the first workpiece W held by the end effector 5 in order to fit (i.e., insert) the first workpiece W held by the end effector 5 into a hole formed in the second workpiece W placed on the support surface S.
  • FIG. 2 shows an example in which the robot 1 is a robot arm 12 (i.e., a vertical articulated robot).
  • the robot 1 may be a robot different from a vertical articulated robot.
  • the robot 1 may be a SCARA robot (i.e., a horizontal articulated robot).
  • the robot 1 may be a parallel link robot.
  • the robot 1 may be a dual-arm robot having two robot arms 12.
  • the robot 1 may be a Cartesian coordinate robot.
  • the robot 1 may be a cylindrical coordinate robot.
  • the robot 1 may be referred to as a movable device.
  • the movable device may include at least one of an automatic guided vehicle and an unmanned aerial vehicle in addition to the robot 1.
  • the robot 1 may be installed in at least one of an automatic guided vehicle and an unmanned aerial vehicle.
  • the imaging unit 2 captures an image of the target object OBJ.
  • the imaging unit 2 includes an imaging device 21 and an illumination device 23.
  • the imaging device 21 is a camera capable of capturing an image of the target object OBJ.
  • the imaging device 21 may capture an image of the target object OBJ under the control of the control device 3.
  • the imaging device 21 generates image data IMG by capturing an image of the target object OBJ. That is, the imaging device 21 generates image data IMG that is the result of capturing an image of the target object OBJ.
  • the image data IMG generated by the imaging device 21 is output from the imaging device 21 to the control device 3.
  • the control device 3 acquires the image data IMG acquired by capturing an image of the target object OBJ by the imaging device 21.
  • the imaging device 21 is a monocular camera.
  • the imaging device 21 is capable of capturing an image of the target object OBJ using a monocular camera (in other words, an image sensor).
  • a monocular camera in other words, an image sensor
  • the imaging device 21 is not limited to being a monocular camera.
  • the imaging device 21 may capture an image of the entire target object OBJ. Alternatively, the imaging device 21 may capture an image of a portion of the target object OBJ. In other words, the imaging device 21 may capture an image of a portion of the target object OBJ while not capturing an image of another portion of the target object OBJ.
  • the imaging device 21 may capture an image of a single target object OBJ. That is, a single target object OBJ may appear in the image represented by the image data IMG.
  • the imaging device 21 may capture an image of multiple target objects OBJ. That is, a multiple target objects OBJ may appear in the image represented by the image data IMG.
  • the control device 3 may determine (in other words, select) one of the multiple target objects OBJ captured by the imaging device 21 as the target object OBJ for which the end effector 5 will actually perform a predetermined process.
  • the target object OBJ for which the end effector 5 will actually perform a predetermined process may be referred to as a processing execution object.
  • the multiple target objects OBJ captured by the imaging device 21 may be arranged such that at least two of the multiple target objects OBJ at least partially overlap.
  • the multiple workpieces W i.e., multiple parts
  • the robot 1 may perform bulk picking, picking workpieces W one by one from the multiple workpieces W that are randomly arranged.
  • the imaging unit 2 is attached to the robot arm 12, similar to the end effector 5. That is, the imaging device 21 and the lighting device 23 are attached to the robot arm 12.
  • the imaging device 21 and the lighting device 23 may be attached to the tip of the robot arm 12, similar to the end effector 5.
  • the imaging device 21 and the lighting device 23 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the imaging device 21 and the lighting device 23.
  • the imaging device 21 may capture an image of the target object OBJ during a period in which the imaging device 21 and the target object OBJ are displaced relative to each other.
  • the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may mean a state in which the relative positional relationship between the imaging device 21 and the target object OBJ is changing.
  • the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may mean a state in which the imaging device 21 and the target object OBJ are moving relative to each other.
  • the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the target object OBJ is moving relative to the imaging device 21.
  • the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the imaging device 21 is moving relative to the target object OBJ.
  • the robot system SYS can efficiently perform a predetermined process on the target object OBJ using the end effector 5.
  • the imaging device 21 may capture the target object OBJ during a period in which there is no relative displacement between the imaging device 21 and the target object OBJ.
  • the state in which there is no relative displacement between the imaging device 21 and the target object OBJ may mean a state in which the relative positional relationship between the imaging device 21 and the target object OBJ has not changed.
  • the state in which there is no relative displacement between the imaging device 21 and the target object OBJ may mean a state in which there is no relative movement between the imaging device 21 and the target object OBJ.
  • the state in which there is no relative displacement between the imaging device 21 and the target object OBJ may mean a state in which the imaging device 21 and the target object OBJ are stationary.
  • the state in which there is no relative displacement between the imaging device 21 and the target object OBJ may mean a state in which the imaging device 21 and the target object OBJ are moving in the same direction at the same moving speed.
  • the illumination device 23 is a device capable of irradiating illumination light onto the target object OBJ.
  • the illumination device 23 may irradiate illumination light onto the target object OBJ under the control of the control device 3.
  • the illumination device 23 is a device capable of irradiating the target object OBJ with illumination light, thereby illuminating the target object OBJ with the illumination light.
  • the illumination light may be light having a uniform intensity distribution.
  • the illumination light may be light having a non-uniform intensity distribution.
  • the imaging device 21 may image the target object OBJ illuminated with the illumination light.
  • the illumination device 23 may not irradiate illumination light onto the target object OBJ.
  • the imaging unit 2 (robot system SYS) may not be equipped with the illumination device 23.
  • the control device 3 performs robot control processing.
  • the robot control device 4 also performs robot control processing.
  • the control device 3 performs robot control processing in cooperation with the robot control device 4.
  • the robot control device 4 performs robot control processing in cooperation with the control device 3.
  • the control device 3 and the robot control device 4, which are different control devices perform robot control processing while cooperating with each other.
  • control device 3 and the robot control device 4 may be integrated.
  • a control device capable of functioning as the control device 3 and the robot control device 4 may perform the robot control process.
  • the robot control process may include a process for generating a control signal for controlling the robot 1.
  • the robot control process may include a process for generating a control signal for controlling the operation of the robot 1.
  • the control signal for controlling the robot 1 may be referred to as a robot control signal.
  • the robot control process may include a process for generating a control signal for controlling the end effector 5 attached to the robot 1.
  • the robot control process may include a process for generating a control signal for controlling the operation of the end effector 5 attached to the robot 1.
  • the control signal for controlling the end effector 5 may be referred to as an end effector control signal.
  • the control signal for controlling the robot 1 may include a signal for controlling the robot arm 12.
  • the robot arm 12 moves the end effector 5.
  • the control signal may include a signal for controlling the robot arm 12 so that the end effector 5 is located at a desired position.
  • the control signal may include a signal for controlling the robot arm 12 so that the end effector 5 moves to a desired position.
  • the control signal may include a signal for controlling the robot arm 12 so that the positional relationship between the end effector 5 and the target object OBJ is a desired positional relationship.
  • the control signal may include a signal for controlling the robot arm 12 so that the end effector 5 moves toward (i.e., approaches) a holding position where the end effector 5 can hold the target object OBJ.
  • the control signal may include a signal for controlling the robot arm 12 so that the end effector 5 is positioned at the holding position.
  • the control signal may include a signal for controlling the end effector 5 so that the end effector 5 positioned at the above-mentioned holding position holds the target object OBJ.
  • the control signal may include a signal for controlling the robot arm 12 so that the end effector 5 moves toward (i.e., approaches) a release position where the target object OBJ held by the end effector 5 should be released.
  • the control signal may include a signal for controlling the robot arm 12 so that the end effector 5 is positioned at the release position.
  • the control signal may include a signal for controlling the end effector 5 so that the end effector 5, which is positioned at the release position described above, releases the target object OBJ held by the end effector 5.
  • the control signal may include a signal that can be used as is to control the operation of the robot 1.
  • the control signal may include a drive signal (robot drive signal) that can be input to the robot 1 to control the operation of the robot 1.
  • the control signal may include a drive signal (robot drive signal) that can be input to the robot 1 to control the amount of drive of the robot 1.
  • the control signal may include a drive signal for driving an actuator built into the joint 122 of the robot arm 12.
  • the control signal may include a drive signal for controlling the amount of drive of an actuator built into the joint 122 of the robot arm 12.
  • the control signal may include a signal that can be used as is to control the operation of the end effector 5.
  • the control signal may include a drive signal (end effector drive signal) that can be input to the end effector 5 to control the operation of the end effector 5.
  • the control signal may include a drive signal (end effector drive signal) that can be input to the end effector 5 to control the drive amount of the end effector 5.
  • the control signal may include a drive signal for driving an actuator that moves a hand clipper that constitutes the end effector 5.
  • the control signal may include a drive signal for controlling the drive amount of an actuator that moves a hand clipper that constitutes the end effector 5.
  • control signal may include a drive signal for driving a vacuum device of a vacuum clipper that constitutes the end effector 5.
  • the control signal may include a drive signal for controlling the drive amount of a vacuum device of a vacuum clipper that constitutes the end effector 5.
  • the control signal may include a signal that can be used to generate the above-mentioned drive signal that can be input to the robot 1 to control the operation of the robot 1.
  • the control signal may include a signal that can be used to generate a drive signal for an actuator built into the joint 122 of the robot arm 12.
  • the control signal may include a signal that can be used to generate the above-mentioned drive signal that can be input to the end effector 5 to control the operation of the end effector 5.
  • the control signal may include a signal that can be used to generate a drive signal for an actuator that moves a hand clipper that constitutes the end effector 5.
  • the control signal may include a signal that can be used to generate a drive signal for a vacuum device of a vacuum clipper that constitutes the end effector 5.
  • the control device 3 may generate a control signal by performing robot control processing.
  • the control signal generated by the control device 3 may be input from the control device 3 to the robot control device 4.
  • the control device 3 may output the control signal generated by the control device 3 to the robot control device 4.
  • the robot control device 4 may control the operation of the robot 1 using the control signal generated by the control device 3 as is. Specifically, the robot control device 4 may control the operation of the robot 1 by outputting the control signal generated by the control device 3 to the robot 1 as a drive signal that can be input to the robot 1 to control the operation of the robot 1.
  • the robot control device 4 may use the control signal generated by the control device 3 as is to control the operation of the end effector 5. Specifically, the robot control device 4 may control the operation of the end effector 5 by outputting the control signal generated by the control device 3 to the end effector 5 as a drive signal that can be input to the end effector 5 to control the operation of the end effector 5. Note that outputting the control signal to the end effector 5 may include at least one of directly outputting the control signal to the end effector 5 and indirectly outputting the control signal to the end effector 5 via the robot 1.
  • the robot control device 4 may generate, as a control signal, a drive signal that can be input to the robot 1 to control the operation of the robot 1, based on the control signal generated by the control device 3. After that, the robot control device 4 may control the operation of the robot 1 by outputting the control signal generated by the robot control device 4 to the robot 1.
  • the robot control device 4 may generate, as a control signal, a drive signal that can be input to the end effector 5 to control the operation of the end effector 5, based on the control signal generated by the control device 3. Thereafter, the robot control device 4 may control the operation of the end effector 5 by outputting the control signal generated by the robot control device 4 to the end effector 5.
  • control device 3 when the control device 3 generates a control signal, the control signal generated by the control device 3 may be input from the control device 3 to the robot 1 or the end effector 5 without going through the robot control device 4.
  • the control device 3 when the control device 3 generates a signal that can be used as is to control the operation of the robot 1 (i.e., a drive signal that can be input to the robot 1 to control the operation of the robot 1) as a control signal, the control device 3 may input the control signal generated by the control device 3 to the robot 1 without going through the robot control device 4.
  • control device 3 when the control device 3 generates a signal that can be used as is to control the operation of the end effector 5 (i.e., a drive signal that can be input to the end effector 5 to control the operation of the end effector 5) as a control signal, the control device 3 may input the control signal generated by the control device 3 to the end effector 5 without going through the robot control device 4.
  • a signal that can be used as is to control the operation of the end effector 5 i.e., a drive signal that can be input to the end effector 5 to control the operation of the end effector 5
  • the control device 3 may input the control signal generated by the control device 3 to the end effector 5 without going through the robot control device 4.
  • the robot control device 4 may generate a control signal by performing robot control processing.
  • the robot control device 4 may generate a drive signal that can be input to the robot 1 to control the operation of the robot 1 as a control signal.
  • the robot control device 4 may then control the operation of the robot 1 by outputting the control signal generated by the robot control device 4 to the robot 1.
  • the robot control device 4 may generate a drive signal that can be input to the end effector 5 to control the operation of the end effector 5 as a control signal.
  • the robot control device 4 may then control the operation of the end effector 5 by outputting the control signal generated by the robot control device 4 to the end effector 5.
  • the control device 3 performs fine movement processing as at least a part of the robot control processing.
  • the robot control device 4 performs rough movement processing as at least a part of the robot control processing.
  • the robot control device 4 may perform fine movement processing as at least a part of the robot control processing. In this case, the robot control device 4 may perform both the rough movement processing and the fine movement processing, and the robot system SYS may not include the control device 3.
  • the control device 3 may perform rough movement processing as at least a part of the robot control processing. In this case, the control device 3 may perform both the rough movement processing and the fine movement processing.
  • the robot system SYS may not include the robot control device 4.
  • the control device 3 performs fine movement processing and the robot control device 4 performs rough movement processing.
  • Both the fine movement process and the rough movement process are processes that move the end effector 5. Specifically, both the fine movement process and the rough movement process are processes that control the robot 1 to move the end effector 5. In other words, both the fine movement process and the rough movement process are processes that generate control signals for controlling the robot 1 to move the end effector 5, and output the generated signals.
  • both the fine movement process and the rough movement process may be considered to be processes (movement processes) on the target object OBJ in which the end effector 5 performs a predetermined process.
  • both the fine movement process and the rough movement process may be considered to be processes performed on the target object OBJ in which the end effector 5 performs a predetermined process.
  • both the fine movement process and the rough movement process may be considered to be processes that move the end effector 5 with respect to the target object OBJ.
  • both the fine movement process and the rough movement process may be considered to be processes that move the end effector 5 so that the end effector 5 approaches the target object OBJ.
  • both the fine movement process and the rough movement process may be considered to be processes that control the robot 1 so that the end effector 5 approaches the target object OBJ by moving the end effector 5.
  • the end effector 5 may perform a holding process, which is an example of a predetermined process, on the target object OBJ in order to hold the target object OBJ.
  • both the fine movement process and the rough movement process may be considered to be processes on the target object OBJ that the end effector 5 is to hold.
  • both the fine movement process and the rough movement process may be considered to be processes that move the end effector 5 toward the target object OBJ that the end effector 5 is to hold.
  • both the fine movement process and the rough movement process may be considered to be processes that control the robot 1 by moving the end effector 5 so that the end effector 5 approaches the target object OBJ that the end effector 5 is to hold.
  • the end effector 5 may perform a release process, which is an example of a predetermined process, to release the held first target object OBJ.
  • the end effector 5 may perform a release process, which is an example of a predetermined process, on the second target object OBJ from which the held first target object OBJ is to be released, in order to place the held first target object OBJ at a desired position of the second target object OBJ.
  • the end effector 5 may perform a release process, which is an example of a predetermined process, on the second target object OBJ from which the held first target object OBJ is to be released, in order to fit the held first target object OBJ into the second target object OBJ.
  • the end effector 5 may perform a release process, which is an example of a predetermined process, on the second target object OBJ from which the held first target object OBJ should be released in order to fit (i.e., insert) the held first target object OBJ into a hole formed in the second target object OBJ.
  • both the fine movement process and the rough movement process may be considered as processes performed on the second target object OBJ from which the end effector 5 should release the first target object OBJ.
  • both the fine movement process and the rough movement process may be considered as processes for moving the end effector 5 holding the first target object OBJ to the second target object OBJ from which the end effector 5 should release the first target object OBJ.
  • both the fine movement process and the rough movement process may be considered to be processes that control the robot 1 by moving the end effector 5 so that the end effector 5 approaches the second target object OBJ from which the end effector 5 is to release the first target object OBJ.
  • At least one of the fine movement processing and the rough movement processing may be performed on a moving target object OBJ.
  • a workpiece W which is an example of a target object OBJ
  • a placement device T that can function as a transport device for transporting the workpiece W
  • at least one of the fine movement processing and the rough movement processing may be performed on the workpiece W being moved by the transport device.
  • the placement device T which is an example of a target object OBJ
  • the placement device T which is an example of a target object OBJ
  • the placement device T which is an example of a target object OBJ
  • at least one of the fine movement processing and the rough movement processing may be performed on the moving placement device T.
  • at least one of the fine movement processing and the rough movement processing may be performed on a stationary target object OBJ.
  • the fine movement process and the rough movement process differ from each other in that the method of controlling the robot 1 to move the end effector 5 by the fine movement process is different from the method of controlling the robot 1 to move the end effector 5 by the rough movement process.
  • the differences between the fine movement process and the rough movement process will be explained in detail later when explaining the flow of the robot control process, but the differences will be briefly explained below.
  • the fine movement process includes a process of controlling the robot 1 to move the end effector 5 based on at least one of the calculation results of the position and orientation of the target object OBJ based on the image data IMG generated by the imaging device 21.
  • the fine movement process includes a process of generating a control signal for controlling the robot 1 to move the end effector 5 based on at least one of the calculation results of the position and orientation of the target object OBJ based on the image data IMG generated by the imaging device 21, and outputting the generated control signal (e.g., outputting to the robot 1 or the robot control device 4).
  • the fine movement process may include a process of acquiring image data IMG generated by the imaging device 21 from the imaging device 21. Since the imaging device 21 generates image data IMG to generate a control signal, the fine movement process may include a process of the imaging device 21 generating image data by imaging the target object OBJ. Since the imaging device 21 generates image data IMG to generate a control signal, the fine movement process may be performed in parallel with a process of the imaging device 21 generating image data by imaging the target object OBJ. In this case, the fine movement process may be considered to include a process of the imaging device 21 generating image data by imaging the target object OBJ.
  • the fine movement process may include a process of calculating at least one of the position and orientation of the target object OBJ based on the image data IMG generated by the imaging device 21.
  • the fine movement process may include a process of generating a control signal for controlling the robot 1 to move the end effector 5 based on the calculation result of at least one of the position and orientation of the target object OBJ.
  • the fine movement process may include a process of outputting the generated control signal (e.g., outputting it to the robot 1 or the robot control device 4).
  • the rough movement process includes a process of controlling the robot 1 to move the end effector 5 without using image data IMG generated by the imaging device 21.
  • the rough movement process includes a process of controlling the robot 1 to move the end effector 5 without using at least one of the calculation results of the position and orientation of the target object OBJ based on the image data IMG.
  • the rough movement process includes a process of generating a control signal to control the robot 1 to move the end effector 5 without using image data IMG generated by the imaging device 21, and outputting the generated control signal (e.g., outputting to the robot 1).
  • the rough movement process includes a process of generating a control signal to control the robot 1 to move the end effector 5 without using at least one of the calculation results of the position and orientation of the target object OBJ based on the image data IMG, and outputting the generated control signal (e.g., outputting to the robot 1).
  • control signal used by the rough movement processing may be generated in advance before the rough movement processing is performed.
  • the rough movement processing may not include processing for generating a control signal that controls the robot 1 to move the end effector 5.
  • the rough movement processing may include processing for outputting (e.g., generating) a control signal that was generated in advance before the rough movement processing is started.
  • the control signal used by at least one of the multiple rough movement processings may be generated in advance.
  • control device 3 starts the fine movement process after the rough movement process.
  • control device 3 performs the fine movement process after the rough movement process.
  • control device 3 may generate a control signal for controlling the robot 1 to move the end effector 5 based on at least one of the calculation results of the position and orientation of the target object OBJ based on the image data IMG generated by the imaging device 21 capturing an image of the target object OBJ after the rough movement process.
  • control device 3 may start the fine movement process after the robot control device 4 starts the rough movement process.
  • the control device 3 may perform the fine movement process after the robot control device 4 starts the rough movement process.
  • the robot control device 4 may start the rough movement process before the control device 3 starts the fine movement process.
  • the robot control device 4 may perform the rough movement process before the control device 3 starts the fine movement process.
  • the control device 3 may generate a control signal for controlling the robot 1 to move the end effector 5 based on at least one of the calculation results of the position and the orientation of the target object OBJ based on the image data IMG generated by the imaging device 21 capturing an image of the target object OBJ after the robot control device 4 starts the rough movement process.
  • the control device 3 starts the fine movement process after the robot control device 4 has performed the rough movement process. That is, the control device 3 performs the fine movement process after the robot control device 4 has performed the rough movement process. In other words, the control device 3 starts the fine movement process after the robot control device 4 has completed (i.e., ended) the rough movement process. That is, the control device 3 performs the fine movement process after the robot control device 4 has completed (i.e., ended) the rough movement process.
  • the control device 3 generates a control signal for controlling the robot 1 to move the end effector 5 based on at least one of the calculation results of the position and orientation of the target object OBJ based on the image data IMG generated by the imaging device 21 capturing the target object OBJ after the robot control device 4 has performed the rough movement process.
  • the control device 3 generates a control signal for controlling the robot 1 to move the end effector 5 based on at least one of the calculation results of the position and orientation of the target object OBJ based on the image data IMG generated by the imaging device 21 capturing the target object OBJ after the robot control device 4 has completed (i.e., ended) the rough movement process.
  • control device 3 may start the fine movement process before the robot control device 4 completes (i.e., ends) the rough movement process.
  • control device 3 may perform at least a part of the fine movement process before the robot control device 4 completes (i.e., ends) the rough movement process.
  • the rough movement process performed before the fine movement process may include a process of moving the end effector 5 so that the end effector 5 approaches the target object OBJ until the distance between the end effector 5 and the target object OBJ becomes a first distance (i.e., a process of controlling the robot 1 to move the end effector 5; the same applies below).
  • the fine movement process performed after the rough movement process may include a process of moving the end effector 5 so that the end effector 5 approaches the target object OBJ until the distance between the end effector 5 and the target object OBJ becomes a second distance that is shorter than the first distance.
  • the fine movement process may include a process of moving the end effector 5 so that the end effector 5, which has approached the target object OBJ by the rough movement process, approaches even closer to the target object OBJ.
  • the fine movement process may include a process of moving the end effector 5 so that it is located at a target processing position TPP where the end effector 5 should be located when performing a predetermined process on the target object OBJ.
  • the rough movement process may include a process of moving the end effector 5 so that it is located at a target movement position TMP that is different from the target processing position TPP.
  • the rough movement process may include a process of moving the end effector 5 so that it is located at a target movement position TMP that is away from the target processing position TPP. Note that the distance from the target object OBJ to the target processing position TPP is shorter than the distance from the target object OBJ to the target movement position TMP.
  • the end effector 5 may perform a holding process, which is an example of a predetermined process, on the target object OBJ in order to hold the target object OBJ.
  • the fine movement process may include a process of moving the end effector 5 so that the end effector 5 is located at a target processing position TPP where the end effector 5 should be located when the end effector 5 holds the target object OBJ.
  • the target processing position TPP may be referred to as a target holding position.
  • the rough movement process may include a process of moving the end effector 5 so that the end effector 5 is located at a target movement position TMP that is different from the target processing position TPP.
  • the end effector 5 may perform a release process, which is an example of a predetermined process, in order to release the held target object OBJ.
  • the fine movement process may include a process of moving the end effector 5 so that it is located at a target processing position TPP where the end effector 5 should be located when the end effector 5 releases the target object OBJ.
  • the target processing position TPP may be referred to as a target release position.
  • the rough movement process may include a process of moving the end effector 5 so that it is located at a target movement position TMP that is different from the target processing position TPP.
  • the robot control device 4 may perform the rough movement process to roughly adjust the position of the end effector 5 relative to the target object OBJ.
  • the control device 3 may perform the fine movement process to finely adjust the position of the end effector 5 relative to the target object OBJ.
  • the movement distance of the end effector 5 by the rough movement process may be longer than the movement distance of the end effector 5 by the fine movement process.
  • the movement distance of the end effector 5 by the fine movement process may be shorter than the movement distance of the end effector 5 by the rough movement process.
  • the movement speed of the end effector 5 by the rough movement process may be faster than the movement speed of the end effector 5 by the fine movement process.
  • the movement speed of the end effector 5 by the fine movement process may be slower than the movement speed of the end effector 5 by the rough movement process.
  • the movement speed of the end effector 5 referred to here may mean the maximum movement speed, the average movement speed, or some other movement speed.
  • the robot system SYS performs robot control processing using the control device 3 and the robot control device 4.
  • an apparatus including the control device 3 and the robot control device 4 may be referred to as a control system performing robot control processing.
  • an apparatus including the control device 3 but not the robot control device 4 may be referred to as a control system performing robot control processing.
  • An apparatus including the robot control device 4 but not the control device 3 may be referred to as a control system performing robot control processing.
  • the imaging unit 2 particularly the imaging device 21
  • an apparatus including the imaging unit 2 and at least one of the control device 3 and the robot control device 4 may be referred to as a control system performing robot control processing.
  • An apparatus including the imaging unit 2 and at least one of the control device 3 and the robot control device 4 may be referred to as an imaging system.
  • Each of Figures 3(a) to 3(d) shows a first example of the operation of the robot 1 controlled by a robot control process including a rough movement process and a fine movement process.
  • each of Figures 3(a) to 3(d) shows an example of the operation of the robot 1 for holding (i.e., picking up) a workpiece W stored in a storage box CB, which is an example of a mounting device T, using the end effector 5.
  • a storage box CB which is an example of a mounting device T, using the end effector 5.
  • each of Figures 3(a) to 3(d) shows an example of the operation of the robot 1 when the end effector 5 performs a holding process.
  • FIGS. 3(a) to 3(d) show an example in which a single workpiece W is stored in the storage box CB.
  • multiple workpieces W may be stored in the storage box CB.
  • the multiple workpieces W may be stored regularly in the storage box CB.
  • the multiple workpieces W may be located at regular positions in the storage box CB.
  • the multiple workpieces W may be located at predetermined regular positions in the storage box CB.
  • the multiple workpieces W having regular postures may be located in the storage box CB.
  • the multiple workpieces W may be stored irregularly in the storage box CB.
  • the multiple workpieces W may be located at irregular positions in the storage box CB.
  • the multiple workpieces W having irregular postures may be located in the storage box CB.
  • the storage box CB may be referred to as a container.
  • a state in which multiple workpieces W are irregularly stored in the storage box CB may be considered equivalent to a state in which multiple workpieces W are randomly or haphazardly stored in the storage box CB.
  • the orientation of a first workpiece W among the multiple workpieces W may be different from the orientation of a second workpiece W among the multiple workpieces W that is different from the first workpiece W.
  • the posture of the first workpiece W may be different from the posture of the second workpiece W.
  • a third workpiece W different from the first workpiece W may overlap the first workpiece W, while a fourth workpiece W different from the second workpiece W may not overlap the second workpiece W.
  • a third workpiece W may overlap a first workpiece W
  • a fourth workpiece W may overlap a second workpiece W
  • the size of the overlapping area between the first workpiece W and the third workpiece W may be different from the size of the overlapping area between the second workpiece W and the fourth workpiece W.
  • the robot control device 4 may first perform a rough movement process. Specifically, the robot control device 4 may perform a rough movement process to move the end effector 5 so that the end effector 5 is positioned at a target movement position TMP#1. In the example shown in FIG. 3(a), a position directly above the workpiece W is used as the target movement position TMP#1.
  • the position of the end effector 5 may mean any position determined based on the end effector 5.
  • An example of the position of the end effector 5 is the tool center point of the end effector 5.
  • the tool center point of the end effector 5 may mean a point at which the end effector 5 contacts the target object OBJ (e.g., the workpiece W).
  • the tool center point may mean a point in the space between the tips of the multiple claw members of the hand clipper.
  • the end effector 5 is a vacuum clipper as described above
  • the tool center point may mean a point in the space between multiple suction ports formed in the vacuum clipper and each of which can suck in gas.
  • Such a tool center point may be set by a user of the robot system SYS. Alternatively, the tool center point may be set in advance for each end effector 5.
  • the tool center point is not limited to a point at which the end effector 5 contacts the target object OBJ, and may be any point on the end effector 5.
  • the position of the end effector 5 may refer to a position determined based on a global coordinate system, which will be described later.
  • the position of the end effector 5 may refer to a position determined based on a robot coordinate system, which will be described later.
  • the control device 3 may perform a fine movement process as shown in FIG. 3(b). Specifically, the control device 3 may perform a fine movement process to move the end effector 5 so that the end effector 5 is positioned at the target processing position TPP#1.
  • the target processing position TPP#1 is the target processing position TPP where the end effector 5 should be positioned when the end effector 5 holds the workpiece W.
  • the control device 3 may generate a control signal for controlling the end effector 5 so that the end effector 5 holds the workpiece W, and output the generated control signal to the end effector 5.
  • the end effector 5 holds the workpiece W under the control of the control device 3 (or under the control of the robot control device 4).
  • the robot control device 4 may perform a rough movement process so that the end effector 5 holding the workpiece W is withdrawn.
  • the robot control device 4 may perform the rough movement process by withdrawing the end effector 5 holding the workpiece W.
  • the robot control device 4 may perform the rough movement process so that the end effector 5 moves from the target processing position TPP#1 to the target processing position TMP#1 along the same movement path as the movement path along which the end effector 5 moved from the target movement position TMP#1 to the target processing position TPP#1.
  • the robot control device 4 may perform a rough movement process so that the end effector 5 holding the workpiece W is withdrawn.
  • each of Fig. 4(a) to Fig. 4(d) shows a second example of the operation of the robot 1 controlled by a robot control process including a rough movement process and a fine movement process.
  • each of Fig. 4(a) to Fig. 4(d) shows an example of the operation of the robot 1 releasing the workpiece W held by the end effector 5 so as to place the workpiece W held by the end effector 5 on a pallet PLT, which is an example of a placement device T.
  • each of Fig. 4(a) to Fig. 4(d) shows an example of the operation of the robot 1 when the end effector 5 performs a release process.
  • the robot control device 4 may first perform a rough movement process. Specifically, the robot control device 4 may perform a rough movement process to move the end effector 5 so that the end effector 5 is positioned at a target movement position TMP#2. In the example shown in FIG. 4(a), a position directly above the pallet PLT is used as the target movement position TMP#2.
  • the control device 3 may perform a fine movement process as shown in FIG. 4(b). Specifically, the control device 3 may perform a fine movement process to move the end effector 5 so that the end effector 5 is positioned at the target processing position TPP#2.
  • the target processing position TPP#2 is the target processing position TPP where the end effector 5 should be positioned when the end effector 5 releases the workpiece W. Note that, when the pallet PLT moves as described above, the target processing position TPP#2 may also move in accordance with the movement of the pallet PLT#2.
  • the control device 3 may generate a control signal for controlling the end effector 5 so that the end effector 5 releases the workpiece W, and output the generated control signal to the end effector 5.
  • the end effector 5 releases the workpiece W under the control of the control device 3 (or under the control of the robot control device 4).
  • the control device 3 may perform fine movement processing so that the end effector 5 that has released the workpiece W retreats, if necessary.
  • the robot control device 4 may perform rough movement processing so that the end effector 5 that has released the workpiece W retreats.
  • the robot control device 4 may perform the rough movement processing to retreat the end effector 5 that has released the workpiece W.
  • the robot control device 4 may perform rough movement processing so that the end effector 5 moves from the target processing position TPP#2 to the target processing position TMP#2 along the same movement path as the movement path along which the end effector 5 moved from the target movement position TMP#2 to the target processing position TPP#2.
  • the robot control device 4 may perform rough movement processing so that the end effector 5 that has released the workpiece W retreats, if necessary.
  • each of Figs. 5(a) to 5(d) and Figs. 6(a) to 6(b) shows a third example of the operation of the robot 1 controlled by the robot control process including the rough movement process and the fine movement process.
  • each of Figs. 5(a) to 5(d) and Figs. 6(a) to 6(b) shows an example of the operation of the robot 1 in which the end effector 5, which contains multiple rod-shaped first workpieces W1 inside, sequentially releases the multiple first workpieces W1 so that the end effector 5 fits (inserts) each of the multiple first workpieces W1 into the multiple holes HL formed in the second workpiece W2.
  • each of Figs. 5(a) to 5(d) and Figs. 6(a) to 6(b) shows an example of the operation of the robot 1 when the end effector 5 performs the release process.
  • the end effector 5 performs a release process on each of the multiple holes HL formed in the second workpiece W2.
  • the second workpiece W2 has multiple target portions (processing target portions) each having a hole HL formed therein, and the end effector 5 performs a release process on each of the multiple target portions of the second workpiece W2 each having a hole HL formed therein.
  • the target object OBJ may have multiple target portions on which the end effector 5 performs a predetermined process, and the end effector 5 may perform the predetermined process on the multiple target portions in sequence.
  • the target portions processing target portions
  • the target members processing target members
  • multiple holes HL including holes HL#1 and HL#2 are formed in the second workpiece W2, and the operation of the robot 1 is described in which the end effector 5 sequentially releases multiple first workpieces W1, including first workpieces W1#1 and W1#2, so that the multiple first workpieces W1 are fitted (inserted) into the multiple holes HL including holes HL#1 and HL#2.
  • the robot control device 4 may first perform a rough movement process. Specifically, the robot control device 4 may perform a rough movement process to move the end effector 5 so that the end effector 5 is positioned at the target movement position TMP#31.
  • a position near the position directly above hole HL#1 is used as the target movement position TMP#31.
  • the target movement position TMP#31 may be the position directly above hole HL#1 or another position.
  • the control device 3 may perform a fine movement process as shown in FIG. 5(b). Specifically, the control device 3 may perform a fine movement process to move the end effector 5 so that the end effector 5 is positioned at the target processing position TPP#31.
  • the target processing position TPP#31 is the target processing position TPP where the end effector 5 should be positioned when the end effector 5 releases the first workpiece W1#1 to fit the first workpiece W1#1 into the hole HL#1.
  • the control device 3 may generate a control signal for controlling the end effector 5 so that the end effector 5 releases the first workpiece W1#1, and output the generated control signal to the end effector 5.
  • the end effector 5 releases the first workpiece W1#1 under the control of the control device 3 (or under the control of the robot control device 4).
  • the first workpiece W1#1 released by the end effector 5 is fitted (inserted) into the hole HL#1.
  • the robot control device 4 may perform the rough movement process again. Specifically, the robot control device 4 may perform the rough movement process to move the end effector 5 so that the end effector 5 is positioned at the target movement position TMP#32.
  • the target movement position TMP#32 may be a position directly above hole HL#1 or another position.
  • the control device 3 may perform the fine movement process again, as shown in FIG. 6(a). Specifically, the control device 3 may perform the fine movement process to move the end effector 5 so that the end effector 5 is positioned at the target processing position TPP#32.
  • the target processing position TPP#32 is the target processing position TPP where the end effector 5 should be positioned when the end effector 5 releases the first workpiece W1#2 to fit the first workpiece W1#2 into the hole HL#2.
  • the control device 3 may generate a control signal for controlling the end effector 5 so that the end effector 5 releases the first workpiece W1#2, and output the generated control signal to the end effector 5.
  • the end effector 5 releases the first workpiece W1#2 under the control of the control device 3 (or under the control of the robot control device 4).
  • the first workpiece W1#2 released by the end effector 5 is fitted (inserted) into the hole HL#2.
  • control device 3 and the robot control device 4 repeat the same operations.
  • the multiple first workpieces W1 are fitted (inserted) into the multiple holes HL formed in the second workpiece W2.
  • Fig. 7 is a block diagram showing the configuration of the control device 3.
  • the control device 3 includes a calculation device 31, a storage device 32, and a communication device 33.
  • the control device 3 may further include an input device 34 and an output device 35.
  • the control device 3 does not have to include at least one of the input device 34 and the output device 35.
  • the calculation device 31, the storage device 32, the communication device 33, the input device 34, and the output device 35 may be connected via a data bus 36.
  • the arithmetic device 31 includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array).
  • the arithmetic device 31 reads a computer program.
  • the arithmetic device 31 may read a computer program stored in the storage device 32.
  • the arithmetic device 31 may read a computer program stored in a computer-readable and non-transient recording medium using a recording medium reading device (not shown) provided in the control device 3.
  • the arithmetic device 31 may obtain (i.e., download or read) a computer program from a device (not shown) located outside the control device 3 via the communication device 33 (or other communication device).
  • the arithmetic device 31 executes the read computer program.
  • a logical function block for executing the processing to be performed by the control device 3 (for example, the robot control processing including the fine movement processing described above) is realized within the calculation device 31.
  • the calculation device 31 can function as a controller for realizing a logical function block for executing the processing to be performed by the control device 3.
  • a computation model that can be constructed by machine learning may be implemented in the computation device 31 by the computation device executing a computer program.
  • An example of a computation model that can be constructed by machine learning is, for example, a computation model including a neural network (so-called artificial intelligence (AI)).
  • learning of the computation model may include learning of parameters of the neural network (for example, at least one of the weight and bias).
  • the computation device 31 may execute a robot control process using the computation model.
  • the operation of executing the robot control process may include the operation of executing the robot control process using the computation model.
  • a computation model that has already been constructed by offline machine learning using teacher data may be implemented in the computation device 31.
  • the computation model implemented in the computation device 31 may be updated by online machine learning on the computation device 31.
  • the computing device 31 may execute the robot control process using a computation model implemented in a device external to the computing device 31 (i.e., a device provided outside the control device 3) in addition to or instead of the computation model implemented in the computing device 31.
  • the recording medium for recording the computer program executed by the arithmetic unit 31 may be at least one of the following: CD-ROM, CD-R, CD-RW, flexible disk, MO, DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW, optical disks such as Blu-ray (registered trademark), magnetic media such as magnetic tape, magneto-optical disk, semiconductor memory such as USB memory, and any other medium capable of storing a program.
  • the recording medium may include a device capable of recording a computer program (for example, a general-purpose device or a dedicated device in which a computer program is implemented in a state in which it can be executed in at least one of the forms of software and firmware, etc.).
  • each process or function included in the computer program may be realized by a logical processing block realized within the computing device 31 (i.e., a computer) by executing the computer program, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) included in the computing device 31, or may be realized in a form that combines logical processing blocks and partial hardware modules that realize some elements of the hardware.
  • FPGA Field Programmable Gate Array
  • ASIC Application Specific Integrated Circuit
  • FIG. 7 shows an example of logical functional blocks realized within the arithmetic device 31 to execute robot control processing.
  • a position and orientation calculation unit 311 and a signal generation unit 312 are realized within the arithmetic device 31.
  • the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ based on the image data IMG.
  • the signal generation unit 312 may generate a control signal for controlling the robot 1 based on at least one of the position and orientation of the target object OBJ calculated by the position and orientation calculation unit 311.
  • the storage device 32 can store desired data.
  • the storage device 32 may temporarily store a computer program executed by the arithmetic device 31.
  • the storage device 32 may temporarily store data that is temporarily used by the arithmetic device 31 when the arithmetic device 31 is executing a computer program.
  • the storage device 32 may store data that the control device 3 stores for a long period of time.
  • the storage device 32 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.
  • the storage device 32 may include a non-temporary recording medium.
  • the communication device 33 can communicate with each of the imaging unit 2 and the robot control device 4 via a communication network not shown.
  • the communication device 33 may be able to communicate with other devices other than the imaging unit 2 and the robot control device 4 in addition to or instead of at least one of the imaging unit 2 and the robot control device 4 via a communication network not shown.
  • the communication device 33 may receive (i.e., acquire) the image data IMG and IMG from the imaging unit 2.
  • the communication device 33 may transmit (i.e., output) a control signal generated by the signal generation unit 312 to the robot control device 4.
  • the communication device 33 may transmit (i.e., output) the control signal generated by the signal generation unit 312 to the robot 1.
  • the communication device 33 that outputs the control signal to the robot 1 or the robot control device 4 may be referred to as an output unit.
  • the input device 34 is a device that accepts information input to the control device 3 from outside the control device 3.
  • the input device 34 may include an operation device (e.g., at least one of a keyboard, a mouse, and a touch panel) that can be operated by a user of the control device 3.
  • the input device 34 may include a recording medium reading device that can read information recorded as data on a recording medium that can be attached externally to the control device 3.
  • information can be input as data to the control device 3 from a device external to the control device 3 via the communication device 33.
  • the communication device 33 may function as an input device that accepts information input to the control device 3 from outside the control device 3.
  • the output device 35 is a device that outputs information to the outside of the control device 3.
  • the output device 35 may output information as an image. That is, the output device 35 may include a display device (a so-called display) capable of displaying images.
  • the output device 35 may output information as sound. That is, the output device 35 may include an audio device (a so-called speaker) capable of outputting sound.
  • the output device 35 may output information on paper. That is, the output device 35 may include a printing device (a so-called printer) capable of printing desired information on paper.
  • the output device 35 may output information as data to a recording medium that can be attached externally to the control device 3.
  • information can be output as data from the control device 3 to a device external to the control device 3 via the communication device 33.
  • the communication device 33 may function as an output device that outputs information to a device external to the control device 3.
  • Fig. 8 is a block diagram showing the configuration of the robot controller 4.
  • the robot control device 4 includes a calculation device 41, a storage device 42, and a communication device 43.
  • the robot control device 4 may further include an input device 44 and an output device 45.
  • the robot control device 4 does not have to include at least one of the input device 44 and the output device 45.
  • the calculation device 41, the storage device 42, the communication device 43, the input device 44, and the output device 45 may be connected via a data bus 46.
  • the arithmetic device 41 includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array).
  • the arithmetic device 41 reads a computer program.
  • the arithmetic device 41 may read a computer program stored in the storage device 42.
  • the arithmetic device 41 may read a computer program stored in a computer-readable and non-transient recording medium using a recording medium reading device (not shown) provided in the robot control device 4.
  • the arithmetic device 41 may obtain (i.e., download or read) a computer program from a device (not shown) located outside the robot control device 4 via the communication device 43 (or other communication device).
  • the arithmetic device 41 executes the read computer program.
  • a logical function block for executing the processing to be performed by the robot control device 4 (for example, the robot control processing including the rough movement processing described above) is realized within the calculation device 41.
  • the calculation device 41 can function as a controller for realizing a logical function block for executing the processing to be performed by the robot control device 4.
  • a computation model that can be constructed by machine learning may be implemented in the computation device 41 by the computation device executing a computer program.
  • An example of a computation model that can be constructed by machine learning is, for example, a computation model including a neural network (so-called artificial intelligence (AI)).
  • learning of the computation model may include learning of parameters of the neural network (for example, at least one of the weight and bias).
  • the computation device 41 may execute a robot control process using the computation model.
  • the operation of executing the robot control process may include the operation of executing the robot control process using the computation model.
  • a computation model that has already been constructed by offline machine learning using teacher data may be implemented in the computation device 41.
  • the computation model implemented in the computation device 41 may be updated by online machine learning on the computation device 41.
  • the computing device 41 may execute the robot control process using a computation model implemented in a device external to the computing device 41 (i.e., a device provided outside the robot control device 4) in addition to or instead of the computation model implemented in the computing device 41.
  • the recording medium for recording the computer program executed by the arithmetic unit 41 may be at least one of the following: CD-ROM, CD-R, CD-RW, flexible disk, MO, DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW, optical disks such as Blu-ray (registered trademark), magnetic media such as magnetic tape, magneto-optical disk, semiconductor memory such as USB memory, and any other medium capable of storing a program.
  • the recording medium may include a device capable of recording a computer program (for example, a general-purpose device or a dedicated device in which a computer program is implemented in a state in which it can be executed in at least one of the forms of software and firmware, etc.).
  • each process or function included in the computer program may be realized by a logical processing block realized within the computing device 41 (i.e., a computer) by executing the computer program, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) included in the computing device 41, or may be realized in a form that combines logical processing blocks and partial hardware modules that realize some elements of the hardware.
  • FPGA Field Programmable Gate Array
  • ASIC Application Specific Integrated Circuit
  • FIG. 8 shows an example of a logical functional block realized within the arithmetic device 41 to execute robot control processing.
  • a signal generating unit 411 is realized within the arithmetic device 41.
  • the signal generating unit 411 may generate a control signal for controlling the robot 1 without using the image data IMG generated by the imaging device 21.
  • the storage device 42 can store desired data.
  • the storage device 42 may temporarily store a computer program executed by the arithmetic device 41.
  • the storage device 42 may temporarily store data that is temporarily used by the arithmetic device 41 when the arithmetic device 41 is executing a computer program.
  • the storage device 42 may store data that is to be stored long-term by the robot control device 4.
  • the storage device 42 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, an optical magnetic disk device, an SSD (Solid State Drive), and a disk array device.
  • the storage device 42 may include a non-temporary recording medium.
  • the communication device 43 is capable of communicating with both the robot 1 and the control device 3 via a communication network (not shown). Alternatively, the communication device 43 may be capable of communicating with another device different from the control device 3 in addition to or instead of at least one of the robot 1 and the control device 3 via a communication network (not shown). In this embodiment, the communication device 43 may receive (i.e., acquire) a control signal generated by the control device 3 from the control device 3. Furthermore, the communication device 43 may transmit (i.e., output) a control signal generated by the signal generation unit 411 to the robot 1 (particularly, at least one of the robot drive signal and the end effector drive signal described above). The communication device 43 that outputs the control signal to the robot 1 may be referred to as an output unit.
  • the input device 44 is a device that accepts information input to the robot control device 4 from outside the robot control device 4.
  • the input device 44 may include an operation device (e.g., at least one of a keyboard, a mouse, and a touch panel) that can be operated by a user of the robot control device 4.
  • the input device 44 may include a recording medium reading device that can read information recorded as data on a recording medium that can be attached externally to the robot control device 4.
  • information can be input as data to the robot control device 4 from a device external to the robot control device 4 via the communication device 43.
  • the communication device 43 may function as an input device that accepts information input to the robot control device 4 from outside the robot control device 4.
  • the output device 45 is a device that outputs information to the outside of the robot control device 4.
  • the output device 45 may output information as an image. That is, the output device 45 may include a display device (so-called a display) capable of displaying images.
  • the output device 45 may output information as sound. That is, the output device 45 may include an audio device (so-called a speaker) capable of outputting sound.
  • the output device 45 may output information on paper. That is, the output device 45 may include a printing device (so-called a printer) capable of printing desired information on paper.
  • the output device 45 may output information as data to a recording medium that can be attached externally to the robot control device 4.
  • information can be output as data from the robot control device 4 to a device external to the robot control device 4 via the communication device 43.
  • the communication device 43 may function as an output device that outputs information to a device external to the robot control device 4.
  • Fig. 9 is a flowchart showing the flow of the robot control processing.
  • the rough movement process is performed first, and then the fine movement process is performed. Therefore, in the following explanation, the rough movement process will be explained first, and then the fine movement process will be explained.
  • the robot control device 4 performs the rough movement processing (step S41).
  • the rough movement processing includes a process of moving the end effector 5 without using image data IMG.
  • the signal generating unit 411 provided in the robot control device 4 generates a control signal (particularly, a robot drive signal that can be input to the robot 1 to control the operation of the robot 1) without using image data IMG, and outputs the generated control signal to the robot 1.
  • the control signal used by the rough movement processing is generated in advance before the rough movement processing is performed, the signal generating unit 411 does not need to generate a control signal.
  • the signal generating unit 411 may output a control signal generated in advance to the robot 1 before the rough movement processing is started. As a result, the robot 1 moves the end effector 5 in a movement mode based on the control signal. Note that, when the rough movement processing is performed multiple times, the control signal used by at least one rough movement processing of the multiple rough movement processing may be generated in advance.
  • the rough movement process may include a process of controlling the robot 1 to move the end effector 5 according to the movement instruction information without using the image data IMG.
  • the movement instruction information includes information instructing the movement mode of the end effector 5.
  • the signal generation unit 411 generates a control signal (in particular, a robot drive signal that can be input to the robot 1 to control the operation of the robot 1) for controlling the robot 1 to move the end effector 5 according to the movement instruction information, and outputs the generated control signal to the robot 1.
  • the end effector 5 moves in the movement mode instructed by the movement instruction information.
  • the movement mode of the end effector 5 by the rough movement processing may be defined by a computer program executed by the robot control device 4 to perform the rough movement processing.
  • the movement instruction information may be considered to be included in at least a part of the computer program executed by the robot control device 4 to perform the rough movement processing.
  • the movement instruction information may be considered to be included in a part of the program portion (program information) in which the movement mode of the end effector 5 is described, of the computer program executed by the robot control device 4 to perform the rough movement processing.
  • the movement mode of the end effector 5 by the rough movement processing may include a position to which the end effector 5 should be moved by the rough movement processing (target movement position TMP shown in FIG. 3, etc.).
  • the movement instruction information may include a target movement position TMP (see FIG. 3, etc.) to which the end effector 5 should be moved by the rough movement processing.
  • the movement instruction information may include information indicating the target movement position TMP.
  • the information indicating the target movement position TMP may be considered to be included in a part of the program portion (program information) of the computer program in which the target movement position TMP is described.
  • the movement mode of the end effector 5 by the rough movement processing may include a target movement distance, which is the distance the end effector 5 should move by the rough movement processing.
  • the movement instruction information may include a target movement distance that the end effector 5 should move by the rough movement processing.
  • the movement instruction information may include information indicating the target movement distance.
  • the information indicating the target movement distance may be considered to be included in a part of the program portion (program information) of the computer program in which the target movement distance is described.
  • the movement mode of the end effector 5 by the rough movement processing may include a target movement direction, which is the direction in which the end effector 5 should move by the rough movement processing.
  • the movement instruction information may include a target movement direction in which the end effector 5 should move by the rough movement processing.
  • the movement instruction information may include information indicating the target movement direction.
  • the information indicating the target movement direction may be considered to be included in a part of the program portion (program information) of the computer program in which the target movement direction is described.
  • the movement mode of the end effector 5 by the rough movement processing may include a target movement posture, which is the posture of the end effector 5 after it has completed its movement by the rough movement processing.
  • the movement instruction information may include the target movement posture.
  • the movement instruction information may include information indicating the target movement posture.
  • the information indicating the target movement posture may be considered to be included in a part of the program portion (program information) of the computer program in which the target movement posture is described.
  • the movement instruction information may be set by a user of the robot system SYS.
  • the movement instruction information may be input to the robot control device 4 by a user of the robot system SYS.
  • the movement instruction information may be set by the user of the robot system SYS inputting the movement instruction information.
  • an information setting device capable of automatically setting movement instruction information may automatically set the movement instruction information. That is, in addition to or instead of a user setting the movement instruction information, the information setting device may automatically set the movement instruction information.
  • the robot control device 4 (particularly the calculation device 41) capable of functioning as an information setting device may automatically set the movement instruction information.
  • an information setting device different from the robot control device 4 may automatically set the movement instruction information.
  • the information setting device may set the movement instruction information using a calculation model capable of outputting the movement instruction information.
  • the calculation model capable of outputting the movement instruction information may include a rule-based calculation model.
  • the calculation model capable of outputting the movement instruction information may include a calculation model capable of learning by machine learning (so-called artificial intelligence).
  • the user may set the movement instruction information by directly setting the movement instruction information.
  • the robot control device 4 may use the output device 35 capable of functioning as a display device to display a GUI (Graphical User Interface) that the user can operate to set the movement instruction information, and the user may set the movement instruction information on the GUI displayed by the output device 35.
  • the GUI may include at least one of a GUI that the user can operate to set the target movement position TMP, a GUI that the user can operate to set the target movement distance, a GUI that the user can operate to set the target movement direction, and a GUI that the user can operate to set the target movement posture.
  • the movement instruction information that the user directly sets may be reflected in a computer program that the robot control device 4 executes to perform the rough movement process.
  • the movement instruction information that the user directly sets may be reflected in a registry in which information regarding the settings of the computer program that the robot control device 4 executes to perform the rough movement process is stored.
  • the user may indirectly set the movement instruction information by creating at least a part of a computer program that the robot control device 4 executes to perform rough movement processing.
  • the user may indirectly set the movement instruction information by creating a computer program that the robot control device 4 executes to perform rough movement processing and that reflects the movement instruction information that the user wants to set.
  • the user may set the movement instruction information by robot teaching.
  • the user may set the movement instruction information by performing robot teaching using a teaching pendant to actually move the robot 1 in a movement pattern that should be achieved by rough movement processing.
  • the user may set the movement instruction information by performing offline teaching without using the robot 1.
  • the target object OBJ for which the end effector 5 is to perform a predetermined process is included in the imaging field of the imaging device 21 when the rough movement process is completed.
  • the movement instruction information may be set so as to satisfy the condition that "the target object OBJ for which the end effector 5 is to perform a predetermined process is included in the imaging field of the imaging device 21 when the rough movement process is completed."
  • the target movement position TMP indicated by the movement instruction information may be set so as to satisfy the condition that "when the end effector 5 moves to the target movement position TMP, the target object OBJ is included in the imaging field of the imaging device 21 that moves together with the end effector 5.”
  • the target movement distance indicated by the movement instruction information may be set so as to satisfy the condition that "when the end effector 5 moves by the target movement distance, the target object OBJ is included in the imaging field of the imaging device 21 that moves together with the end effector 5.”
  • the target movement direction indicated by the movement instruction information may be set so as to satisfy the condition that "when the end effector 5 moves toward the target movement direction, the target object OBJ is included in the imaging field of view of the imaging device 21 that moves together with the end effector
  • the movement instruction information may be set based on the transport mode of the target object OBJ.
  • the rough movement process may include a process of moving the end effector 5 so that the end effector 5 approaches the target object OBJ.
  • the movement mode of moving the end effector 5 so that the target object OBJ approaches the target object OBJ under a situation in which the target object OBJ is not moving (not being transported) is likely to be different from the movement mode of moving the end effector 5 so that the target object OBJ approaches the target object OBJ under a situation in which the target object OBJ is moving (being transported).
  • the movement mode of moving the end effector 5 so that the target object OBJ approaches the target object OBJ under a situation in which the target object OBJ is moving in a first transport mode is likely to be different from the movement mode of moving the end effector 5 so that the target object OBJ approaches the target object OBJ under a situation in which the target object OBJ is moving (being transported) in a second transport mode different from the first transport mode.
  • the movement instruction information may be set based on the transport mode of the target object OBJ so that the end effector 5 appropriately approaches the target object OBJ being transported in any transport mode (for example, as described above, the end effector 5 is located at the target movement position TMP determined based on the target object OBJ being transported in any transport mode).
  • the robot control device 4 can move the end effector 5 by performing rough movement processing so that the end effector 5 appropriately approaches the target object OBJ.
  • the transport mode of the target object OBJ by the transport device may be referred to as the movement mode of the target object OBJ.
  • the transport mode may include a transport speed.
  • the movement instruction information may be set based on the transport speed of the target object OBJ so that the end effector 5 appropriately approaches the target object OBJ being transported at an arbitrary transport speed (for example, as described above, the end effector 5 is located at the target movement position TMP determined based on the target object OBJ being transported at an arbitrary transport speed).
  • the transport speed of the target object OBJ is a first transport speed
  • the movement instruction information may be set so that the end effector 5 moves at the first movement speed.
  • the movement instruction information may be set so that the end effector 5 moves at the second movement speed different from the first movement speed.
  • the movement instruction information may be set so that the end effector 5 moves at the second movement speed faster than the first movement speed.
  • the transport speed of the target object OBJ is a second transport speed that is slower than the first transport speed
  • the movement instruction information may be set so that the end effector 5 moves at the second transport speed that is slower than the first transport speed.
  • the transport speed may be referred to as the movement speed of the target object OBJ.
  • the transport mode may include the movement speed.
  • the transport mode may include the transport direction.
  • the movement instruction information may be set based on the transport direction of the target object OBJ so that the end effector 5 appropriately approaches the target object OBJ being transported in an arbitrary transport direction (for example, as described above, the end effector 5 is located at the target movement position TMP determined based on the target object OBJ being transported in an arbitrary transport direction).
  • the movement instruction information may be set so that the end effector 5 moves toward the first movement direction.
  • the movement instruction information may be set so that the end effector 5 moves toward the second movement direction different from the first movement direction.
  • the movement instruction information may be set so that the end effector 5 moves toward a movement direction including a directional component along the transport direction of the target object OBJ.
  • the movement instruction information may be set so that the end effector 5 moves toward the same movement direction as the transport direction of the target object OBJ.
  • the movement instruction information may be set so that the end effector 5 moves in a movement direction that diagonally intersects with the transport direction of the target object OBJ.
  • the transport direction may also be referred to as the movement direction of the target object OBJ.
  • the transport mode may include the movement direction.
  • the movement instruction information may be set based on the positional relationship (e.g., relative positional relationship) between at least two of the plurality of different target objects OBJ.
  • the positional relationship e.g., relative positional relationship
  • the movement instruction information may be set based on the positional relationship between the first and second target objects OBJ.
  • the movement instruction information may be set so that the end effector 5 moves from the first target object OBJ toward the second target object OBJ by a rough movement process performed after the end effector 5 performs predetermined processing on the first target object OBJ.
  • the movement instruction information may be set so that the end effector 5 moves in a movement direction including a directional component connecting the first target object OBJ and the second target object OBJ by a rough movement process performed after the end effector 5 performs predetermined processing on the first target object OBJ.
  • the movement instruction information may be set so that the end effector 5 moves at a movement speed according to the relative speed of the second target object OBJ with respect to the first target object OBJ by the rough movement process performed after the end effector 5 performs a predetermined process on the first target object OBJ.
  • the movement instruction information may be set so that the end effector 5 moves from the first target object OBJ to the second target object OBJ at a movement speed that allows the end effector 5 to move within a desired time by the rough movement process performed after the end effector 5 performs a predetermined process on the first target object OBJ.
  • the movement instruction information may be set so that the end effector 5 moves a movement distance according to the distance between the first target object OBJ and the second target object OBJ by the rough movement process performed after the end effector 5 performs a predetermined process on the first target object OBJ.
  • the robot control device 4 can move the end effector 5 so that the end effector 5 approaches the multiple target objects OBJ in sequence by performing the rough movement process.
  • the movement instruction information may be set based on the positional relationship (e.g., relative positional relationship) between at least two of the multiple different target parts.
  • the movement instruction information may be set based on the positional relationship between the first and second target parts. For example, the movement instruction information may be set so that the end effector 5 moves from the first target part to the second target part by a rough movement process performed after the end effector 5 performs a predetermined process on the first target part.
  • the movement instruction information may be set so that the end effector 5 moves in a movement direction including a directional component connecting the first target portion and the second target portion by a rough movement process performed after the end effector 5 performs a predetermined process on the first target portion.
  • the movement instruction information may be set so that the end effector 5 moves from the first target portion to the second target portion at a movement speed that allows the end effector 5 to move within a desired time by a rough movement process performed after the end effector 5 performs a predetermined process on the first target portion.
  • the movement instruction information may be set so that the end effector 5 moves a movement distance according to the distance between the first target portion and the second target portion by a rough movement process performed after the end effector 5 performs a predetermined process on the first target portion.
  • the robot control device 4 can move the end effector 5 so that the end effector 5 approaches multiple target portions in sequence by performing the rough movement process.
  • the user may set the movement instruction information in advance.
  • the user may set the movement instruction information in advance before the robot control device 4 starts the rough movement processing.
  • the rough movement processing may be considered to be processing that controls the robot 1 based on the movement instruction information that has been set in advance.
  • the rough movement processing may be considered to be processing that controls the robot 1 based on a computer program that has been set (generated) in advance.
  • the user may set the movement instruction information after the robot control device 4 starts the rough movement process. That is, the user may set (in other words, update or change) the movement instruction information instructing the movement mode of the end effector 5 by the rough movement process in real time after the robot control device 4 starts the rough movement process. For example, the user may obtain information regarding the position of the target object OBJ during the period when the rough movement process is being performed, and set the movement instruction information based on the obtained information.
  • a workpiece W which is an example of the target object OBJ
  • a placement device T that can function as a transport device that transports (moves) the workpiece W
  • the user may obtain information regarding the position of the workpiece W moving (being transported) on the placement device T from a position detection device (e.g., an encoder) provided on the placement device T, and set the movement instruction information based on the obtained information.
  • a position detection device e.g., an encoder
  • the movement instruction information may be fixed during the period when the rough movement process is being performed. In other words, the movement instruction information may not be changed during the period when the rough movement process is being performed.
  • the target movement position TMP indicated by the movement instruction information is set in advance
  • the target movement position TMP may be fixed at the preset position during the period when the rough movement process is being performed.
  • the target movement distance indicated by the movement instruction information is set in advance
  • the target movement distance may be fixed at the preset distance during the period when the rough movement process is being performed.
  • the target movement direction indicated by the movement instruction information is set in advance
  • the target movement direction may be fixed at the preset direction during the period when the rough movement process is being performed.
  • the target movement posture indicated by the movement instruction information may be set in advance, the target movement posture may be fixed at the preset posture during the period when the rough movement process is being performed. However, even if the movement instruction information is set in advance, the movement instruction information may be changed during the period when the rough movement process is being performed.
  • the robot controller 4 transmits a control permission signal to the controller 3 at a desired timing after starting the rough movement process (step S42). For example, the robot controller 4 transmits a control permission signal to the controller 3 after completing the rough movement process (step S42). That is, the robot controller 4 transmits a control permission signal to the controller 3 after the end effector 5 has finished moving by the rough movement process. However, the robot controller 4 may transmit a control permission signal to the controller 3 before completing the rough movement process. The robot controller 4 transmits a control permission signal to the controller 3 before the end effector 5 has finished moving by the rough movement process (step S42).
  • the control device 3 After the robot control device 4 transmits a control permission signal to the control device 3, the control device 3 receives the control permission signal transmitted from the robot control device 4.
  • the control device 3 that receives the control permission signal starts the fine movement process (step S32). That is, after the control device 3 receives the control permission signal, the control device 3 starts the fine movement process (step S32). In other words, after the control device 3 receives the control permission signal, the control device 3 moves the end effector 5 by performing the fine movement process (step S32).
  • a control device 3 that does not receive a control permission signal does not start the fine movement process. That is, before the control device 3 receives a control permission signal, the control device 3 does not start the fine movement process. In other words, before the control device 3 receives a control permission signal, the control device 3 does not move the end effector 5 by performing the fine movement process.
  • control permission signal may be considered to be a signal by which the robot control device 4 authorizes the control device 3 to control the robot 1 to move the end effector 5 by performing fine movement processing.
  • the robot control device 4 may transmit the control permission signal to the control device 3 at a desired timing at which the robot control device 4 may authorize the control device 3 to control the robot 1 to move the end effector 5 by performing fine movement processing.
  • control device 3 After the control device 3 receives the control permission signal, the control device 3 transmits the control signal generated by the control device 3 to the robot control device 4 in order to control the robot 1 to move the end effector 5 by the fine movement processing.
  • the robot control device 4 controls the robot 1 to move the end effector 5 based on the control signal generated by the control device 3.
  • the control device 3 may be considered to be controlling the robot 1 via the robot control device 4.
  • the control device 3 may transmit the control signal generated by the control device 3 to the robot 1 without going through the robot control device 4 in order to control the robot 1 to move the end effector 5 by the fine movement processing.
  • control device 3 transmits the control signal to the robot 1, after the control device 3 receives the control permission signal, the control signal generated by the control device 3 is actually reflected in the control of the robot 1.
  • the control device 3 may be considered to be controlling the robot 1 without going through the robot control device 4.
  • the control device 3 may not generate a control signal.
  • the control device 3 may generate a control signal, but may not transmit the control signal generated by the control device 3 to the robot control device 4.
  • the control device 3 may transmit the control signal generated by the control device 3 to the robot control device 4, but the robot control device 4 may control the robot 1 using the control signal generated by the robot control device 4 without using the control signal generated by the control device 3.
  • the control device 3 may transmit the control signal generated by the control device 3 to the robot 1, but the robot 1 may not use the control signal generated by the control device 3.
  • the control permission signal may be considered as a signal that the robot control device 4 gives permission to the control device 3 to actually reflect the control signal generated by the control device 3 in the control of the robot 1.
  • the control device 3 After the control device 3 receives the control permission signal, the control signal generated by the control device 3 is actually reflected in the control of the robot 1, so the computer program executed by the robot control device 4 to perform the rough movement process may be at least partially stopped. In this case, the robot 1 operates to move the end effector 5 under the control of the control device 3. In other words, after the control device 3 receives the control permission signal, it may be considered that the control right of the robot 1 is held by the control device 3. On the other hand, before the control device 3 receives the control permission signal, the robot control device 4 controls the robot 1 by executing the computer program executed to perform the rough movement process. In this case, the robot 1 operates to move the end effector 5 under the control of the robot control device 4.
  • control permission signal may be considered to be a signal that passes the control right of the robot 1 from the robot control device 4 to the control device 3.
  • the control device 3 may perform the fine movement process without receiving a signal transmitted from the robot control device 4. In other words, after the control device 3 receives the control permission signal, the control device 3 may perform the fine movement process even if the control device 3 does not receive a signal transmitted from the robot control device 4. After the control device 3 receives the control permission signal, the control device 3 may perform the fine movement process even if no signal is transmitted from the robot control device 4 to the control device 3. This is because, after the control device 3 receives the control permission signal, it can be considered that the control right of the robot 1 is held by the control device 3.
  • the robot control device 4 does not perform the rough movement processing. In other words, while the control device 3 is performing the fine movement processing, the robot control device 4 does not perform the rough movement processing. However, even during the period when the control device 3 holds the control rights of the robot 1 (i.e., the fine movement processing is being performed), the robot control device 4 may perform at least a part of the rough movement processing. In other words, even during the period when the control device 3 is performing the fine movement processing, the robot control device 4 may perform at least a part of the rough movement processing.
  • the control device 3 does not perform the fine movement processing. In other words, while the robot control device 4 is performing the rough movement processing, the control device 3 does not perform the fine movement processing. However, even during the period when the robot control device 4 holds the control rights of the robot 1 (i.e., the rough movement processing is being performed), the control device 3 may perform at least a part of the fine movement processing. That is, even while the robot control device 4 is performing the rough movement process, the control device 3 may perform at least a part of the fine movement process. That is, at least a part of the rough movement process and at least a part of the fine movement process may be performed in parallel. Note that an example in which at least a part of the rough movement process and at least a part of the fine movement process are performed in parallel will be described in the first modified example below.
  • the robot control device 4 may generate a drive signal that can be input to the robot 1 or end effector 5 as a control signal to control the operation of the robot 1 or end effector 5 based on the control signal generated by the control device 3, and may continue the process of outputting the control signal generated by the robot control device 4 to the robot 1 or end effector 5.
  • the robot control device 4 may continue the process of converting the control signal generated by the control device 3 into a control signal (drive signal) that can be input to the robot 1 or end effector 5, and outputting the converted control signal to the robot 1 or end effector 5.
  • the robot control device 4 may stop the process of converting the control signal generated by the control device 3 into a control signal (drive signal) that can be input to the robot 1 or end effector 5, and outputting the converted control signal to the robot 1 or end effector 5.
  • Fig. 10 is a flow chart showing the flow of the fine movement processing.
  • the fine move process may be started after the rough move process is started and before the rough move process is completed.
  • the control device 3 may perform at least a part of the fine move process shown in FIG. 10 during at least a part of the period during which the rough move process is being performed.
  • the position and orientation calculation unit 311 of the control device 3 acquires image data IMG from the imaging device 21 using the communication device 33 (step S321). Specifically, the imaging device 21 captures an image of the target object OBJ at a predetermined imaging rate. For example, the imaging device 21 may capture an image of the target object OBJ at an imaging rate of capturing an image of the target object OBJ tens to hundreds of times (for example, 30 to 500 times) per second. As a result, the imaging device 21 generates image data IMG at a period corresponding to the predetermined imaging rate. For example, the imaging device 21 may generate tens to hundreds of pieces of image data IMG (for example, 30 to 500 pieces) per second. The control device 3 acquires image data IMG every time the imaging device 21 generates image data IMG. In other words, the control device 3 may acquire tens to hundreds of pieces of image data IMG (for example, 30 to 500 pieces) per second.
  • the imaging device 21 may capture an image of the target object OBJ after the rough movement process is completed.
  • the position and orientation calculation unit 311 acquires image data IMG that is generated by the imaging device 21 capturing an image of the target object OBJ after the rough movement process is completed.
  • the imaging device 21 may capture an image of another object different from the target object OBJ, in addition to the target object OBJ on which the end effector 5 performs a predetermined process. For example, when both the target object OBJ and the other object are included in the imaging range (field of view) of the imaging device 21, the imaging device 21 may capture both the target object OBJ and the other object. As a result, the imaging device 21 may generate image data IMG showing an image in which both the target object OBJ and the other object are reflected. Note that since the other object is not a target for the robot 1 to perform a predetermined process, in the following description, the other object different from the target object OBJ is referred to as a non-target object.
  • non-target objects include at least a part of the end effector 5, at least a part of the robot arm 12, and at least one of the peripheral objects that are objects located in the periphery of the robot 1.
  • the imaging device 21 may capture an image of the target object OBJ, but not the non-target object. That is, the imaging device 21 may generate image data IMG showing an image in which the target object OBJ is captured, but no non-target objects are captured. In either case, the imaging device 21 generates image data IMG showing an image in which at least the target object OBJ is captured. That is, the imaging device 21 generates image data IMG that includes at least image data of the target object OBJ.
  • the position and orientation calculation unit 311 Every time the position and orientation calculation unit 311 acquires image data IMG in step S321, the position and orientation calculation unit 311 calculates at least one of the position and orientation of the target object OBJ based on the image data IMG acquired in step S321 (step S322). As a result, the position and orientation calculation unit 311 generates position and orientation information POI that indicates at least one of the position and orientation of the target object OBJ (step S322).
  • step S322 the position and orientation calculation unit 311 calculates at least one of the position and orientation of the target object OBJ in the global coordinate system.
  • the position and orientation calculation unit 311 generates position and orientation information POI that indicates at least one of the position and orientation of the target object OBJ in the global coordinate system.
  • the global coordinate system is a coordinate system that is the reference for the robot system SYS.
  • the global coordinate system is a coordinate system used to control the robot 1.
  • the robot coordinate system may be used as the global coordinate system.
  • the robot coordinate system may be a three-dimensional coordinate system that is determined based on the robot 1.
  • the robot coordinate system is, for example, a three-dimensional coordinate system set on the base 11 provided on the robot 1.
  • the robot coordinate system is, for example, a three-dimensional coordinate system set on the base 11 that does not move even when the robot 1 moves (i.e., the robot arm 12 moves).
  • the robot coordinate system may be, for example, a three-dimensional coordinate system different from the three-dimensional coordinate system set on the base 11 provided on the robot 1.
  • the robot coordinate system may be a three-dimensional coordinate system set on the tip (i.e., the hand) of the robot arm 12.
  • the robot coordinate system may be a three-dimensional coordinate system set on the end effector 5 attached to the robot arm 12.
  • the robot coordinate system may be a three-dimensional coordinate system set at the tool center point of the end effector 5 attached to the robot arm 12.
  • a coordinate system different from the robot coordinate system may be used as the global coordinate system.
  • the global coordinate system is not limited to the robot coordinate system, and may be a three-dimensional coordinate system with a fixed reference point as the origin. The fixed reference point may be set in the space (e.g., a factory) in which the robot 1 is installed.
  • the control device 3 and the robot control device 4 may each control the robot 1 so that the end effector 5 is positioned at a desired position in the global coordinate system.
  • the global coordinate system is a coordinate system defined by an X-axis, a Y-axis, and a Z-axis that are perpendicular to each other.
  • the X-axis may be an axis along a horizontal plane.
  • the Y-axis may be an axis along a horizontal plane.
  • the Z-axis may be an axis perpendicular to the horizontal plane.
  • the Z-axis may be an axis extending along the direction of gravity.
  • the X-axis, Y-axis, and Z-axis shown in FIG. 2 may be the X-axis, Y-axis, and Z-axis, respectively.
  • the origin of the global coordinate system does not have to be the origin of the X-axis, Y-axis, and Z-axis shown in FIG. 2.
  • the position and orientation calculation unit 311 may calculate, as the position of the target object OBJ in the global coordinate system, at least one of the position Tx of the target object OBJ in the X-axis direction parallel to the X-axis, the position Ty of the target object OBJ in the Y-axis direction parallel to the Y-axis, and the position Tz of the target object OBJ in the Z-axis direction parallel to the Z-axis.
  • the position and orientation calculation unit 311 may calculate, as the orientation of the target object OBJ in the global coordinate system, at least one of the rotation amount Rx of the target object OBJ around the X-axis, the rotation amount Ry of the target object OBJ around the Y-axis, and the rotation amount Rz of the target object OBJ around the Z-axis.
  • the rotation amount Rx of the target object OBJ around the X-axis, the rotation amount Ry of the target object OBJ around the Y-axis, and the rotation amount Rz of the target object OBJ around the Z-axis are equivalent to a parameter representing the orientation of the target object OBJ around the X-axis, a parameter representing the orientation of the target object OBJ around the Y-axis, and a parameter representing the orientation of the target object OBJ around the Z-axis, respectively.
  • the amount of rotation Rx of the target object OBJ around the X-axis, the amount of rotation Ry of the target object OBJ around the Y-axis, and the amount of rotation Rz of the target object OBJ around the Z-axis will be referred to as the orientation Rx of the target object OBJ around the X-axis, the orientation Ry of the target object OBJ around the Y-axis, and the orientation Rz of the target object OBJ around the Z-axis, respectively.
  • orientation Rx of the target object OBJ around the X-axis, the orientation Ry of the target object OBJ around the Y-axis, and the orientation Rz of the target object OBJ around the Z-axis may be considered to indicate the position of the target object OBJ in the rotation direction around the X-axis, the position of the target object OBJ in the rotation direction around the Y-axis, and the position of the target object OBJ in the rotation direction around the Z-axis, respectively.
  • the orientation Rx of the target object OBJ around the X-axis, the orientation Ry of the target object OBJ around the Y-axis, and the orientation Rz of the target object OBJ around the Z-axis may all be considered to be parameters that indicate the position of the target object OBJ.
  • the position and orientation calculation unit 311 may calculate at least one of the position Tx, position Ty, position Tz, orientation Rx, orientation Ry, and orientation Rz as at least one of the position and orientation of the target object OBJ in the global coordinate system.
  • the position and orientation calculation unit 311 may calculate the position and orientation of the target object OBJ by performing a matching process using the image data IMG acquired in step S321. Specifically, the position and orientation calculation unit 311 may calculate the position and orientation of the target object OBJ by performing a matching process using the image data IMG and the model data MDL.
  • the model data MDL is data indicating the object model OBM, which is a model of the target object OBJ.
  • the object model OBM is, for example, a two-dimensional model and includes a two-dimensional image.
  • the model data MDL is data indicating the object model OBM, which is a two-dimensional model having a two-dimensional shape that serves as a reference for the target object OBJ.
  • the model data MDL may be, for example, two-dimensional image data indicating a two-dimensional model of the target object OBJ that is generated by virtually projecting a CAD model (or any three-dimensional model) of the target object OBJ from a plurality of different directions onto a virtual plane perpendicular to each of the plurality of different directions.
  • the model data MDL may be image data representing a two-dimensional image generated by capturing an image of an actual target object OBJ in advance.
  • the actual target object OBJ measured in advance to generate the model data MDL may be a reference or non-defective target object OBJ.
  • model data showing a model of at least a portion of the edges of the target object OBJ may be used as the model data MDL.
  • the position and orientation calculation unit 311 may perform a matching process (in other words, a 2D matching process, a template matching process) on the image indicated by the image data IMG using the object model OBM indicated by the model data MDL as a template. Specifically, the position and orientation calculation unit 311 may perform an object detection process as the matching process to detect a target object OBJ indicated by the object model OBM in the image indicated by the image data IMG. In other words, the position and orientation calculation unit 311 may perform an object detection process as the matching process to detect a target object OBJ in the image indicated by the image data IMG by detecting a similar image portion similar to the object model OBM in the image indicated by the image data IMG.
  • a matching process in other words, a 2D matching process, a template matching process
  • the matching process itself may be the same as an existing matching process.
  • the position and orientation calculation unit 311 may perform matching processing using well-known methods such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed-Upped Robust Feature).
  • SIFT Scale-Invariant Feature Transform
  • SURF Speed-Upped Robust Feature
  • the position and orientation calculation unit 311 may translate, enlarge, reduce, and/or rotate the object model OBM in the imaging coordinate system, which is the coordinate system of the imaging device 21 that generated the image data IMG, so that characteristic locations in the entirety of the object model OBM indicated by the model data MDL approach (e.g., match) characteristic locations in the entirety of the target object OBJ reflected in the image indicated by the image data IMG.
  • the position and orientation calculation unit 311 may change the positional relationship between the coordinate system of the model data MDL and the imaging coordinate system so that characteristic locations in the entirety of the target object OBJ reflected in the image indicated by the image data IMG approach (e.g., match) characteristic locations in the entirety of the object model OBM.
  • the position and orientation calculation unit 311 can identify the positional relationship between the coordinate system of the model data MDL and the imaging coordinate system. Thereafter, the position and orientation calculation unit 311 may calculate the position and orientation of the target object OBJ in the imaging coordinate system from the position and orientation of the object model OBM in the coordinate system of the model data MDL based on the positional relationship between the coordinate system of the model data MDL and the imaging coordinate system.
  • the position and orientation calculation unit 311 may calculate at least one of the positions and orientations of the target object OBJ in the global coordinate system from at least one of the positions and orientations of the target object OBJ in the imaging coordinate system using a transformation matrix for transforming three-dimensional coordinates in either the global coordinate system or the imaging coordinate system into three-dimensional coordinates in the other of the global coordinate system or the imaging coordinate system.
  • the position and orientation calculation unit 311 may calculate a matching similarity between the object model OBM and the image represented by the image data IMG (particularly, the image part to which the object model OBM is fitted) so as to maximize the matching similarity.
  • the image part to which the object model OBM is fitted in the image represented by the image data IMG under the condition that the matching similarity is maximized is detected as a similar image part similar to the object model OBM in the image represented by the image data IMG.
  • the target object OBJ represented by the similar image part is detected.
  • the matching similarity may be considered to be equivalent to the correlation degree indicating the correlation between the object model OBM and the image represented by the image data IMG.
  • the matching similarity may be considered to be a parameter calculated as a detection result of the target object OBJ. In other words, the matching similarity may be considered to be information related to the detection result of the target object OBJ.
  • the correlation degree may also be considered to be an index showing the correlation between the object model OBM and the target object OBJ that appears in the image represented by the image data IMG.
  • the matching similarity may also be called a matching score.
  • the position and orientation calculation unit 311 may select the target object OBJ as a processing target object for which the end effector 5 should actually perform a predetermined processing.
  • the position and orientation calculation unit 311 may not select the target object OBJ as a processing target object for which the end effector 5 should actually perform a predetermined processing.
  • the matching judgment threshold is a threshold used to detect the target object OBJ from the image indicated by the image data IMG by the matching process.
  • the imaging device 21 may capture multiple target objects OBJ.
  • multiple target objects OBJ may appear in the image represented by the image data IMG generated by the imaging device 21.
  • the position and orientation calculation unit 311 may select one of the multiple target objects OBJ reflected in the image represented by the image data IMG as a processing target object on which the end effector 5 should actually perform a predetermined process.
  • the position and orientation calculation unit 311 may select one of the multiple target objects OBJ reflected in the image represented by the image data IMG as a processing target object.
  • the position and orientation calculation unit 311 may select one of the multiple target objects OBJ reflected in the image represented by the image data IMG as a processing target object.
  • the control device 3 may move the imaging device 21.
  • the state in which "the matching process does not detect a target object OBJ whose matching similarity is below the matching judgment threshold” may include a state in which the matching similarity of a single target object OBJ detected by the matching process is below the matching judgment threshold.
  • the state in which "the matching process does not detect a target object OBJ whose matching similarity is below the matching judgment threshold” may include a state in which all of the matching similarities of the multiple target objects O detected by the matching process are below the matching judgment threshold.
  • the imaging device 21 may image the target object OBJ again.
  • the imaging device 21 may image the target object OBJ from a position different from the position of the imaging device 21 before the imaging device 21 moves.
  • the imaging device 21 may image the target object OBJ in a posture different from the posture of the imaging device 21 before the imaging device 21 moves.
  • the control device 3 may select the object to be processed by performing the processes of steps S321 to S322 again. In this case, the way the target object OBJ appears in the image represented by the image data IMG changes as the imaging device 21 moves.
  • the method of calculating at least one of the position and orientation of the target object OBJ is not limited to the above-mentioned matching process using the image data IMG.
  • the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ using other well-known methods of calculating at least one of the position and orientation of the target object OBJ using the image data IMG (i.e., a part of the image data IMG).
  • the method of calculating at least one of the position and orientation of the target object OBJ may be a well-known method of calculating at least one of the position and orientation of the target object OBJ based on the image data IMG (i.e., a part of the image data IMG) without using the model data MDL, or a well-known method of calculating at least one of the position and orientation of the target object OBJ using data other than the model data MDL and the image data IMG (i.e., a part of the image data IMG).
  • the method of calculating at least one of the position and orientation of the target object OBJ may be a method of calculating at least one of the position and orientation of the target object OBJ based on the image data IMG (i.e., a part of the image data IMG) by machine learning or deep learning.
  • a prediction model for outputting at least one of the position and orientation of the target object OBJ when image data IMG (i.e., a part of the image data IMG) is input may be constructed in advance by machine learning or deep learning.
  • the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by inputting the image data IMG (i.e., a part of the image data IMG) into this prediction model.
  • This prediction model may be stored in the position and orientation calculation unit 311. Note that the position and orientation calculation unit 311 may read out this prediction model stored in the storage device 32.
  • the signal generation unit 312 generates a control signal using the position and orientation information POI generated in step S322 (step S323).
  • the signal generating unit 312 may generate a control signal so that the positional relationship between the end effector 5 and the target object OBJ becomes a predetermined target positional relationship.
  • the signal generating unit 312 may generate a control signal for controlling the robot 1 so that the positional relationship between the end effector 5 and the target object OBJ becomes a predetermined target positional relationship.
  • the specified target positional relationship may include a positional relationship in which the end effector 5 is located at a target processing position TPP where the end effector 5 should be located when performing a specified processing on the target object OBJ.
  • the signal generating unit 312 may generate a control signal so that the end effector 5 is located at the target processing position TPP that is determined based on the target object OBJ.
  • the signal generating unit 312 may generate a control signal for controlling the robot 1 so that the end effector 5 is located at the target processing position TPP that is determined based on the target object OBJ.
  • the signal generating unit 312 recognizes at least one of the position and orientation of the target object OBJ in the global coordinate system based on the position and orientation information POI generated in step S322. Furthermore, the signal generating unit 312 recognizes the target processing position TPP determined based on the target object OBJ based on the work information described later (or input from the user specifying the target processing position TPP). In other words, the signal generating unit 312 recognizes the target processing position TPP in a coordinate system based on the target object OBJ.
  • the state in which "the end effector 5 is located at the target processing position TPP” may include a state in which "the position of the end effector 5 (e.g., the position of the tool center point) completely coincides with the target processing position TPP.”
  • the state in which "the end effector 5 is located at the target processing position TPP” may include a state in which "the position of the end effector 5 does not completely coincide with the target processing position TPP, but the amount of deviation between the position of the end effector 5 and the target processing position TPP is small enough to be equal to or less than the upper tolerance threshold for the position of the end effector 5.”
  • the predetermined target positional relationship may be set by the user.
  • the target processing position TPP that defines the predetermined target positional relationship may be set by the user.
  • the allowable upper limit threshold that defines the predetermined target positional relationship i.e., the allowable upper limit value for the position of the end effector 5
  • the predetermined target positional relationship may be automatically set by the control device 3.
  • the target processing position TPP that defines the predetermined target positional relationship may be automatically set by the control device 3.
  • the allowable upper limit threshold that defines the predetermined target positional relationship i.e., the allowable upper limit value for the position of the end effector 5 may be automatically set by the control device 3.
  • the signal generating unit 312 may generate a control signal so that the posture relationship between the end effector 5 and the target object OBJ becomes a predetermined target posture relationship.
  • the signal generating unit 312 may generate a control signal for controlling the robot 1 so that the posture relationship between the end effector 5 and the target object OBJ becomes a predetermined target posture relationship.
  • the specified target posture relationship may include a positional relationship in which the end effector 5 assumes a target processing posture that the end effector 5 should assume when performing a specified process on the target object OBJ.
  • the signal generating unit 312 may generate a control signal so that the end effector 5 assumes a target processing posture determined based on the target object OBJ.
  • the signal generating unit 312 may generate a control signal for controlling the robot 1 so that the end effector 5 assumes a target processing posture determined based on the target object OBJ.
  • the signal generating unit 312 recognizes at least one of the position and orientation of the target object OBJ in the global coordinate system based on the position and orientation information POI generated in step S322. Furthermore, the signal generating unit 312 recognizes the target processing orientation determined based on the target object OBJ, based on the work information described below (or input from the user specifying the target processing orientation). In other words, the signal generating unit 312 recognizes the target processing orientation in a coordinate system based on the target object OBJ. In this case, the signal generating unit 312 may calculate the target processing orientation in the global coordinate system based on at least one of the position and orientation of the target object OBJ in the global coordinate system and the target processing orientation determined based on the target object OBJ.
  • the signal generating unit 312 may convert the target processing orientation in the coordinate system based on the target object OBJ to a target processing orientation in the global coordinate system.
  • both the target processing posture in the coordinate system based on the target object OBJ and the target processing posture in the global coordinate system indicate the posture that the end effector 5 should take when performing a predetermined process on the target object OBJ (i.e., the target processing posture in the broad sense).
  • the signal generating unit 312 may generate a control signal for controlling the robot 1 so that the end effector 5 takes the target processing posture in the global coordinate system.
  • the state where "the end effector 5 takes the target processing posture” may mean a state where "the posture of the end effector 5 matches the target processing posture".
  • the state where "the posture of the end effector 5 matches the target processing posture” may include a state where "the posture of the end effector 5 matches completely with the target processing posture”.
  • the state where "the posture of the end effector 5 matches the target processing posture” may include a state where "the posture of the end effector 5 does not match completely with the target processing posture, but the deviation between the posture of the end effector 5 and the target processing posture is small enough to be equal to or less than the upper tolerance limit for the posture of the end effector 5".
  • the predetermined target attitude relationship may be set by the user.
  • the target processing attitude that defines the predetermined target attitude relationship may be set by the user.
  • the upper tolerance threshold that defines the predetermined target attitude relationship i.e., the upper tolerance value for the attitude of the end effector 5
  • the predetermined target attitude relationship may be automatically set by the control device 3.
  • the target processing attitude that defines the predetermined target attitude relationship may be automatically set by the control device 3.
  • the upper tolerance threshold that defines the predetermined target attitude relationship i.e., the upper tolerance value for the attitude of the end effector 5) may be automatically set by the control device 3.
  • the signal generating unit 312 may use the position and orientation information POI generated in step S322 to generate a control signal to perform feedback control of the robot 1.
  • the signal generating unit 312 may use the position and orientation information POI to generate a control signal to perform feedback control including P (Proportional) control.
  • the signal generating unit 312 may use the position and orientation information POI to generate a control signal to perform feedback control including PI (Proportional-Integral) control.
  • the signal generating unit 312 may use the position and orientation information POI to generate a control signal to perform feedback control including PID (Proportional-Integral-Derivative) control.
  • the fine movement process may be considered to be a process based on closed-loop control.
  • the closed-loop control may be at least one of the above-mentioned P control, PI control, and PID control.
  • the rough movement process which moves the end effector 5 in accordance with the movement instruction information included in the computer program, may be considered to be a process based on open-loop control.
  • the fine movement process and the rough movement process may differ in that the loop control methods are different from each other.
  • the control device 3 needs to calculate the position and orientation information POI.
  • the robot control device 4 does not need to calculate the position and orientation information POI.
  • the computation cost of the fine movement processing that includes the process of calculating the position and orientation information POI is higher than the computation cost of the rough movement processing that does not need to include the process of calculating the position and orientation information POI.
  • the computation cost of the rough movement processing that does not include the process of calculating the position and orientation information POI is lower than the computation cost of the fine movement processing that includes the process of calculating the position and orientation information POI.
  • the fine movement processing and the rough movement processing may differ in that the computation costs required are different from each other.
  • the signal generating unit 312 After generating the control signal, the signal generating unit 312 outputs the control signal generated in step S323 (step S324). For example, the signal generating unit 312 may output the control signal generated in step S323 to the robot control device 4. For example, the signal generating unit 312 may output the control signal generated in step S323 to the robot 1. As a result, the robot 1 moves the end effector 5 in accordance with the control signal generated by the signal generating unit 312.
  • the signal generating unit 312 determines whether a predetermined convergence condition is met (step S33).
  • the predetermined convergence condition may include a position convergence condition whereby the positional relationship between the end effector 5 and the target object OBJ becomes a predetermined target positional relationship.
  • An example of a position convergence condition is a condition whereby the end effector 5 is located at a target processing position TPP that is determined based on the target object OBJ.
  • the predetermined convergence condition may include, in addition to or instead of the position convergence condition, a posture convergence condition whereby the posture relationship between the end effector 5 and the target object OBJ becomes a predetermined target posture relationship.
  • An example of a posture convergence condition is a condition whereby the end effector 5 takes a target processing posture that is determined based on the target object OBJ.
  • the signal generating unit 312 may determine that a predetermined convergence condition is satisfied when both the position convergence condition and the posture convergence condition are satisfied.
  • the signal generating unit 312 may determine that a predetermined convergence condition is not satisfied when at least one of the position convergence condition and the posture convergence condition is not satisfied. However, the signal generating unit 312 may determine that a predetermined convergence condition is satisfied when at least one of the position convergence condition and the posture convergence condition is satisfied.
  • step S33 If it is determined in step S33 that the predetermined convergence condition is not satisfied (step S33: No), the control device 3 performs the fine movement process again (step S32). Specifically, the control device 3 acquires the image data IMG again (step S321 in FIG. 10), recalculates at least one of the position and orientation of the target object OBJ (step S322 in FIG. 10), generates the control signal again (step S323 in FIG. 10), and outputs the control signal again (step S324 in FIG. 10). In other words, the control device 3 may repeat the fine movement process multiple times until the predetermined convergence condition is satisfied. However, if the predetermined convergence condition is satisfied after the fine movement process has been performed once, the control device 3 does not need to repeat the fine movement process multiple times.
  • the robot 1 Since the control device 3 outputs a control signal each time the control device 3 performs fine movement processing, the robot 1 moves the end effector 5 each time the control device 3 performs fine movement processing. Therefore, the robot 1 essentially moves the end effector 5 gradually according to the control signal that is appropriately generated (updated) by the fine movement processing that is performed multiple times. In other words, each time the control device 3 performs fine movement processing, the positional relationship between the end effector 5 and the target object OBJ gradually changes. At this time, the signal generation unit 312 may generate (in this case, update) a control signal so that the positional relationship between the end effector 5 and the target object OBJ gradually approaches a predetermined target positional relationship.
  • the signal generation unit 312 may generate (in this case, update) a control signal so that the posture relationship between the end effector 5 and the target object OBJ gradually approaches a predetermined target posture relationship. As a result, the fine movement processing is performed multiple times, gradually increasing the possibility that a predetermined convergence condition will be established.
  • the proportion of the target object OBJ that occupies the imaging field of view of the imaging device 21 increases, so the shorter the distance between the imaging device 21 and the target object OBJ, the higher the calculation accuracy of at least one of the position and orientation of the target object OBJ.
  • the closer the imaging device 21 is to the target object OBJ the higher the calculation accuracy of at least one of the position and orientation of the target object OBJ.
  • the imaging device 21 approaches the target object OBJ.
  • the control device 3 can calculate at least one of the position and orientation of the target object OBJ with higher accuracy. As a result, if the robot 1 moves in accordance with a control signal generated based on at least one of the position and orientation of the target object OBJ calculated at the second timing, the end effector 5 approaches the target object OBJ more accurately. That is, the end effector 5 approaches the target processing position TPP more accurately.
  • the end effector 5 may not be able to get sufficiently close to the target object OBJ. That is, the end effector 5 may not be able to get sufficiently close to the target processing position TPP (in other words, may not be able to reach it).
  • the control device 3 repeats the fine movement process, thereby improving the calculation accuracy of at least one of the position and orientation of the target object OBJ, and as a result, the end effector 5 can accurately reach the target processing position TPP.
  • the target object OBJ moves while the robot 1 is moving. Therefore, if the fine movement process is not repeated, the robot 1 can move the end effector 5 to approach the target object OBJ before it is moved, but cannot move the end effector 5 to approach the target object OBJ after it is moved.
  • the control device 3 since the control device 3 repeats the fine movement process, the calculation results of at least one of the position and orientation of the moving target object OBJ are successively updated. In other words, the control device 3 can accurately calculate at least one of the current position and current orientation of the moving target object OBJ. Therefore, the robot 1 can move the end effector 5 to approach the target object OBJ after it is moved, according to a control signal generated based on at least one of the current position and current orientation of the moving target object OBJ.
  • the control device 3 may repeat the fine movement process at a period corresponding to the imaging rate of the imaging device 21. For example, if the imaging device 21 images the target object OBJ N times (N is a variable indicating an integer equal to or greater than 1) (e.g., 200 times) per second, the control device 3 may repeat the fine movement process N times per second. In other words, the period in which the imaging device 21 images the target object OBJ and the period in which the control device 3 repeats the fine movement process may be the same. However, the period in which the imaging device 21 images the target object OBJ and the period in which the control device 3 repeats the fine movement process may be different.
  • the control device 3 may repeat the fine movement process M times (M is a variable indicating an integer equal to or greater than 1 that is different from N) (e.g., 100 times) per second.
  • the control device 3 may change the imaging conditions of the imaging device 21 every time the control device 3 performs the fine movement processing.
  • the control device 3 may change the imaging conditions so that the imaging device 21, which has moved by the fine movement processing, can properly image the target object OBJ.
  • One example of the imaging conditions is at least one of the exposure time and the intensity of the illumination light.
  • the control device 3 may perform a predetermined image processing on the image indicated by the image data IMG acquired from the imaging device 21.
  • the control device 3 may change the image processing conditions (hereinafter referred to as "image processing conditions") each time the control device 3 performs fine movement processing.
  • the image processing may include gamma correction processing.
  • the image processing conditions may include gamma correction processing conditions related to the gamma correction processing.
  • the gamma correction processing conditions may include at least one of a condition regarding whether or not to perform gamma correction processing and a condition regarding a gamma value used in the gamma correction processing.
  • the image processing may include HDR (High Dynamic Range) processing.
  • the image processing conditions may include HDR processing conditions related to HDR processing.
  • the HDR processing conditions may include at least one of a condition regarding whether or not to perform HDR processing and a condition regarding the number of images to be synthesized in the HDR processing.
  • the image processing may include noise removal processing.
  • the image processing conditions may include noise removal processing conditions related to the noise removal processing.
  • the noise removal processing conditions may include at least one of the following: a condition regarding whether or not to perform noise removal processing, a condition regarding the type of filter used in the noise removal processing, a condition regarding a combination of at least two filters used in the noise removal processing, and a condition regarding the parameters of the filters used in the noise removal processing.
  • the control device 3 may perform fine movement processing by moving the imaging device 21 (i.e., moving the end effector 5) while maintaining the distance between the end effector 5 and the target object OBJ, so that the imaging device 21 can capture an image of the target object OBJ from a different imaging position and/or from a different imaging direction.
  • moving the imaging device 21 i.e., moving the end effector 5
  • the imaging device 21 can capture an image of the target object OBJ from a different imaging position and/or from a different imaging direction.
  • an obstacle other than the target object OBJ exists between the imaging device 21 and the target object OBJ, there is a possibility that part of the target object OBJ is located in the blind spot of the imaging device 21, and therefore the imaging device 21 may not be able to properly capture an image of the target object OBJ.
  • an example of an obstacle is the above-mentioned surrounding object (i.e., a surrounding object that is an object located in the vicinity of the robot 1).
  • the control device 3 may generate a control signal for controlling the robot 1 so as to move the imaging device 21 to avoid the obstacle (for example, by changing the attitude of the imaging device 21) while maintaining the distance between the end effector 5 and the target object OBJ, thereby reducing the influence of the obstacle on the image captured by the imaging device 21.
  • the imaging device 21 will capture an image of the obstacle in addition to the target object OBJ.
  • control device 3 may calculate at least one of the position and attitude of the obstacle in addition to at least one of the position and attitude of the target object OBJ. After that, the control device 3 may generate a control signal for controlling the robot 1 so as to move the imaging device 21 to avoid the obstacle (for example, by changing the attitude of the imaging device 21) while maintaining the distance between the end effector 5 and the target object OBJ, thereby reducing the influence of the obstacle on the image captured by the imaging device 21.
  • control device 3 may generate a control signal for controlling the robot 1 so as to move the imaging device 21 to avoid the obstacle while allowing the distance between the end effector 5 and the target object OBJ to change, thereby reducing the effect of the obstacle on the image captured by the imaging device 21.
  • control device 3 may generate a control signal for controlling the robot 1 so as to move the imaging device 21 to avoid the obstacle while allowing the distance between the end effector 5 and the target object OBJ to increase, thereby reducing the effect of the obstacle on the image captured by the imaging device 21.
  • control device 3 may generate a control signal for controlling the robot 1 so as to move the imaging device 21 to avoid the obstacle while allowing the distance between the end effector 5 and the target object OBJ to decrease, thereby reducing the effect of the obstacle on the image captured by the imaging device 21.
  • the accuracy of calculation of the position and orientation of the target object OBJ is improved, and the control device 3 can accurately move the end effector 5 so that the end effector 5 approaches the target object OBJ.
  • the control device 3 does not need to receive a signal (e.g., a control permission signal) from the robot control device 4 in order to perform the fine movement process again. In other words, even if the control device 3 does not receive a signal from the robot control device 4, the control device 3 may perform the fine movement process again. Even if the control device 3 does not receive a signal from the robot control device 4, the control device 3 may repeat the fine movement process multiple times. In other words, even if the robot control device 4 does not send a signal to the control device 3, the control device 3 may perform the fine movement process again. Even if the robot control device 4 does not send a signal to the control device 3, the control device 3 may repeat the fine movement process multiple times.
  • a signal e.g., a control permission signal
  • step S33 determines whether the predetermined convergence condition is met. If the result of the determination in step S33 indicates that the predetermined convergence condition is met (step S33: Yes), the control device 3 does not need to perform fine movement processing.
  • the end effector 5 which has approached the target object OBJ through the rough movement processing, will approach even closer to the target object OBJ through the fine movement processing. Therefore, the difference between the position of the end effector 5 at the time when the fine movement processing is completed and the target processing position TPP where the end effector 5 should be when performing a specified processing on the target object OBJ, will be smaller than, for example, the difference between the position of the end effector 5 at the time when the rough movement processing is completed and the target processing position TPP.
  • the fine movement process may be considered to include a process of generating a control signal for controlling the robot 1 so that the difference between the position of the end effector 5 and the target processing position TPP at the time when the fine movement process is completed is smaller than the difference between the position of the end effector 5 and the target processing position TPP at the time when the rough movement process is completed.
  • the fine movement process may be considered to include a process of generating a control signal for controlling the robot 1 so that the distance between the end effector 5 and the target processing position TPP at the time when the fine movement process is completed is shorter than the distance between the end effector 5 and the target processing position TPP at the time when the rough movement process is completed.
  • the fine movement process is performed after the rough movement process, the difference between the posture of the end effector 5 at the time when the fine movement process is completed and the target processing posture that the end effector 5 should take when performing a specified process on the target object OBJ will be smaller than, for example, the difference between the posture of the end effector 5 at the time when the rough movement process is completed and the target processing posture.
  • the fine movement process may be considered to include a process of generating a control signal for controlling the robot 1 so that the difference between the posture of the end effector 5 at the time when the fine movement process is completed and the target processing posture is smaller than the difference between the posture of the end effector 5 at the time when the rough movement process is completed and the target processing posture.
  • the convergence conditions include the position convergence conditions described above, the difference between the positional relationship of the end effector 5 and the target object OBJ at the time when the fine movement process is completed and a predetermined target positional relationship is smaller than, for example, the difference between the positional relationship of the end effector 5 and the target object OBJ at the time when the rough movement process is completed and a predetermined target positional relationship.
  • the fine movement process may be considered to include a process of generating a control signal for controlling the robot 1 so that the difference between the positional relationship of the end effector 5 and the target object OBJ at the time when the fine movement process is completed and a predetermined target positional relationship is smaller than the difference between the positional relationship of the end effector 5 and the target object OBJ at the time when the rough movement process is completed and a predetermined target positional relationship.
  • the convergence conditions include the above-mentioned posture convergence conditions
  • the difference between the posture relationship of the end effector 5 and target object OBJ at the time when the fine movement process is completed and the predetermined target posture relationship is smaller than, for example, the difference between the posture relationship of the end effector 5 and target object OBJ at the time when the rough movement process is completed and the predetermined target posture relationship.
  • the fine movement process may be considered to include a process of generating a control signal for controlling the robot 1 so that the difference between the posture relationship of the end effector 5 and target object OBJ at the time when the fine movement process is completed and the predetermined target posture relationship is smaller than the difference between the posture relationship of the end effector 5 and target object OBJ at the time when the rough movement process is completed and the predetermined target posture relationship.
  • the signal generating unit 312 of the control device 3 controls the end effector 5 to perform a predetermined process on the target object OBJ (step S34). Specifically, the signal generating unit 312 generates a control signal for controlling the end effector 5 to perform a predetermined process on the target object OBJ, and outputs the generated control signal to the robot control device 4 or the end effector 5. As a result, the end effector 5 performs the predetermined process on the target object OBJ.
  • the signal generating unit 411 of the robot control device 4 may control the end effector 5 to perform a predetermined process on the target object OBJ (step S34). Specifically, the signal generating unit 411 may generate a control signal for controlling the end effector 5 to perform a predetermined process on the target object OBJ, and output the generated control signal to the end effector 5. As a result, the end effector 5 performs the predetermined process on the target object OBJ.
  • the control device 3 may perform fine movement processing as necessary. Specifically, the control device 3 may calculate at least one of the position and orientation of the target object OBJ based on the image data IMG generated by the imaging device 21 during the processing period. The control device 3 may then generate a control signal based on at least one of the position and orientation of the target object OBJ. For example, the control device 3 may generate a control signal so that the positional relationship between the end effector 5 and the target object OBJ is maintained at a predetermined target positional relationship. For example, the control device 3 may generate a control signal so that the attitude relationship between the end effector 5 and the target object OBJ is maintained at a predetermined target attitude relationship.
  • the control device 3 may perform fine movement processing even during the processing period.
  • the end effector 5 can perform a predetermined processing on the target object OBJ while following the moving target object OBJ.
  • the end effector 5 can perform a predetermined processing on the target object OBJ in the same way as when the target object OBJ is stationary during the processing period.
  • the state of the end effector 5 may be set to an ON state in which a control signal (particularly, an end effector drive signal) can be input during the period in which the fine movement process is performed.
  • the ON state may mean a state in which, when a control signal (particularly, an end effector drive signal) is input to the end effector 5, the end effector 5 can perform a predetermined process without requiring any process other than the process of inputting the control signal.
  • the state of the end effector 5 does not have to be set to an ON state during the period in which the rough movement process is performed.
  • the state of the end effector 5 may be set to an OFF state in which a control signal (particularly, an end effector drive signal) cannot be input.
  • the off state may mean a state in which, when a control signal (particularly an end effector drive signal) is input to the end effector 5, the end effector 5 cannot perform a predetermined process unless a process other than the process of inputting the control signal (e.g., a startup process) is performed. As a result, it is possible to reduce the standby power consumption of the end effector 5.
  • the control device 3 may further perform fine movement processing as necessary. For example, as described with reference to FIG. 3(c) etc., in the case where fine movement processing is performed to retract the end effector 5 after the end effector 5 has performed a predetermined process on the target object OBJ, the control device 3 may perform fine movement processing to retract the end effector 5 after the end effector 5 has performed a predetermined process on the target object OBJ. However, the control device 3 does not have to perform fine movement processing after the end effector 5 has performed a predetermined process on the target object OBJ.
  • control device 3 may control the end effector 5 again so that the end effector 5 performs the predetermined process on the target object OBJ.
  • control device 3 may alternately repeat an operation of performing the fine movement process until the convergence condition is satisfied and an operation of controlling the end effector 5 after the convergence condition is satisfied.
  • the robot 1 uses the end effector 5 to hold the workpiece W and then performs a series of processes to release the held workpiece W onto the placement device T.
  • control device 3 may perform the fine movement process on the workpiece W that the end effector 5 should hold until the convergence condition for the workpiece W is satisfied. Thereafter, the control device 3 may control the end effector 5 to hold the workpiece W. Thereafter, the control device 3 may perform the fine movement process on the placement device T from which the end effector 5 should release the workpiece W until the convergence condition for the placement device T is satisfied. The control device 3 may then control the end effector 5 to release the held workpiece W onto the placement device T.
  • the control device 3 transmits a completion notification signal to the robot control device 4 at a desired timing after controlling the end effector 5 to perform a predetermined process on the target object OBJ (step S35). For example, the control device 3 transmits a completion notification signal to the robot control device 4 after controlling the end effector 5 to perform a predetermined process on the target object OBJ and after determining that fine movement processing does not need to be performed further (step S35). That is, the control device 3 transmits a completion notification signal to the robot control device 4 after controlling the end effector 5 to perform a predetermined process on the target object OBJ and after completing the fine movement processing (step S35).
  • the completion notification signal may be considered to be a signal notifying the robot control device 4 that the control device 3 has completed the fine movement processing.
  • the completion notification signal may be considered to be a signal notifying the robot control device 4 that the control device 3 has completed control of the end effector 5 to perform a predetermined process on the target object OBJ.
  • the control device 3 may transmit a completion notification signal to the robot control device 4 at a desired timing before the robot control device 4 controls the end effector 5 to perform a predetermined process on the target object OBJ.
  • the control device 3 may transmit a completion notification signal to the robot control device 4 after completing the fine movement process (for example, after it is determined in step S33 that the convergence condition is satisfied).
  • the robot control device 4 After the control device 3 transmits the completion notification signal to the robot control device 4, the robot control device 4 receives the completion notification signal transmitted from the control device 3 (step S43).
  • the robot control device 4 that receives the completion notification signal may start the rough movement process. That is, after the robot control device 4 receives the completion notification signal, the robot control device 4 may start the rough movement process. In other words, after the robot control device 4 receives the completion notification signal, the robot control device 4 may move the end effector 5 by performing the rough movement process.
  • a robot control device 4 that has not received a completion notification signal does not start the rough movement process. That is, before the robot control device 4 receives the completion notification signal, the robot control device 4 does not start the rough movement process. In other words, before the robot control device 4 receives the completion notification signal, the robot control device 4 does not move the end effector 5 by performing the rough movement process.
  • the control device 3 does not perform the fine movement process.
  • the control device 3 may be restricted (e.g., prohibited, the same applies below) from moving the end effector 5 (imaging device 21) by performing the fine movement process.
  • the control device 3 may be restricted from acquiring image data IMG from the imaging device 21.
  • the imaging device 21 may be restricted from capturing an image of the target object OBJ.
  • the control device 3 may be restricted from calculating the position and orientation of the target object OBJ based on the image data IMG.
  • control device 3 may be restricted from generating a control signal. For example, even if the control device 3 is permitted to generate a control signal, the control device 3 may be restricted from outputting the control signal to the robot control device 4 and the robot 1. For example, even if the control device 3 is permitted to generate and output a control signal, the robot control device 4 and the robot 1 may be restricted from accepting the control signal output by the control device 3.
  • the completion notification signal may be considered as a signal that passes (in this case, returns) the control right of the robot 1 from the control device 3 to the robot control device 4.
  • the robot control device 4 may determine whether or not to end the robot control process (step S44).
  • step S44 If it is determined in step S44 that the robot control process is to be terminated (step S44: Yes), the robot control device 4 does not need to perform the rough movement process again. In this case, the control device 3 and the robot control device 4 may terminate the robot control process.
  • step S44 if the result of the judgment in step S44 is that the robot control process is not to be terminated (step S44: No), the robot control device 4 may perform the rough movement process again (step S41). Thereafter, the control device 3 may perform the fine movement process (step S32). In other words, the robot system SYS may alternately repeat the rough movement process and the fine movement process.
  • Robot control process in which rough movement process and fine movement process are alternately repeated 2-5-1) First Example of Robot Control Processing for Alternately Repeating Rough Movement Processing and Fine Movement Processing
  • a robot control processing performed when the end effector 5 performs predetermined processing in sequence on a first target portion for example, a portion in which a hole HL#1 shown in FIG. 5 and FIG.
  • the target object OBJ and a second target portion (for example, a portion in which a hole HL#2 shown in FIG. 5 and FIG. 6 is formed) of the target object OBJ different from the first target portion will be described.
  • the following robot control processing may be performed in which the rough movement processing and the fine movement processing are alternately repeated as many times as the number of target portions.
  • the robot control device 4 may first perform a rough movement process on the first target portion (step S41 in FIG. 9). For example, the robot control device 4 may generate and output a control signal for controlling the robot 1 so that the end effector 5 approaches the first target portion. As a result, the end effector 5 moves under the control of the robot control device 4.
  • the robot control device 4 transmits a control permission signal to the control device 3 (step S42 in FIG. 9), and the control device 3 receives a control permission signal from the robot control device 4 (step S31 in FIG. 9).
  • the control device 3 may then perform fine movement process for the first target portion (step S32 in FIG. 9).
  • the control device 3 may acquire image data IMG generated by the imaging device 21 capturing an image of the target object OBJ (particularly, the first target portion) after the rough movement process for the first target portion.
  • the control device 3 may then calculate at least one of the position and orientation of the target object OBJ (particularly, at least one of the position and orientation of the first target portion of the target object OBJ) based on the acquired image data IMG.
  • the control device 3 may then generate and output a control signal for controlling the robot 1 to move the end effector 5 based on the calculation result of at least one of the position and orientation of the first target portion. As a result, the end effector 5 moves under the control of the control device 3.
  • the control device 3 may perform fine movement processing on the first target part until the above-mentioned predetermined convergence condition is met (step S33 in FIG. 9). In other words, the control device 3 may repeat fine movement processing on the first target part until the above-mentioned predetermined convergence condition is met. In particular, the control device 3 may repeat fine movement processing on the first target part until the above-mentioned predetermined convergence condition is met during the period from when the rough movement processing on the first target part is completed to when the rough movement processing on the second target part is started.
  • the control device 3 may perform fine movement processing on the first target portion until a position convergence condition is met in which the positional relationship between the end effector 5 and the first target portion becomes a first target positional relationship.
  • An example of the first target positional relationship is a positional relationship in which the end effector 5 is located at a first target processing position TPP where the end effector 5 should be located when the end effector 5 performs a predetermined processing on the first target portion.
  • the control device 3 may perform fine movement processing on the first target portion until a posture convergence condition is met in which the posture relationship between the end effector 5 and the first target portion becomes a first target posture relationship.
  • An example of the first target posture relationship is a positional relationship in which the end effector 5 takes a first target processing posture where the end effector 5 should be located when the end effector 5 performs a predetermined processing on the first target portion.
  • step S33 in FIG. 9: Yes the control device 3 ends the fine movement process for the first target part.
  • the control device 3 controls the end effector 5 to perform a predetermined process for the first target part (step S34 in FIG. 9).
  • the control device 3 transmits a completion notification signal to the robot control device 4 (step S35 in FIG. 9), and the robot control device 4 receives the completion notification signal transmitted from the control device 3 (step S43 in FIG. 9).
  • the robot control device 4 may then perform a rough movement process for the second target portion (step S41 in FIG. 9).
  • the robot control device 4 may generate and output a control signal for controlling the robot 1 so that the end effector 5 approaches the second target portion from the first target portion. As a result, the end effector 5 moves under the control of the robot control device 4.
  • the control device 3 may perform the rough movement process in addition to or instead of the robot control device 4. Therefore, after the fine movement process for the first target portion, the control device 3 may perform the rough movement process for the second target portion. In this case, the control device 3 does not need to receive a signal from the robot control device 4 to perform the rough movement process for the second target portion. In other words, even if the control device 3 does not receive a signal from the robot control device 4, the control device 3 may perform the rough movement process for the second target portion. In other words, even if the robot control device 4 does not send a signal to the control device 3, the control device 3 may perform the rough movement process for the second target portion.
  • the control device 3 may start the rough movement process for the second target portion without sending a completion notification signal to the robot control device 4 in step S35 of FIG. 9.
  • the time required to perform the rough movement process for the second target portion is shortened compared to the case where the control device 3 needs to receive a signal from the robot control device 4 to perform the rough movement process for the second target portion. This is because the control device 3 does not need to wait to start the rough movement process for the second target portion until it receives some signal from the robot control device 4.
  • the robot control device 4 sends a control permission signal to the control device 3 (step S42 in FIG. 9), and the control device 3 receives a control permission signal from the robot control device 4 (step S31 in FIG. 9). Thereafter, the control device 3 may start rough movement processing on the second target portion.
  • the control device 3 may perform rough movement processing on the first target portion in addition to or instead of the rough movement processing on the second target portion.
  • the operation of the control device 3 performing rough movement processing on the first target portion may be the same as the operation of the control device 3 performing rough movement processing on the second target portion, so a detailed description thereof will be omitted in order to avoid redundant description.
  • the robot control device 4 transmits a control permission signal to the control device 3 (step S42 in FIG. 9), and the control device 3 receives a control permission signal from the robot control device 4 (step S31 in FIG. 9).
  • the control device 3 may then perform fine movement process for the second target portion (step S32 in FIG. 9).
  • the control device 3 may acquire image data IMG generated by the imaging device 21 capturing an image of the target object OBJ (particularly, the second target portion) after the rough movement process for the second target portion.
  • the control device 3 may then calculate at least one of the position and orientation of the target object OBJ (particularly, at least one of the position and orientation of the second target portion of the target object OBJ) based on the acquired image data IMG.
  • the control device 3 may then generate and output a control signal for controlling the robot 1 to move the end effector 5 based on the calculation result of at least one of the position and orientation of the second target portion. As a result, the end effector 5 moves under the control of the control device 3.
  • the control device 3 may perform fine movement processing on the second target portion until the above-mentioned predetermined convergence condition is satisfied (step S33 in FIG. 9). In other words, the control device 3 may repeat fine movement processing on the second target portion until the above-mentioned predetermined convergence condition is satisfied.
  • the control device 3 may perform fine movement processing on the second target portion until a position convergence condition is met in which the positional relationship between the end effector 5 and the second target portion becomes a second target positional relationship.
  • An example of the second target positional relationship is a positional relationship in which the end effector 5 is located at a second target processing position TPP where the end effector 5 should be located when the end effector 5 performs a predetermined processing on the second target portion.
  • the control device 3 may perform fine movement processing on the second target portion until a posture convergence condition is met in which the posture relationship between the end effector 5 and the second target portion becomes a second target posture relationship.
  • An example of the second target posture relationship is a positional relationship in which the end effector 5 takes the second target processing posture where the end effector 5 should be located when the end effector 5 performs a predetermined processing on the second target portion.
  • the second target position relationship may be the same as or different from the first target position relationship described above.
  • the second target processing position TPP may be the same as or different from the first target processing position TPP described above.
  • the second target attitude relationship may be the same as or different from the first target attitude relationship described above.
  • the second target processing attitude may be the same as or different from the first target processing attitude described above.
  • step S33 in FIG. 9 Yes
  • the control device 3 ends the fine movement process for the second target portion.
  • the control device 3 controls the end effector 5 to perform a predetermined process for the second target portion (step S34 in FIG. 9).
  • the control device 3 transmits a completion notification signal to the robot control device 4 (step S35 in FIG. 9), and the robot control device 4 receives the completion notification signal transmitted from the control device 3 (step S43 in FIG. 9).
  • a second example of a robot control processing in which rough movement processing and fine movement processing are repeated alternately is a robot control processing performed when the end effector 5 performs predetermined processing on a plurality of target objects OBJ in sequence.
  • a robot control processing performed when the end effector 5 performs predetermined processing on a first target object OBJ and a second target object OBJ different from the first target object OBJ in sequence will be described.
  • the following robot control processing may be performed in which rough movement processing and fine movement processing are alternately repeated the number of times equal to the number of target objects OBJ.
  • the robot control device 4 may first perform a rough movement process on the first target object OBJ (step S41 in FIG. 9). For example, the robot control device 4 may generate and output a control signal for controlling the robot 1 so that the end effector 5 approaches the first target object OBJ. As a result, the end effector 5 moves under the control of the robot control device 4.
  • the robot control device 4 transmits a control permission signal to the control device 3 (step S42 in FIG. 9), and the control device 3 receives the control permission signal from the robot control device 4 (step S31 in FIG. 9).
  • the control device 3 may then perform a fine movement process for the first target object OBJ (step S32 in FIG. 9).
  • the control device 3 may acquire image data IMG generated by the imaging device 21 capturing an image of the first target object OBJ after the rough movement process for the first target object OBJ.
  • the control device 3 may then calculate at least one of the position and orientation of the first target object OBJ based on the acquired image data IMG.
  • the control device 3 may then generate and output a control signal for controlling the robot 1 to move the end effector 5 based on the calculation result of at least one of the position and orientation of the first target object OBJ. As a result, the end effector 5 moves under the control of the control device 3.
  • the control device 3 may perform fine movement processing on the first target object OBJ until the above-mentioned predetermined convergence condition is met (step S33 in FIG. 9). In other words, the control device 3 may repeat the fine movement processing on the first target object OBJ until the above-mentioned predetermined convergence condition is met. In particular, the control device 3 may repeat the fine movement processing on the first target object OBJ until the above-mentioned predetermined convergence condition is met during the period from when the rough movement processing on the first target object OBJ is completed to when the rough movement processing on the second target object OBJ is started.
  • the control device 3 may perform fine movement processing on the first target object OBJ until a position convergence condition is met in which the positional relationship between the end effector 5 and the first target object OBJ becomes a third target positional relationship.
  • An example of the third target positional relationship is a positional relationship in which the end effector 5 is located at a third target processing position TPP where the end effector 5 should be located when the end effector 5 performs a predetermined process on the first target object OBJ.
  • the control device 3 may perform fine movement processing on the first target object OBJ until a posture convergence condition is met in which the posture relationship between the end effector 5 and the first target object OBJ becomes a third target posture relationship.
  • An example of the third target posture relationship is a positional relationship in which the end effector 5 takes the third target processing posture where the end effector 5 should be located when the end effector 5 performs a predetermined process on the first target object OBJ.
  • the third target position relationship may be the same as or different from at least one of the first and second target position relationships described above.
  • the third target processing position TPP may be the same as or different from at least one of the first and second target processing positions TPP described above.
  • the third target attitude relationship may be the same as or different from at least one of the first and second target attitude relationships described above.
  • the third target processing attitude may be the same as or different from at least one of the first and second target processing attitudes described above.
  • step S33 in FIG. 9: Yes the control device 3 ends the fine movement process for the first target object OBJ.
  • the control device 3 controls the end effector 5 to perform a predetermined process for the first target object OBJ (step S34 in FIG. 9).
  • the control device 3 transmits a completion notification signal to the robot control device 4 (step S35 in FIG. 9), and the robot control device 4 receives the completion notification signal transmitted from the control device 3 (step S43 in FIG. 9).
  • the robot control device 4 may then perform a rough movement process on the second target object OBJ (step S41 in FIG. 9).
  • the robot control device 4 may generate and output a control signal for controlling the robot 1 so that the end effector 5 moves from the first target object OBJ toward the second target object OBJ. As a result, the end effector 5 moves under the control of the robot control device 4.
  • the control device 3 may perform the rough movement process in addition to or instead of the robot control device 4. Therefore, after the fine movement process for the first target object OBJ, the control device 3 may perform the rough movement process for the second target object OBJ. In this case, the control device 3 does not need to receive a signal from the robot control device 4 in order to perform the rough movement process for the second target object OBJ. In other words, even if the control device 3 does not receive a signal from the robot control device 4, the control device 3 may perform the rough movement process for the second target object OBJ. In other words, even if the robot control device 4 does not transmit a signal to the control device 3, the control device 3 may perform the rough movement process for the second target object OBJ.
  • the control device 3 may start the rough movement process for the second target object OBJ without transmitting a completion notification signal to the robot control device 4 in step S35 of FIG. 9.
  • the time required to perform the rough movement process for the second target object OBJ is shortened compared to the case where the control device 3 needs to receive a signal from the robot control device 4 in order to perform the rough movement process for the second target object OBJ. This is because the control device 3 does not need to wait to start the rough movement process for the second target object OBJ until it receives a signal from the robot control device 4.
  • the robot control device 4 transmits a control permission signal to the control device 3 (step S42 in FIG. 9), and the control device 3 receives a control permission signal from the robot control device 4 (step S31 in FIG. 9). Thereafter, the control device 3 may start the rough movement processing on the second target object OBJ.
  • the control device 3 may perform rough movement processing on the first target object OBJ in addition to or instead of the rough movement processing on the second target object OBJ.
  • the operation of the control device 3 performing the rough movement processing on the first target object OBJ may be the same as the operation of the control device 3 performing the rough movement processing on the second target object OBJ, and therefore a detailed description thereof will be omitted in order to avoid redundant description.
  • the robot control device 4 transmits a control permission signal to the control device 3 (step S42 in FIG. 9), and the control device 3 receives the control permission signal from the robot control device 4 (step S31 in FIG. 9).
  • the control device 3 may then perform fine movement process for the second target object OBJ (step S32 in FIG. 9).
  • the control device 3 may acquire image data IMG generated by the imaging device 21 capturing an image of the second target object OBJ after the rough movement process for the second target object OBJ.
  • the control device 3 may then calculate at least one of the position and orientation of the second target object OBJ based on the acquired image data IMG.
  • the control device 3 may then generate and output a control signal for controlling the robot 1 to move the end effector 5 based on the calculation result of at least one of the position and orientation of the second target object OBJ. As a result, the end effector 5 moves under the control of the control device 3.
  • the control device 3 may perform fine movement processing on the second target object OBJ until the above-mentioned predetermined convergence condition is satisfied (step S33 in FIG. 9). In other words, the control device 3 may repeat the fine movement processing on the second target object OBJ until the above-mentioned predetermined convergence condition is satisfied.
  • the control device 3 may perform fine movement processing on the second target object OBJ until a position convergence condition is met in which the positional relationship between the end effector 5 and the second target object OBJ becomes a fourth target positional relationship.
  • An example of the fourth target positional relationship is a positional relationship in which the end effector 5 is located at a fourth target processing position TPP where the end effector 5 should be located when the end effector 5 performs a predetermined process on the second target object OBJ.
  • the control device 3 may perform fine movement processing on the second target object OBJ until a posture convergence condition is met in which the posture relationship between the end effector 5 and the second target object OBJ becomes a fourth target posture relationship.
  • An example of the fourth target posture relationship is a positional relationship in which the end effector 5 takes a fourth target processing posture where the end effector 5 should be located when the end effector 5 performs a predetermined process on the second target object OBJ.
  • the fourth target position relationship may be the same as or different from at least one of the first to third target position relationships described above.
  • the fourth target processing position TPP may be the same as or different from at least one of the first to third target processing positions TPP described above.
  • the fourth target attitude relationship may be the same as or different from at least one of the first to third target attitude relationships described above.
  • the fourth target processing attitude may be the same as or different from at least one of the first to third target processing attitudes described above.
  • step S33 in FIG. 9 Yes
  • the control device 3 ends the fine movement process for the second target object OBJ.
  • the control device 3 controls the end effector 5 to perform a predetermined process for the second target object OBJ (step S34 in FIG. 9).
  • the control device 3 transmits a completion notification signal to the robot control device 4 (step S35 in FIG. 9), and the robot control device 4 receives the completion notification signal transmitted from the control device 3 (step S43 in FIG. 9).
  • the robot system SYS can perform robot control processing including rough movement processing and fine movement processing.
  • the robot system SYS can move the end effector 5 efficiently and with high accuracy, compared to the case where a robot control processing including only the rough movement processing or only the fine movement processing is performed.
  • the rough movement process is a process for roughly moving the end effector 5 without using image data IMG as described above.
  • a robot control process including only the rough movement process may not include a process for calculating at least one of the position and the orientation of the target object OBJ based on the image data IMG.
  • the time required to move the end effector 5 is shortened, but the difference between the position of the end effector 5 and the above-mentioned target processing position TPP may become so large that it exceeds the allowable amount.
  • the difference between the orientation of the end effector 5 and the above-mentioned target processing orientation may become so large that it exceeds the allowable amount.
  • the fine movement process is a process for moving the end effector 5 with high accuracy using the calculation result of at least one of the position and the orientation of the target object OBJ based on the image data IMG as described above.
  • the time required to move the end effector 5 by only the fine movement process is longer than the time required to move the end effector 5 by only the rough movement process.
  • the robot system SYS moves the end effector 5 with high accuracy by performing the rough movement process after roughly moving the end effector 5. Therefore, the time required to move the end effector 5 is reduced compared to when the robot control process including only the fine movement process is performed. In other words, the robot system SYS can move the end effector 5 efficiently. Furthermore, compared to when the robot control process including only the rough movement process is performed, the robot system SYS can move the end effector 5 with high accuracy.
  • the control device 3 may repeat the fine movement process multiple times.
  • the end effector 5 imaging device 21
  • the calculation accuracy of at least one of the position and orientation of the target object OBJ gradually increases.
  • the movement accuracy of the end effector 5, which moves according to a control signal generated based on the calculation result of at least one of the position and orientation of the target object OBJ also increases.
  • the robot system SYS can move the end effector 5 with high accuracy. In other words, the robot system SYS can move the end effector 5 with high accuracy so that it can accurately reach the target processing position TPP.
  • a control permission signal is sent from the robot control device 4 to the control device 3.
  • the control device 3 starts the fine movement processing.
  • the control device 3 can perform the fine movement processing without receiving any signal from the robot control device.
  • the control device 3 can perform the fine movement processing multiple times without receiving any signal from the robot control device 4.
  • the control device 3 can perform the fine movement processing even if it does not receive any signal from the robot control device 4.
  • control device 3 can perform the fine movement processing multiple times even if it does not receive any signal from the robot control device 4.
  • the control device 3 can repeat the fine movement processing even if it does not receive any signal (e.g., a control permission signal) for each fine movement processing. Therefore, the time required to perform the fine movement processing is reduced compared to a comparative example in which the control device 3 does not generate a control signal based on at least one of the calculation results of the position and orientation of the target object OBJ by the control device 3, but the robot control device 4 generates a control signal based on at least one of the calculation results of the position and orientation of the target object OBJ by the control device 3, and moves the end effector 5 based on the generated control signal.
  • the control device 3 does not generate a control signal based on at least one of the calculation results of the position and orientation of the target object OBJ by the control device 3, but the robot control device 4 generates a control signal based on at least one of the calculation results of the position and orientation of the target object OBJ
  • the time required to perform the fine movement processing is reduced compared to a comparative example in which part of the fine movement processing is performed by the control device 3 and another part of the fine movement processing is performed by the robot control device 4.
  • the time required to perform the fine movement processing multiple times is reduced.
  • the robot controller 4 In a comparative example in which the robot controller 4 generates a control signal based on at least one of the calculation results of the position and orientation of the target object OBJ by the controller 3 without the controller 3 generating a control signal based on at least one of the calculation results of the position and orientation of the target object OBJ by the controller 3, and moves the end effector 5 based on the generated control signal, it is preferable that the process of moving the end effector 5 by the robot controller 4 and the process of calculating at least one of the position and orientation of the target object OBJ by the controller 3 are performed synchronously with each other.
  • the controller 3 that has calculated at least one of the position and orientation of the target object OBJ is not permitted to recalculate at least one of the position and orientation of the target object OBJ unless it has received a signal transmitted by the robot controller 4 to the controller 3 as a substantial synchronization signal.
  • An example of a signal transmitted by the robot controller 4 to the controller 3 is a signal for notifying the controller 3 that the robot controller 4 has moved the end effector 5 based on at least one of the calculation results of the position and orientation of the target object OBJ by the controller 3.
  • the control device 3 needs to wait without performing a process of calculating at least one of the position and the orientation of the target object OBJ again until it receives a signal transmitted from the robot control device 4 to the control device 3.
  • both the process of calculating at least one of the position and the orientation of the target object OBJ and the process of moving the end effector 5 based on the calculation result of at least one of the position and the orientation of the target object OBJ (specifically, the process of generating a control signal based on the calculation result of at least one of the position and the orientation of the target object OBJ) are performed by the same control device 3.
  • the control device 3 even if the control device 3 does not receive a signal transmitted from the robot control device 4 to the control device 3, it can synchronously perform the process of calculating at least one of the position and the orientation of the target object OBJ and the process of moving the end effector 5 based on the calculation result of at least one of the position and the orientation of the target object OBJ (specifically, the process of generating a control signal based on the calculation result of at least one of the position and the orientation of the target object OBJ). Therefore, in this embodiment, the control device 3 does not need to wait until it receives a signal sent from the robot control device 4 to the control device 3, and the time required to perform the fine movement process (particularly the time required to perform the fine movement process multiple times) is reduced.
  • the effect of reducing the time required to perform the fine movement process multiple times becomes greater the more times the fine movement process is performed. For example, if the number of times the fine movement is repeated is N (N is an integer equal to or greater than 2), and the time required from when the control device 3 transmits the calculation results of at least one of the position and orientation of the target object OBJ to the robot control device 4 in the comparative example until the control device 3 receives the signal transmitted from the robot control device 4 to the control device 3 (i.e., the time the control device 3 waits) is T, then the time required to perform the fine movement process N times in this embodiment is shorter by T x N compared to the time required to perform the fine movement process N times in the comparative example. Therefore, the effect of reducing the time required to perform the fine movement process multiple times in this embodiment becomes greater the more times the fine movement process is performed.
  • the control device 3 may also perform the rough movement process after performing the fine movement process.
  • the control device 3 may perform the fine movement process for the first part of the target object OBJ, and then perform the rough movement process for the second part of the target object OBJ.
  • the control device 3 may perform the fine movement process for the first target object OBJ, and then perform the rough movement process for the second target object OBJ.
  • the time required to perform the rough movement process for the second target part or the second target object is particularly shortened. This is because the control device 3 can start the rough movement process for the second target part or the second target object OBJ even if the control device 3 does not receive a signal transmitted from the robot control device 4 to the control device 3 after performing the fine movement process for the first target part or the first target object OBJ.
  • the robot system SYS of this embodiment can efficiently move the end effector 5, also in that it can reduce the time required to perform at least one of the rough movement process and the fine movement process.
  • the control device 3 may perform preparation processing for performing fine movement processing before the robot control device 4 completes (i.e., ends) the rough movement processing (step S30a). Specifically, the control device 3 may perform preparation processing while the robot control device 4 is performing the rough movement processing (step S30a). The control device 3 may perform preparation processing before the control device 3 receives a control permission signal from the robot control device 4 (step S30a). The control device 3 may perform preparation processing before the control device 3 starts the fine movement processing (step S30a).
  • the preparation process may include pre-processing that is performed to start the fine movement process.
  • the preparation process may include pre-processing that the control device 3 can perform in advance even if the control device 3 has not received a control permission signal from the robot control device 4, in order to start the fine movement process after the control device 3 receives a control permission signal.
  • the time required from when the robot control device 4 completes the rough movement processing until the control device 3 starts the fine movement processing is shortened compared to when the control device 3 does not perform preparatory processing before the robot control device 4 completes the rough movement processing.
  • the work information may include target object information, which is information about a target object OBJ on which the end effector 5 performs a predetermined process.
  • the target object information may include information about the type of the target object OBJ.
  • the work information may include surrounding object information, which is information about surrounding objects located around a target object OBJ on which the end effector 5 performs a specified process.
  • the surrounding object information may include information about at least one of the position, orientation, shape, and size of the surrounding object. Such surrounding object information may be used to generate a control signal for controlling the robot 1 to move the end effector 5 while avoiding interference of the end effector 5 with the surrounding objects indicated by the surrounding object information.
  • the work information may include matching information used in a matching process for detecting the target object OBJ.
  • the matching information may include at least one of information on the model data MDL, information on a matching judgment threshold, and information on a convergence condition.
  • the matching information may include at least one of information specifying the model data MDL, information specifying the matching judgment threshold, and information specifying the convergence condition.
  • the information on the convergence condition may include at least one of information specifying a target position relationship, information specifying a target posture relationship, information specifying a target processing position TPP, information specifying a target processing posture, an upper tolerance threshold that determines the target position relationship (i.e., an upper tolerance upper limit value for the position of the end effector 5), and an upper tolerance threshold that determines the target posture relationship (i.e., an upper tolerance upper limit value for the posture of the end effector 5).
  • the work information may include imaging condition information, which is information about the imaging conditions of the imaging device 21.
  • the work information may include imaging condition information, which is information that specifies the imaging conditions of the imaging device 21.
  • the imaging condition information may include at least one of information about the exposure time and information about the intensity of the illumination light.
  • the imaging condition information may include at least one of information that specifies the exposure time and information that specifies the intensity of the illumination light.
  • the work information may include image processing condition information, which is information on image processing conditions.
  • the work information may include image processing condition information, which is information specifying image processing conditions.
  • the image processing condition information may include at least one of information on gamma correction processing conditions, information on HDR processing conditions, and information on noise removal processing conditions.
  • the image processing condition information may include at least one of information specifying gamma correction processing conditions, information specifying HDR processing conditions, and information specifying noise removal processing conditions.
  • the control device 3 may read the work information from the storage device 32 provided in the control device 3.
  • the control device 3 may read the work information from a recording medium that can be attached externally to the control device 3 or that is built into the control device 3.
  • the control device 3 may read the work information from a device (not shown) that is located outside the control device 3 (in this case, the work information may be downloaded).
  • the control device 3 reads the work information before the robot control device 4 completes the rough movement process.
  • a certain amount of time is required to read the work information. Therefore, if the control device 3 has completed reading the work information before the robot control device 4 completes the rough movement process, the control device 3 does not need to read the work information after the robot control device 4 completes the rough movement process.
  • the amount of work information that the control device 3 needs to read after the robot control device 4 completes the rough movement process is reduced compared to when the work information is not read before the robot control device 4 completes the rough movement process. Therefore, the time required from when the robot control device 4 completes the rough movement process to when it actually starts the fine movement process (in other words, the time required for the control device 3 to complete the fine movement process) is shortened.
  • the preparation processing may include processing for starting auto exposure processing related to the auto exposure function.
  • the preparation processing may include processing for adjusting the exposure conditions in advance by starting the auto exposure processing.
  • An example of the exposure conditions is at least one of the aperture value and the shutter speed.
  • the control device 3 starts the automatic exposure process before the robot control device 4 completes the rough movement process.
  • a certain amount of time is required from the start of the automatic exposure process until the adjustment of the exposure conditions is actually completed. For this reason, if the control device 3 starts the automatic exposure process and completes the adjustment of the exposure conditions before the robot control device 4 completes the rough movement process, the control device 3 does not need to start the automatic exposure process and adjust the exposure conditions after the robot control device 4 completes the rough movement process.
  • the time required from the completion of the rough movement process by the robot control device 4 to the completion of the adjustment of the exposure conditions is shortened compared to the case where the automatic exposure process is not started before the robot control device 4 completes the rough movement process, and as a result, the time that the control device 3 waits until the adjustment of the exposure conditions is completed after the robot control device 4 completes the rough movement process is shortened. This reduces the time required from when the robot control device 4 completes the rough movement process to when it actually starts the fine movement process using the imaging device 21 for which exposure condition adjustment has been completed (in other words, the time required for the control device 3 to complete the fine movement process).
  • the preparation processing may include a process of controlling the illumination device 23 to start irradiating the target object OBJ with illumination light.
  • the preparation processing may include a process of controlling the illumination device 23 to start emitting illumination light.
  • the lighting device 23 starts emitting the lighting light before the robot control device 4 completes the rough movement process.
  • a certain amount of time is required from when the lighting device 23 starts emitting the lighting light until the intensity of the lighting light stabilizes (specifically, the intensity of the lighting light becomes a constant intensity). Therefore, if the intensity of the lighting light stabilizes before the robot control device 4 completes the rough movement process, the control device 3 does not need to wait until the intensity of the lighting light stabilizes.
  • the time required for the intensity of the lighting light to stabilize after the robot control device 4 completes the rough movement process is shortened compared to when the lighting device 23 is not controlled before the robot control device 4 completes the rough movement process, and as a result, the time that the control device 3 waits until the intensity of the lighting light stabilizes after the robot control device 4 completes the rough movement process is shortened. Therefore, the time required from when the robot control device 4 completes the rough movement process until the fine movement process is actually started using the lighting device 23 capable of emitting lighting light of a stable intensity is shortened.
  • the control device 3 may perform preparatory processing after the robot control device 4 has completed the rough movement processing and before the control device 3 receives a control permission signal transmitted by the robot control device 4. Even in this case, the time required from receiving the control permission signal to starting the fine movement processing is shortened compared to the case in which the control device 3 does not perform preparatory processing before receiving the control permission signal.
  • control device 3 may perform the preparatory process before the control device 3 completes the rough movement process. Even in this case, the time required from when the control device 3 completes the rough movement process until when the control device 3 starts the fine movement process is reduced, compared to when the control device 3 does not perform the preparatory process before completing the rough movement process.
  • the control device 3 may perform preparatory processing before starting fine movement processing. In other words, if the control device 3 starts fine movement processing without the robot control device 4 performing rough movement processing, the control device 3 may perform preparatory processing before starting fine movement processing. Even in this case, the time required from when the control device 3 completes the fine movement processing to when it starts the fine movement processing is reduced, compared to when the control device 3 does not perform preparatory processing before starting the fine movement processing.
  • the control device 3 may, in addition to or instead of performing the preparatory process, perform at least a part of the fine movement process in advance before the robot control device 4 completes (i.e., ends) the rough movement process.
  • the control device 3 may, in addition to or instead of performing the preparatory process, perform at least a part of the fine movement process in advance before the control device 3 receives a control permission signal from the robot control device 4.
  • the control device 3 may, in addition to or instead of performing the preparatory process, perform at least a part of the fine movement process in advance during the period in which the robot control device 4 is performing the rough movement process.
  • At least a part of the rough movement process and at least a part of the fine movement process may be performed in parallel.
  • the time required to complete a series of robot control processes including the rough movement process and the fine movement process is shortened compared to the case in which the control device 3 does not perform the fine movement process before the robot control device 4 completes the rough movement process.
  • the control device 3 may perform a process of acquiring image data IMG generated by the imaging device 21 from the imaging device 21 as at least part of the fine movement process before the robot control device 4 completes the rough movement process.
  • the image data IMG may be considered to be information used to move the end effector 5 by the fine movement process.
  • the control device 3 may consider the process of acquiring image data IMG generated by the imaging device 21 from the imaging device 21 before the robot control device 4 completes the rough movement process to be at least part of the above-mentioned pre-processing.
  • the control device 3 may perform a process in which the imaging device 21 generates image data by imaging the target object OBJ as at least a part of the fine movement process before the robot control device 4 completes the rough movement process.
  • the imaging device 21 may perform a process in which the imaging device 21 generates image data by imaging the target object OBJ as at least a part of the fine movement process before the robot control device 4 completes the rough movement process.
  • the control device 3 may perform a process of calculating at least one of the position and orientation of the target object OBJ based on the image data IMG as at least a part of the fine movement process before the robot control device 4 completes the rough movement process.
  • the control device 3 may perform a process of calculating at least one of the position and orientation of the target object OBJ based on the image data IMG generated by the imaging device 21 capturing an image of the target object OBJ before the robot control device 4 completes the rough movement process.
  • the calculation result of at least one of the position and orientation of the target object OBJ may be considered to be information used to move the end effector 5 by the fine movement process.
  • control device 3 may consider the process of obtaining the calculation result of at least one of the position and orientation of the target object OBJ by calculating at least one of the position and orientation of the target object OBJ before the robot control device 4 completes the rough movement process to be performed as at least a part of the above-mentioned pre-processing.
  • the shorter the time difference between the imaging time when the imaging device 21 images the target object OBJ and the rough movement completion time when the robot control device 4 completes the rough movement processing the smaller the difference between the position of the target object OBJ at the imaging time and the position of the target object OBJ at the rough movement completion time.
  • the closer the imaging time is to the rough movement processing the smaller the difference between the position of the target object OBJ at the imaging time and the position of the target object OBJ at the rough movement completion time.
  • the difference between the posture of the target object OBJ at the imaging time and the posture of the target object OBJ at the rough movement completion time becomes smaller.
  • the calculation result of at least one of the position and orientation of the target object OBJ calculated before the robot control device 4 completes the rough movement process may be considered to indicate at least one of the position and orientation of the target object OBJ at the rough movement completion time with a reasonable degree of accuracy.
  • the control device 3 may calculate at least one of the position and orientation of the target object OBJ based on image data IMG generated by the imaging device 21 capturing an image of the target object OBJ before the robot control device 4 completes the rough movement process. For the same reason, when the time difference between the image capture time and the rough movement completion time is shorter than a predetermined time threshold, before the robot control device 4 completes the rough movement process, the control device 3 may acquire image data IMG from the imaging device 21. Furthermore, before the robot control device 4 completes the rough movement process, the imaging device 21 may capture an image of the target object OBJ and generate image data IMG at an imaging time that is shorter than a predetermined time threshold value from the rough movement completion time.
  • the control device 3 may calculate at least one of the position and orientation of the target object OBJ based on image data IMG generated by the imaging device 21 capturing an image of the target object OBJ before the robot control device 4 completes the rough movement process.
  • the calculation results of at least one of the position and orientation of the target object OBJ calculated before the robot control device 4 completes the rough movement process may be deemed to indicate at least one of the position and orientation of the target object OBJ at the rough movement completion time with a reasonable degree of accuracy.
  • the control device 3 may calculate at least one of the position and orientation of the target object OBJ based on image data IMG generated by the imaging device 21 imaging the target object OBJ before the robot control device 4 completes the rough movement process.
  • control device 3 may perform a process to generate a control signal for controlling the robot 1 to move the end effector 5 after the rough movement process is completed, based on at least one of the calculation results of the position and orientation of the target object OBJ, before the robot control device 4 completes the rough movement process.
  • the control device 3 may perform a process of generating a control signal for controlling the robot 1 to move the end effector 5 after the rough movement process is completed.
  • the control device 3 may output the control signal generated by the control device 3 before the robot control device 4 completes the rough movement process to the robot 1 or the robot control device 4 after the robot control device 4 completes the rough movement process.
  • the robot 1 may move the end effector 5 based on the control signal generated by the control device 3 before the robot control device 4 completes the rough movement process.
  • control device 3 may output a control signal generated by the control device 3 to the robot 1 or the robot control device 4 before the robot control device 4 completes the rough movement processing.
  • the robot 1 may move the end effector 5 based on the control signal generated by the control device 3 before the robot control device 4 completes the rough movement processing.
  • the control device 3 may move the end effector 5 by performing fine movement processing during the period when the robot control device 4 is moving the end effector 5 by performing rough movement processing.
  • the controller 3 may output a control signal generated by the controller 3 before the robot controller 4 completes the rough movement process to the robot 1 or the robot controller 4 after the robot controller 4 completes the rough movement process and before the controller 3 receives the control permission signal transmitted by the robot controller 4.
  • the robot 1 may move the end effector 5 based on the control signal generated by the controller 3 before the robot controller 4 completes the rough movement process.
  • the control device 3 may perform at least a part of the fine movement process after the robot control device 4 has completed the rough movement process and before receiving a control permission signal transmitted by the robot control device 4. Even in this case, the time required to complete a series of robot control processes including the rough movement process and the fine movement process is reduced compared to a case in which the control device 3 does not perform the fine movement process before receiving a control permission signal.
  • the robot control device 4 may send a control permission signal to the control device 3.
  • the control device 3 may receive the control permission signal even before the robot control device 4 completes the rough movement process.
  • the control device 3 may perform at least a part of the fine movement process, triggered by receiving the control permission signal.
  • the control device 3 can start the fine movement process at an earlier timing than when the robot control device 4 does not send a control permission signal to the control device 3 before the robot control device 4 completes the rough movement process (i.e., the control device 3 does not receive the control permission signal before the robot control device 4 completes the rough movement process).
  • the time required to complete a series of robot control processes including the rough movement process and the fine movement process is shortened.
  • control device 3 may perform at least a part of the fine movement process before the control device 3 completes the rough movement process. Even in this case, the time required to complete a series of robot control processes including the rough movement process and the fine movement process is reduced compared to when the control device 3 does not perform the fine movement process before the control device 3 completes the rough movement process.
  • the control device 3 may generate a control signal used in the fine movement process performed after the rough movement process, based on a control signal generated by the fine movement process performed before the rough movement process.
  • a first example of a robot control process that alternately repeats rough movement processing and fine movement processing is, as described above, a robot control process that is performed when the end effector 5 sequentially performs predetermined processing on a plurality of target portions of the target object OBJ.
  • a first example of a robot control process that alternately repeats rough movement processing and fine movement processing is, as described above, a robot control process that is performed when the end effector 5 sequentially performs predetermined processing on a first target portion of the target object OBJ (for example, the portion in which hole HL#1 shown in FIGS. 5 and 6 is formed) and a second target portion of the target object OBJ that is different from the first target portion (for example, the portion in which hole HL#2 shown in FIGS. 5 and 6 is formed).
  • a first example of a method for generating a control signal in the second modified example will be described with reference to a first example of a robot control process performed when the end effector 5 sequentially performs a predetermined process on a first target portion TP#1 (specifically, the portion where hole HL#1 is formed) of the target object OBJ (workpiece W2) and a second target portion TP#2 (specifically, the portion where hole HL#2 is formed) of the target object OBJ that is different from the first target portion TP#1, as shown in FIG. 13.
  • the robot control device 4 performs a rough movement process on the first target part TP#1, and then, as shown in the first row of FIG. 13, the control device 3 performs a fine movement process on the first target part TP#1, and then, as shown in the second row of FIG. 13, the control device 3 performs a fine movement process on the second target part TP#2.
  • the control device 3 may generate a control signal used in the fine movement process on the second target part TP#2 based on the control signal generated by the fine movement process on the first target part TP#1.
  • control device 3 may perform a process of generating a control signal based on the control signal generated by the fine movement process on the first target part TP#1 as at least a part of the fine movement process on the second target part TP#2.
  • the fine movement process on the second target part TP#2 may include a process of generating a control signal based on the control signal generated by the fine movement process on the first target part TP#1.
  • the control device 3 performs fine movement processing on the first target part TP#1 during a first period from when the rough movement processing on the first target part TP#1 is completed until when the rough movement processing on the second target part TP#2 is started. Furthermore, the control device 3 performs fine movement processing on the second target part TP#2 during a second period after the rough movement processing on the second target part TP#2 is completed. In this case, the control device 3 may generate control signals in the second period based on the control signals generated in the first period. The control device 3 may generate control signals used to control the robot 1 in the second period based on the control signals generated in the first period.
  • control device 3 may generate the first control signal in the second period based on the last control signal generated in the first period. For example, an example will be described in which the control device 3 generates N1 control signals in the first period by repeating the fine movement process for the first target part TP#1 N1 times (note that N1 is a variable indicating an integer of 2 or more) and generates N2 control signals in the second period by repeating the fine movement process for the second target part TP#2 N2 times (note that N2 is a variable indicating an integer of 2 or more).
  • the N1 control signals generated in the first period are respectively referred to as control signal #1[1] to control signal #1[N1].
  • Control signal #1[i] indicates the control signal generated in the i-th fine movement process performed on the first target part TP#1.
  • the N2 control signals generated in the second period are referred to as control signal #2[1] to control signal #2[N2], respectively.
  • Control signal #2[j] indicates the control signal generated in the jth fine movement process performed on the second target portion TP#2.
  • the control device 3 may generate control signal #2[1] based on control signal #1[N1].
  • control signal #1[N1] is the latest control signal that was generated at the time when it was determined that the convergence condition was satisfied during the first period in which the fine movement process for the first target part TP#1 is performed.
  • generating the control signal #2[1] based on the control signal #1[N1] is essentially equivalent to reflecting the control state (e.g., movement) of the robot 1 at the time when it was determined that the convergence condition was satisfied in the control signal #2[1].
  • the control state e.g., movement
  • the robot 1 is controlled using the control signal #2[1] generated based on the control signal #1[N1]
  • the movement required to satisfy the convergence condition is reflected in the robot 1.
  • the time required until the convergence condition is satisfied is reduced compared to when the robot 1 is controlled using the control signal #2[1] generated without being based on the control signal #1[N1].
  • the control signal #1[N1] is the latest control signal generated at the time when it is determined that the convergence condition is satisfied during the first period in which the fine movement process is performed on the first target part TP#1 that moves with the movement of the target object OBJ. For this reason, generating the control signal #2[1] based on the control signal #1[N1] is essentially equivalent to reflecting the control state (e.g., movement) of the robot 1 for making the end effector 5 follow the moving target object OBJ in the control signal #2[1].
  • generating the control signal #2[1] based on the control signal #1[N1] is essentially equivalent to reflecting the movement mode of the target object OBJ and the control state (e.g., movement) of the robot 1 at the time when it is determined that the convergence condition is satisfied in the control signal #2[1].
  • the control signal #2[1] generated based on the control signal #1[N1] the movement required to make the end effector 5 follow the moving target object OBJ is reflected in the robot 1.
  • the control device 3 may generate a control signal #2[j] used to control the robot 1 by the jth fine movement process by adding a correction term based on a differential component U indicating the difference between the actual position of the end effector 5 and the target processing position and the difference between the actual posture of the end effector 5 and the target processing posture to the control signal #2[j-1] generated in the j-1th fine movement process.
  • the correction term may include a term calculated based on the differential component U and the proportional gain Kp.
  • the correction term may include a term calculated based on the differential component U and the integral gain Ki.
  • the correction term may include a term calculated based on the differential component U and the differential gain Kd.
  • the control device 3 may use a default initial signal, which is preset as the initial value of the control signal #2[j], as the control signal #2[0].
  • control device 3 may generate the control signal #2[1] by using the control signal #1[N1] as the control signal #2[0] (i.e., using it as the initial signal).
  • generating the control signal #2[1] based on the control signal #1[N1] may include generating the control signal #2[1] by using the control signal #1[N1] as the control signal #2[0], which is the initial signal.
  • the control device 3 generates the control signal #2[1] that is generated first in the second period based on the control signal #1[N1] that was generated last in the first period.
  • the control device 3 may generate the control signal #2[j] at any second time in the second period based on the control signal #1[i] that was generated at any first time in the first period.
  • the control device 3 may generate the control signal #2[j] used to control the robot 1 at any second time in the second period based on the control signal #1[i] that was generated at any first time in the first period.
  • the control device 3 may generate the control signal #2[j] by using the control signal #1[i] as the control signal #2[j-1] to which a correction term is added to generate the control signal #2[j].
  • the movement of the robot 1 by the fine movement process for the first target part TP#1 is reflected in the control signal #2[j] that is used to control the robot 1 by the fine movement process for the second target part TP#2.
  • the movement mode of the first target part TP#1 i.e., the movement mode of the target object OBJ
  • the movement of the robot 1 are reflected in the control signal #2[j] used to control the robot 1 by the fine movement process for the second target part TP#2. Therefore, the time required until the convergence condition is satisfied is reduced compared to when the robot 1 is controlled using the control signal #2[j] generated without being based on the control signal #1[i].
  • the control device 3 may generate the control signal #2[j] based on the control signal #1[i] that satisfies the condition that the movement of the robot 1 required to satisfy the convergence condition by the fine movement processing on the first target portion TP#1 is appropriately reflected in the control signal #2[j].
  • the control device 3 may generate the control signal #2[j] based on the control signal #1[N1-1].
  • the control device 3 may generate the control signal #2[j] based on the control signal #1[N1-2].
  • the closer the time when the control signal #2[j] is generated to the start of the second period i.e., the smaller the variable j
  • control device 3 may generate a control signal #2[j] based on the control signal #1[i] that satisfies the condition that the time required from the start of the fine movement process for the second target part TP#2 until the convergence condition is satisfied is appropriately short.
  • the control device 3 may generate the control signal #2[2] based on the control signal #1[i].
  • the control device 3 may generate the control signal #2[3] based on the control signal #1[i].
  • control device 3 generates the control signal #2[j] in the second period based on a single control signal #1[i] generated in the first period.
  • control device 3 may generate the control signal #2[j] in the second period based on multiple control signals #1[i] generated in the first period.
  • control device 3 may generate the control signal #2[j] in the second period based on the average value of the multiple control signals #1[i] generated in the first period.
  • the control device 3 may generate a control signal used in the fine movement process for the second target part TP#2 based on a control signal generated by the fine movement process for the first target part TP#1. For example, even if the control device 3 performs a fine movement process for the first target part TP#1 and then performs a fine movement process for the second target part TP#2 without the robot control device 4 performing a rough movement process for the second target part TP#2, the control device 3 may generate a control signal used in the fine movement process for the second target part TP#2 based on a control signal generated by the fine movement process for the first target part TP#1.
  • a second example of the robot control process in which rough movement processing and fine movement processing are alternately repeated is the robot control process performed when the end effector 5 sequentially performs predetermined processing on a plurality of target objects OBJ, as described above.
  • a robot control process performed when the end effector 5 sequentially performs predetermined processing on a first target object OBJ and a second target object OBJ different from the first target object OBJ will be described.
  • the robot control device 4 performs a rough movement process on the first target object OBJ#1, and then, as shown in the first row of FIG. 14, the control device 3 performs a fine movement process on the first target object OBJ#1, and then, as shown in the second row of FIG. 14, the robot control device 4 performs a rough movement process on the second target object OBJ#2, and then, as shown in the second row of FIG. 14, the control device 3 performs a fine movement process on the second target object OBJ#2.
  • the control device 3 may generate a control signal used in the fine movement process on the second target object OBJ#2 based on the control signal generated by the fine movement process on the first target object OBJ#1.
  • control device 3 may perform a process of generating a control signal based on the control signal generated by the fine movement process on the first target object OBJ#1 as at least a part of the fine movement process on the second target object OBJ#2.
  • the fine movement process for the second target object OBJ#2 may include a process of generating a control signal based on the control signal generated by the fine movement process for the first target object OBJ#1.
  • the control device 3 performs fine movement processing on the first target object OBJ#1 in a third period from when the rough movement processing on the first target object OBJ#1 is completed until when the rough movement processing on the second target object OBJ#2 is started. Furthermore, the control device 3 performs fine movement processing on the second target object OBJ#2 in a fourth period after the rough movement processing on the second target object OBJ#2 is completed. In this case, the control device 3 may generate control signals in the fourth period based on the control signals generated in the third period. The control device 3 may generate control signals used to control the robot 1 in the fourth period based on the control signals generated in the third period.
  • control device 3 may generate the first control signal in the fourth period based on the last control signal generated in the third period. For example, an example will be described in which the control device 3 generates N3 control signals in the third period by repeating the fine movement process for the first object OBJ#1 N3 times (note that N3 is a variable indicating an integer equal to or greater than 2) and generates N4 control signals in the fourth period by repeating the fine movement process for the second object OBJ#2 N4 times (note that N4 is a variable indicating an integer equal to or greater than 2).
  • the N3 control signals generated in the third period are referred to as control signal #3[1] to control signal #3[N3], respectively.
  • Control signal #3[m] (note that m is a variable indicating an integer equal to or greater than 1 and equal to or less than N3) indicates the control signal generated in the mth fine movement process performed on the first object OBJ#1.
  • the N4 control signals generated in the fourth period are referred to as control signal #4[1] to control signal #4[N4], respectively.
  • Control signal #4[n] (n is a variable indicating an integer greater than or equal to 1 and less than or equal to N4) indicates the control signal generated in the nth fine movement process performed on the second target object OBJ#2.
  • the control device 3 may generate control signal #4[1] based on control signal #3[N3].
  • control signal #4[1] is generated based on the control signal #3[N3]
  • the time required from the start of the fine movement process for the second object OBJ#2 until the convergence condition is satisfied is shortened, compared to the case where the control signal #4[1] is generated without being based on the control signal #3[N3].
  • the control signal #3[N3] is the latest control signal generated at the time when it is determined that the convergence condition is satisfied during the third period in which the fine movement process for the first object OBJ#1 is performed.
  • generating the control signal #4[1] based on the control signal #3[N3] is essentially equivalent to reflecting the control state (e.g., movement) of the robot 1 at the time when it is determined that the convergence condition is satisfied in the control signal #2[1].
  • the control state e.g., movement
  • the robot 1 is controlled using the control signal #4[1] generated based on the control signal #3[N3]
  • the movement required to satisfy the convergence condition is reflected in the robot 1.
  • the time required until the convergence condition is satisfied is reduced compared to when the robot 1 is controlled using the control signal #4[1] generated without being based on the control signal #3[N3].
  • the effect of shortening the time required for the convergence condition to be satisfied is particularly noticeable when the first target object OBJ#1 and the second target object OBJ#2 are moving.
  • the effect of shortening the time required for the convergence condition to be satisfied is particularly noticeable when the first target object OBJ#1 and the second target object OBJ#2 are moving in the same moving manner. The reason for this is explained below.
  • the control signal #3[N3] is the latest control signal that was generated at the time when it was determined that the convergence condition was satisfied during the third period in which the fine movement process is performed on the moving first target object OBJ#1.
  • generating the control signal #4[1] based on the control signal #3[N3] is essentially equivalent to reflecting the control state (e.g., movement) of the robot 1 for making the end effector 5 follow the moving first target object OBJ#1 in the control signal #4[1].
  • generating the control signal #4[1] based on the control signal #3[N3] is essentially equivalent to reflecting the movement mode of the first object OBJ#1 and the control state (e.g., movement) of the robot 1 at the time when it is determined that the convergence condition is satisfied in the control signal #4[1].
  • the control device 3 may generate a control signal #4[n] used to control the robot 1 by the nth fine movement process by adding a correction term based on a differential component U indicating the difference between the actual position of the end effector 5 and the target processing position and the difference between the actual posture of the end effector 5 and the target processing posture to the control signal #4[n-1] generated in the n-1th fine movement process.
  • the correction term may include a term calculated based on the differential component U and the proportional gain Kp.
  • the correction term may include a term calculated based on the differential component U and the integral gain Ki.
  • the correction term may include a term calculated based on the differential component U and the differential gain Kd.
  • control device 3 may generate the control signal #4[1] by using the control signal #3[N3] as the control signal #4[0] (i.e., using it as the initial signal).
  • generating the control signal #4[1] based on the control signal #3[N3] may include generating the control signal #4[1] by using the control signal #3[N3] as the control signal #4[0], which is the initial signal.
  • the control device 3 generates the control signal #4[1] that is generated first in the fourth period based on the control signal #3[N3] that was generated last in the third period.
  • the control device 3 may generate the control signal #4[n] at any fourth time during the fourth period based on the control signal #3[m] that was generated at any third time during the third period.
  • the control device 3 may generate the control signal #4[n] used to control the robot 1 at any fourth time during the fourth period based on the control signal #3[m] that was generated at any third time during the third period.
  • the control device 3 may generate the control signal #4[n] by using the control signal #3[m] as the control signal #4[n-1] to which a correction term is added to generate the control signal #4[n].
  • the movement of the robot 1 by the fine movement process for the first object OBJ#1 is reflected in the control signal #4[n] that is used to control the robot 1 by the fine movement process for the second object OBJ#2.
  • the control signal #4[n] used to control the robot 1 by fine movement processing for the second target object OBJ#2. Therefore, the time required until the convergence condition is satisfied is reduced compared to when the robot 1 is controlled using the control signal #4[n] generated without being based on the control signal #3[m].
  • the control device 3 may generate the control signal #4[n] based on the control signal #3[m] that satisfies the condition that the movement of the robot 1 required to satisfy the convergence condition by the fine movement processing on the first target object OBJ#1 is appropriately reflected in the control signal #4[n].
  • the control device 3 may generate the control signal #4[n] based on the control signal #3[N3-1].
  • the control device 3 may generate the control signal #4[n] based on the control signal #3[N3-2].
  • the closer the time when the control signal #4[n] is generated to the start of the fourth period i.e., the smaller the variable n
  • control device 3 may generate a control signal #4[n] based on the control signal #3[m] that satisfies the condition that the time required from the start of the fine movement process for the second target object OBJ#2 until the convergence condition is satisfied is appropriately short.
  • control device 3 may generate a control signal #4[2] based on the control signal #3[m].
  • control device 3 may generate a control signal #4[3] based on the control signal #3[m].
  • control device 3 generates the control signal #4[n] in the fourth period based on a single control signal #3[m] generated in the third period.
  • control device 3 may generate the control signal #4[n] in the fourth period based on multiple control signals #3[m] generated in the third period.
  • control device 3 may generate the control signal #4[n] in the fourth period based on the average value of the multiple control signals #3[m] generated in the third period.
  • the control device 3 may generate a control signal used in the fine movement processing for the second target object OBJ#2 based on a control signal generated by the fine movement processing for the first target object OBJ#1. For example, even if the control device 3 performs fine movement processing for the first target object OBJ#1 and then performs fine movement processing for the second target object OBJ#2 without the robot control device 4 performing rough movement processing for the second target object OBJ#2, the control device 3 may generate a control signal used in the fine movement processing for the second target object OBJ#2 based on a control signal generated by the fine movement processing for the first target object OBJ#1.
  • the control device 3 may use edge model data EMD as the model data MDL in order to calculate the position and orientation of the target object OBJ by performing the above-mentioned matching process.
  • the edge model data EMD is model data of the edge model EM that can be used for the matching process similarly to the object model OBM.
  • the edge model EM is a model of at least a part of the edges of the target object OBJ.
  • the edge model EM is a model showing at least a part of the edges of the target object OBJ.
  • the edge model EM is a model showing at least a part of the edges of the target object OBJ.
  • the edge model EM may be a model showing at least a part of the edges of the target object OBJ while not showing parts of the target object OBJ other than the edges.
  • the edge model EM may be a model of at least a part of the contour of the target object OBJ.
  • the edge model EM may be a model showing at least a part of the contour of the target object OBJ.
  • the control device 3 may calculate at least one of the position and orientation of the target object OBJ by performing a matching process using the image data IMG and the edge model data EMD. That is, the control device 3 (particularly, the position and orientation calculation unit 311) may perform a matching process (in other words, a 2D matching process, a template matching process) using the edge model EM indicated by the edge model data EMD as a template for the image indicated by the image data IMG.
  • a matching process in other words, a 2D matching process, a template matching process
  • the matching process using the image data IMG and the edge model data EMD may be the same as the matching process using the image data IMG and the model data MDL described above, except that the edge model data EMD is used instead of the model data MDL. For this reason, a detailed description of the matching process using the image data IMG and the edge model data EMD will be omitted.
  • the edge model data EMD may be generated by the following method.
  • the imaging device 21 (or an imaging device different from the imaging device 21, the same in the following paragraph) may capture an image of the target object OBJ before the fine movement process is started.
  • the target object OBJ captured in advance may be a reference or non-defective target object OBJ.
  • the imaging device 21 generates image data IMG showing an image in which the target object OBJ is captured.
  • the control device 3 (or a model generation device different from the control device 3, the same in the following paragraph) may detect (in other words, extract) the edge of the target object OBJ captured in the image data IMG.
  • control device 3 may detect, as the edge of the target object OBJ, a set of pixels whose pixel value (e.g., luminance value) change exceeds a predetermined edge threshold value among a plurality of pixels constituting the image shown by the image data IMG. Then, as shown in FIG. 15, the control device 3 may generate a model of the extracted edge as an edge model EM to generate the edge model data EMD. In other words, the control device 3 may generate edge model data EMD that indicates the extracted edges.
  • pixel value e.g., luminance value
  • the imaging device 21 may capture an image of a sample object having the same shape as the target object OBJ in addition to or instead of the target object OBJ before the fine movement process is started. As a result, the imaging device 21 generates image data IMG showing an image in which the sample object is reflected. Thereafter, the control device 3 (or a model generation device different from the control device 3, the same applies hereinafter in this paragraph) may detect (in other words, extract) the edge of the sample object reflected in the image data IMG.
  • control device 3 may detect, as the edge of the sample object, a set of pixels whose pixel value (e.g., luminance value) change amount exceeds a predetermined edge threshold value among a plurality of pixels constituting the image shown by the image data IMG. Thereafter, the control device 3 may generate edge model data EMD by generating a model of the extracted edge as an edge model EM. In other words, the control device 3 may generate edge model data EMD showing the extracted edge.
  • pixel value e.g., luminance value
  • control device 3 may detect edges of a two-dimensional model of the target object OBJ that is generated by virtually projecting a CAD model (or any three-dimensional model) of the target object OBJ from a certain direction onto a virtual plane perpendicular to the certain direction, and generate edge model data EMD indicating the detected edges.
  • the edge model EM selectively indicates the edges of the target object OBJ
  • the above-mentioned object model OBM indicates parts of the target object OBJ other than the edges in addition to or instead of the edges of the target object OBJ. Therefore, the data size of the edge model data EMD indicating the edge model EM is smaller than the data size of the model data MDL indicating the object model OBM. Therefore, when a matching process is performed using the edge model data EMD (i.e., the model data MDL indicating the edge model EM), the time required before completing the matching process is shortened compared to when a matching process is performed using the model data MDL indicating the above-mentioned object model OBM.
  • the detection accuracy of the target object OBJ i.e., the calculation accuracy of at least one of the position and orientation of the target object OBJ
  • the edge model data EMD i.e., the model data MDL indicating the edge model EM
  • the control device 3 may generate edge model data EMD indicating all edges detected (extracted) from the image data.
  • the control device 3 may generate edge model data EMD selectively indicating a portion of the edges detected (extracted) from the image data. For example, the control device 3 may select at least a portion of the edges that can be effectively used to detect the target object OBJ by a matching process from the edges detected from the image data, and generate edge model data EMD indicating the selected edges. As one example, the control device 3 may select at least a portion of the edges that have high contrast and are easy to recognize from the edges detected from the image data, and generate edge model data EMD indicating the selected edges. As another example, the control device 3 may select at least a portion of the edges that are in focus from the edges detected from the image data, and generate edge model data EMD indicating the selected edges.
  • control device 3 may select at least a portion of the edges specified by the user from the edges detected from the image data, and generate edge model data EMD indicating the selected edges.
  • edge model data EMD selectively indicating a part of the detected (extracted) edges is used in the matching process
  • the control device 3 can detect the target object OBJ with high accuracy by performing the fine movement process (particularly, performing the matching process) compared to when edge model data EMD indicating all of the detected (extracted) edges is used in the matching process. This is because the target object OBJ is detected using at least a part of the edges that can be effectively used to detect the target object OBJ by the matching process.
  • the target object OBJ is detected without using at least a part of the other edges that cannot necessarily be effectively used to detect the target object OBJ by the matching process.
  • the control device 3 reduces the possibility of erroneously detecting an object that should not be detected as the target object OBJ as the target object OBJ by performing the fine movement process (particularly, performing the matching process).
  • the control device 3 may use the edge model data EMD as the model data MDL to calculate the position and orientation of the target object OBJ by performing the above-mentioned matching process.
  • the control device 3 may use the edge model data EMD as the model data MDL to calculate the position and orientation of the target object OBJ by performing the matching process.
  • control device 3 may calculate at least one of the position and orientation of the end effector 5 in addition to at least one of the position and orientation of the target object OBJ by performing a matching process in step S322 of FIG. 10.
  • control device 3 may generate position and orientation information POI indicating at least one of the position and orientation of the target object OBJ and at least one of the position and orientation of the end effector 5.
  • control device 3 may calculate, as the position of the end effector 5, at least one of the following: the position of the end effector 5 in an X-axis direction parallel to the X-axis in the global coordinate system; the position of the end effector 5 in a Y-axis direction parallel to the Y-axis in the global coordinate system; and the position of the end effector 5 in a Z-axis direction parallel to the Z-axis in the global coordinate system.
  • the control device 3 may calculate, as the orientation of the end effector 5, at least one of the rotation amount (orientation) of the end effector 5 around the X-axis in the global coordinate system, the rotation amount (orientation) of the end effector 5 around the Y-axis in the global coordinate system, and the rotation amount (orientation) of the end effector 5 around the Z-axis in the global coordinate system.
  • the imaging device 21 may capture an image of the end effector 5 in addition to the target object OBJ.
  • the imaging field of view of the imaging device 21 may include the end effector 5 in addition to the target object OBJ.
  • the imaging field of view of the imaging device 21 may include the end effector 5 in addition to the target object OBJ.
  • the control device 3 may generate a control signal using the position and orientation information POI including at least one of the calculation results of the position and orientation of the target object OBJ and at least one of the calculation results of the position and orientation of the end effector 5.
  • the control device 3 may generate a control signal so that the positional relationship between the end effector 5 whose position is indicated by the position and orientation information POI and the target object OBJ whose position is indicated by the position and orientation information POI becomes a predetermined target positional relationship.
  • control device 3 may generate a control signal so that the attitude relationship between the end effector 5 whose attitude is indicated by the position and orientation information POI and the target object OBJ whose attitude is indicated by the position and orientation information POI becomes a predetermined target attitude relationship.
  • the control device 3 may use model data MDL indicating the end effector model EEM in addition to the object model OBM (or the edge model EM described in the third modified example) as shown in FIG. 16.
  • the model data MDL used for the fine movement process (particularly the matching process) in the fourth modified example may indicate the end effector model EEM in addition to the object model OBM (or the edge model EM described in the third modified example).
  • the end effector model EEM may be referred to as an equipment model.
  • the end effector model EEM is a model of the end effector 5.
  • the end effector model EEM is, for example, a two-dimensional model and includes a two-dimensional image.
  • the model data MDL is data indicating the end effector model EEM, which is a two-dimensional model having a two-dimensional shape that serves as a reference for the end effector 5.
  • the end effector model EEM may be, for example, a two-dimensional model (two-dimensional image) of the end effector 5 that is generated by virtually projecting a CAD model (or any three-dimensional model) of the end effector 5 from a plurality of different directions onto a virtual plane perpendicular to each of the plurality of different directions.
  • the end effector model EEM may be a two-dimensional image generated by capturing an image of the actual end effector 5 in advance.
  • the actual end effector 5 that is measured in advance to generate the end effector model EEM may be a reference or non-defective end effector 5.
  • the end effector model EEM may be a model of at least a portion of the edge of the end effector 5, similar to the edge model EM which is a model of the edge of the target object OBJ described in the third modified example.
  • the end effector model EEM may be a model showing at least a portion of the edge of the end effector 5.
  • the end effector model EEM may be a model that represents at least a portion of the edge of the end effector 5.
  • the end effector model EEM may be a model that shows at least a portion of the edge of the end effector 5, while not showing any part of the end effector 5 other than the edge.
  • the end effector model EEM may be a model of at least a portion of the contour of the end effector 5.
  • the end effector model EEM may be a model that shows at least a portion of the contour of the end effector 5.
  • the method for generating the end effector model EEM which is a model of the edge of the end effector 5 may be the same as the method for generating the edge model EM, which is a model of the edge of the target object OBJ, described in the third modified example.
  • the method for generating the model data MDL representing the end effector model EEM may be the same as the method for generating the edge model data EMD representing the edge model EM, described in the third modified example. Therefore, in order to avoid redundant explanations, details of the method for generating the model data MDL representing the end effector model EEM will be omitted.
  • the model data MDL indicates an end effector model EEM that indicates parts other than the edges of the end effector 5 in addition to or instead of the edges of the end effector 5
  • the model data MDL may indicate the end effector model EEM and an object model OBM that indicates parts other than the edges of the target object OBJ in addition to or instead of the edges of the target object OBJ.
  • a matching process may be performed using the model data MDL that indicates the end effector model EEM that indicates parts other than the edges of the end effector 5 in addition to or instead of the edges of the end effector 5 and the partial object model OBM that indicates parts other than the edges of the target object OBJ in addition to or instead of the edges of the target object OBJ.
  • the model data MDL may indicate the end effector model EEM and an edge model EM that indicates the edges of the target object OBJ but does not indicate any part of the target object OBJ other than the edges.
  • a matching process may be performed using the model data MDL that indicates the end effector model EEM, which is an edge model, and the edge model EM.
  • the model data MDL may indicate an object model OBM (or an edge model EM, the same applies below in the fourth modified example) and an end effector model EEM that are aligned with each other, as shown in FIG. 16.
  • the model data MDL may include information regarding the positional relationship between the object model OBM and the end effector model EEM.
  • the model data MDL may include information regarding the attitude relationship between the object model OBM and the end effector model EEM in addition to or instead of information regarding the positional relationship between the object model OBM and the end effector model EEM.
  • the control device 3 may perform, as at least a part of the fine movement process, a process of generating a control signal so that the difference between the positional relationship between the target object OBJ and the end effector 5 and the positional relationship between the aligned object model OBM and the end effector model EEM is within a predetermined upper tolerance threshold.
  • the control device 3 may move the end effector 5 so that the difference between the positional relationship between the target object OBJ and the end effector 5 and the positional relationship between the aligned object model OBM and the end effector model EEM is within a predetermined upper tolerance threshold.
  • the control device 3 may calculate the respective positions of the target object OBJ and the end effector 5 by performing a matching process using the model data MDL. Thereafter, as part of the fine movement process, the control device 3 may calculate the positional relationship between the target object OBJ and the end effector 5 based on the calculation results of the respective positions of the target object OBJ and the end effector 5. Thereafter, as part of the fine movement process, the control device 3 may move the end effector 5 so that the difference between the calculation result of the positional relationship between the target object OBJ and the end effector 5 and the positional relationship between the object model OBM and the end effector model EEM indicated by the model data MDL is within an allowable upper limit threshold.
  • control device 3 may repeat the fine movement process as necessary until the difference between the calculation result of the positional relationship between the target object OBJ and the end effector 5 and the positional relationship between the object model OBM and the end effector model EEM indicated by the model data MDL is within an allowable upper limit threshold.
  • the predetermined upper tolerance threshold may be zero.
  • the state in which "the difference between the positional relationship between the target object OBJ and the end effector 5 and the positional relationship between the object model OBM and the end effector model EEM is within the predetermined upper tolerance threshold" may include the state in which "the positional relationship between the target object OBJ and the end effector 5 is completely the same as the positional relationship between the object model OBM and the end effector model EEM.”
  • the predetermined upper tolerance threshold may be greater than zero.
  • the state in which "the difference between the positional relationship between the target object OBJ and the end effector 5 and the positional relationship between the object model OBM and the end effector model EEM is within a predetermined allowable upper threshold" may include, in addition to a state in which "the positional relationship between the target object OBJ and the end effector 5 is completely the same as the positional relationship between the object model OBM and the end effector model EEM", a state in which "the positional relationship between the target object OBJ and the end effector 5 is not completely the same as the positional relationship between the object model OBM and the end effector model EEM, but the deviation amount of the positional relationship between the target object OBJ and the end effector 5 from the positional relationship between the object model OBM and the end effector model EEM is small enough to be equal to or less than a predetermined allowable upper threshold".
  • the state in which "the difference between the positional relationship between the target object OBJ and the end effector 5 and the positional relationship between the object model OBM and the end effector model EEM is within a predetermined allowable upper threshold value" may be considered to include the state in which "the positional relationship between the target object OBJ and the end effector 5 is substantially the same as the positional relationship between the object model OBM and the end effector model EEM.”
  • the object model OBM and the end effector model EEM indicated by the model data MDL may be aligned with each other so that the positional relationship between the object model OBM and the end effector model EEM is the same as the target positional relationship described above.
  • the object model OBM and the end effector model EEM indicated by the model data MDL may be aligned with each other so that the positional relationship between the object model OBM and the end effector model EEM is the same as the positional relationship between the target object OBJ and the end effector 5 to be realized by the fine movement processing.
  • the control device 3 may perform, as at least a part of the fine movement process, a process of generating a control signal so that the difference between the attitude relationship between the target object OBJ and the end effector 5 and the attitude relationship between the object model OBM and the end effector model EEM that are aligned with each other is within a predetermined upper tolerance threshold.
  • control device 3 may move the end effector 5 so that the difference between the attitude relationship between the target object OBJ and the end effector 5 and the attitude relationship between the object model OBM and the end effector model EEM that are aligned with each other is within a predetermined upper tolerance threshold.
  • the control device 3 may calculate the orientations of the target object OBJ and the end effector 5 by performing a matching process using the model data MDL. Thereafter, as part of the fine movement process, the control device 3 may calculate the orientation relationship between the target object OBJ and the end effector 5 based on the calculation results of the orientations of the target object OBJ and the end effector 5. Thereafter, as part of the fine movement process, the control device 3 may move the end effector 5 so that the difference between the calculation result of the orientation relationship between the target object OBJ and the end effector 5 and the orientation relationship between the object model OBM and the end effector model EEM indicated by the model data MDL is within an allowable upper threshold.
  • control device 3 may repeat the fine movement process as necessary until the difference between the calculation result of the orientation relationship between the target object OBJ and the end effector 5 and the orientation relationship between the object model OBM and the end effector model EEM indicated by the model data MDL is within an allowable upper threshold.
  • the predetermined upper tolerance threshold may be zero.
  • the state in which "the difference between the attitude relationship between the target object OBJ and the end effector 5 and the attitude relationship between the object model OBM and the end effector model EEM is within the predetermined upper tolerance threshold" may include the state in which "the attitude relationship between the target object OBJ and the end effector 5 is completely the same as the attitude relationship between the object model OBM and the end effector model EEM.”
  • the predetermined upper tolerance threshold may be greater than zero.
  • the state in which "the difference between the attitude relationship between the target object OBJ and the end effector 5 and the attitude relationship between the object model OBM and the end effector model EEM is within a predetermined allowable upper threshold value" may include, in addition to a state in which "the attitude relationship between the target object OBJ and the end effector 5 is completely the same as that between the object model OBM and the end effector model EEM", a state in which "the attitude relationship between the target object OBJ and the end effector 5 is not completely the same as that between the object model OBM and the end effector model EEM, but the deviation amount of the attitude relationship between the target object OBJ and the end effector 5 with respect to the attitude relationship between the object model OBM and the end effector model EEM is small enough to be equal to or less than a predetermined allowable upper threshold value".
  • the state in which "the difference between the attitude relationship between the target object OBJ and the end effector 5 and the attitude relationship between the object model OBM and the end effector model EEM is within a predetermined allowable upper threshold" may be considered to include the state in which "the attitude relationship between the target object OBJ and the end effector 5 is substantially the same as the attitude relationship between the object model OBM and the end effector model EEM.”
  • the object model OBM and the end effector model EEM indicated by the model data MDL may be aligned with each other so that the attitude relationship between the object model OBM and the end effector model EEM is the same as the target attitude relationship described above.
  • the object model OBM and the end effector model EEM indicated by the model data MDL may be aligned with each other so that the attitude relationship between the object model OBM and the end effector model EEM is the same as the attitude relationship between the target object OBJ and the end effector 5 that should be realized by the fine movement process.
  • the control device 3 may perform, as at least a part of the fine movement process, a process of selecting one target object OBJ (i.e., a process execution object) for which the end effector 5 actually performs a predetermined process from among the multiple target objects OBJ, based on the positional relationship between the object model OBM and the end effector model EEM that are aligned with each other.
  • a process execution object i.e., a process execution object
  • the control device 3 may select one target object OBJ (i.e., a process execution object) that satisfies the condition that "the positional relationship between the target object OBJ and the end effector 5 is closest to the positional relationship between the object model OBM and the end effector model EEM that are aligned with each other.”
  • a process execution object i.e., a process execution object
  • FIG. 17 shows an image represented by image data IMG when target objects OBJ#A and OBJ#B are detected by matching processing.
  • the image represented by image data IMG includes target objects OBJ#A and OBJ#B, and the end effector 5.
  • the matching similarity of the target object OBJ#A and the matching similarity of the target object OBJ#B both exceed the above-mentioned matching judgment threshold. Therefore, both the target objects OBJ#A and OBJ#B are detected by the matching process. Furthermore, in the example shown in FIG. 17, the matching similarity of the target object OBJ#A is higher than the matching similarity of the target object OBJ#B. In this case, if one target object OBJ with the maximum matching similarity is selected as the processing object from among multiple target objects OBJ, the control device 3 selects the target object OBJ#A with the maximum matching similarity as the processing object. However, in the example shown in FIG.
  • the distance between the target object OBJ#A and the end effector 5 is longer than the distance between the target object OBJ#B and the end effector 5. For this reason, the distance that the end effector 5 moves when the target object OBJ#A is selected as the object to be processed by the fine movement process is longer than the distance that the end effector 5 moves when the target object OBJ#B is selected as the object to be processed by the fine movement process.
  • the control device 3 from the viewpoint of efficiently moving the end effector 5 by reducing the amount of movement of the end effector 5, it may not be necessary for the control device 3 to select the target object OBJ#A as the object to be processed. In other words, from the viewpoint of efficiently moving the end effector 5 by reducing the amount of movement of the end effector 5, it may be preferable for the control device 3 to select the target object OBJ#B as the object to be processed.
  • the control device 3 first calculates, by a matching process, at least one of the position and orientation of the target object OBJ#A, at least one of the position and orientation of the target object OBJ#B, and at least one of the position and orientation of the end effector 5. After that, the control device 3 calculates the distance between the target object OBJ#A and the end effector 5 and the distance between the target object OBJ#B and the end effector 5 based on the calculation results of at least one of the positions and orientations of the target object OBJ#A, target object OBJ#B, and end effector 5. In this case, the control device 3 selects one target object OBJ with the shorter calculated distance as the object to be processed.
  • the control device 3 selects the target object OBJ#B as the object to be processed.
  • the control device 3 compares the positional relationship between each of the target objects OBJ#A and OBJ#B detected by the matching process and the end effector 5 with the object model OBM and the end effector model EEM that are aligned with each other.
  • the object model OBM and the end effector model EEM that are aligned with each other.
  • the positional relationship between the target object OBJ#B and the end effector 5 is closer to the positional relationship between the object model OBM and the end effector model EEM that are aligned with each other than the positional relationship between the target object OBJ#A and the end effector 5.
  • the translation difference i.e., the difference in the translation direction
  • the translation difference between the position of the object model OBM aligned with the end effector model EEM and the position of the target object OBJ#B is smaller than the translation difference between the position of the object model OBM and the position of the target object OBJ#A.
  • the control device 3 selects the target object OBJ#B as the object to be processed. Therefore, the control device 3 can generate a control signal for controlling the robot 1 to move the end effector 5 efficiently by reducing the amount of movement of the end effector 5. In other words, the control device 3 can move the end effector 5 efficiently by reducing the amount of movement of the end effector 5.
  • the control device 3 may calculate at least one of the position and orientation of the end effector 5. In other words, when the robot control device 4 does not perform rough movement processing and the control device 3 starts fine movement processing, the control device 3 may calculate at least one of the position and orientation of the end effector 5. Furthermore, the control device 3 may generate a control signal based on the calculation result of at least one of the position and orientation of the end effector 5.
  • the control device 3 calculates at least one of the positions and orientations of the target object OBJ and the end effector 5.
  • the control device 3 may calculate at least one of the positions and orientations of the object held by the end effector 5 in addition to or instead of at least one of the positions and orientations of the end effector 5.
  • the control device 3 may calculate at least one of the positions and orientations of the workpiece W held by the end effector 5.
  • the control device 3 may calculate at least one of the positions and orientations of the workpiece W#1 held by the end effector 5.
  • the control device 3 may calculate at least one of the positions and orientations of the workpiece W#2 held by the end effector 5.
  • the robot 1 may perform a placement process. For example, as described above, the robot 1 may hold a first target object OBJ using the end effector 5, and then perform a placement process to place the first target object OBJ held by the end effector 5 at a desired position of a second target object OBJ different from the first target object OBJ.
  • the control device 3 may calculate at least one of the position and orientation of the first target object OBJ held by the end effector 5, and at least one of the position and orientation of the second target object OBJ from which the end effector 5 releases the first target object OBJ.
  • the control device 3 may calculate at least one of the position and orientation of the first target object OBJ held by the end effector 5 and at least one of the position and orientation of the second target object OBJ at which the end effector 5 releases the first target object OBJ. In this case, when at least one of the positions and orientations of the first target object OBJ and the second target object OBJ is calculated, the control device 3 may generate a control signal for controlling the robot 1 so that the positional relationship between the first target object OBJ and the second target object OBJ becomes a predetermined target positional relationship.
  • the control device 3 may generate a control signal for controlling the robot 1 so that the orientation relationship between the first target object OBJ and the second target object OBJ becomes a predetermined target orientation relationship.
  • the control device 3 may repeat the fine movement process until the positional relationship between the first target object OBJ and the second target object OBJ becomes a predetermined target positional relationship. In this case, the control device 3 may repeat the fine movement process until the orientation relationship between the first target object OBJ and the second target object OBJ becomes a predetermined target orientation relationship.
  • the target positional relationship referred to here may be the ideal positional relationship between the first target object OBJ and the second target object OBJ when the end effector 5 holding the first target object OBJ is located at a target processing position TPP where the end effector 5 performs a predetermined processing on the second object OBJ.
  • the target posture relationship referred to here may be the ideal posture relationship between the first target object OBJ and the second target object OBJ when the end effector 5 holding the first target object OBJ is located at a target processing position TPP where the end effector 5 performs a predetermined processing on the second object OBJ.
  • the control device 3 may perform the fine movement processing as necessary, as described above.
  • the control device 3 may perform the fine movement processing even during the processing period so that the end effector 5 follows the moving target object OBJ, as described above.
  • the target object OBJ will not be included in the imaging field of view of the imaging device 21 during the processing period.
  • the first workpiece W1 released from the end effector 5 may be located between the imaging device 21 and the hole HL into which the first workpiece W should be fitted (i.e., the target portion in which the hole HL is formed, the same applies below in the fifth modified example).
  • the hole HL may be hidden by the first workpiece W1, and as a result, the hole HL may not be included in the imaging field of the imaging device 21. In other words, the imaging device 21 may not be able to image the hole HL.
  • the control device 3 may not be able to detect the hole HL by the matching process. In this case, the control device 3 may not be able to calculate at least one of the position and orientation of the hole HL, and as a result, the control device 3 may not be able to generate a control signal for controlling the robot 1 so that the end effector 5 follows the hole HL.
  • the end effector 5 when the end effector 5 holds the workpiece W, the end effector 5 may approach the workpiece W (and as a result, the imaging device 21 may approach the workpiece W), which may cause part of the workpiece W to be excluded from the imaging field of the imaging device 21. In other words, the imaging device 21 may not be able to image part of the workpiece W. As a result, the control device 3 may not be able to detect the workpiece W by the matching process. Even if the control device 3 is able to detect the workpiece W by the matching process, the calculation accuracy of at least one of the position and orientation of the workpiece W may decrease.
  • control device 3 may not be able to accurately calculate at least one of the position and orientation of the workpiece W, and as a result, the control device 3 may not be able to generate a control signal that controls the robot 1 so that the end effector 5 follows the workpiece W.
  • the target object OBJ may be hidden by an obstacle, and therefore a part of the workpiece W may not be included in the imaging field of the imaging device 21.
  • the end effector 5 may get close to the target object OBJ (and as a result, the imaging device 21 gets close to the target object OBJ), and therefore a part of the workpiece W may not be included in the imaging field of the imaging device 21.
  • the control device 3 may not be able to generate a control signal for controlling the robot 1 so that the end effector 5 follows the target object OBJ.
  • the control device 3 may perform processing to estimate at least one of the position and orientation of the target object OBJ as at least a part of the fine movement processing.
  • the fine movement processing may include processing to estimate at least one of the position and orientation of the target object OBJ when an imaging abnormality condition occurs.
  • FIG. 18 is a flowchart showing the flow of the robot control process in the fifth modified example.
  • the robot control process in the fifth modified example is the same as the robot control process shown in FIG. 9 in that the robot control device 4 first performs rough movement processing (step S41), then the robot control device 4 transmits a control permission signal to the control device 3 (step S42), and the control device 3 receives a control permission signal from the robot control device 4 (step S31), and then the control device 3 performs fine movement processing until the convergence condition is satisfied (steps S32 to S33), and after the convergence condition is satisfied, the control device 3 controls the end effector 5 so that a predetermined process is performed on the target object OBJ (step S34). Note that, as shown in FIG.
  • step S34 the control device 3 performs fine movement processing during a processing period in which a predetermined process is performed on the target object OBJ (i.e., a processing period in which the control device 3 controls the end effector 5). Therefore, in the fifth modified example, unless otherwise specified, the processing period may mean a period in which the end effector 5 performs a predetermined process on the target object OBJ and the control device 3 performs fine movement processing.
  • step S372e determines whether or not an imaging abnormality condition is met.
  • the control device 3 may determine that the imaging abnormal condition is satisfied when the target object OBJ is not reflected in the image represented by the image data IMG generated by the imaging device 21 during the processing period. In other words, the control device 3 may determine that the imaging abnormal condition is satisfied when the imaging device 21 generates image data IMG representing an image in which the target object OBJ is not reflected during the processing period. On the other hand, for example, the control device 3 may determine that the imaging abnormal condition is not satisfied when the target object OBJ is reflected in the image represented by the image data IMG generated by the imaging device 21 during the processing period.
  • control device 3 may determine that the imaging abnormal condition is not satisfied when the imaging device 21 generates image data IMG representing an image in which the target object OBJ is reflected during the processing period.
  • the imaging abnormal condition in which the target object OBJ is not included in the imaging field of view of the imaging device 21 during the processing period may be considered to be equivalent to the condition in which the target object OBJ is not reflected in the image represented by the image data IMG generated during the processing period.
  • the control device 3 may determine that the imaging abnormal condition has been established when at least a portion of the target object OBJ is not reflected in the image represented by the image data IMG generated by the imaging device 21 during the processing period. In other words, the control device 3 may determine that the imaging abnormal condition has been established when the imaging device 21 generates image data IMG representing an image in which at least a portion of the target object OBJ is not reflected during the processing period. On the other hand, for example, the control device 3 may determine that the imaging abnormal condition has not been established when the entire target object OBJ is reflected in the image represented by the image data IMG generated by the imaging device 21 during the processing period.
  • the control device 3 may determine that the imaging abnormal condition has not been established when the imaging device 21 generates image data IMG representing an image in which the entire target object OBJ is reflected during the processing period.
  • the imaging abnormality condition in which at least a portion of the target object OBJ is not included in the imaging field of view of the imaging device 21 during the processing period may be considered equivalent to the condition in which at least a portion of the target object OBJ is not captured in the image represented by the image data IMG generated during the processing period.
  • control device 3 may determine that an abnormal imaging condition has been established if only a small amount of the target object OBJ is captured in the image represented by the image data IMG generated by the imaging device 21 during the processing period. In this case, the matching similarity calculated by the matching process becomes significantly small, so the control device 3 may determine that an abnormal imaging condition has been established if the imaging device 21 generates image data IMG in which only objects with matching similarity lower than a predetermined threshold value are detected.
  • the control device 3 when the control device 3 is unable to detect the target object OBJ by performing a matching process based on the image data IMG generated by the imaging device 21 during the processing period, the control device 3 may determine that the imaging abnormal condition has been established.
  • the state in which the target object OBJ cannot be detected by the matching process may include a state in which the matching process is unable to detect any object that is the target object OBJ or is different from the target object OBJ in the first place.
  • the state in which the target object OBJ cannot be detected by the matching process may include a state in which the matching process is able to detect any object that is the target object OBJ or is different from the target object OBJ, but is unable to detect an object whose matching similarity exceeds a matching threshold (i.e., the target object OBJ).
  • the control device 3 may determine that the imaging abnormal condition has not been established.
  • the state in which the target object OBJ can be detected by the matching process may include a state in which the matching process can detect an object (i.e., the target object OBJ) whose matching similarity exceeds a matching threshold.
  • the imaging abnormality condition in which the target object OBJ is not included in the imaging field of view of the imaging device 21 during the processing period may be considered equivalent to a condition in which the target object OBJ cannot be detected by the matching process performed based on the image data IMG during the processing period.
  • step S372e If it is determined in step S372e that the imaging abnormality condition is not satisfied (step S372e: No), there is a high possibility that the target object OBJ is captured in the image represented by the latest image data IMG generated by the imaging device 21. Therefore, the control device 3 should be able to calculate at least one of the position and orientation of the target object OBJ based on the latest image data IMG generated by the imaging device 21. In this case, the control device 3 does not need to perform a process of estimating at least one of the position and orientation of the target object OBJ (step S373e described below). In other words, the control device 3 may perform a process of calculating at least one of the position and orientation of the target object OBJ based on the image data IMG as usual, as at least a part of the fine movement process.
  • control device 3 continues to control the end effector 5 to perform a predetermined process on the target object OBJ while performing the fine movement process (step S34).
  • the control device 3 performs a process of calculating at least one of the position and orientation of the target object OBJ based on the latest image data IMG generated by the imaging device 21, and generates a control signal based on the result of the calculation of at least one of the position and orientation of the target object OBJ in step S373e.
  • step S372e determines whether the imaging abnormality condition is satisfied. If it is determined in step S372e that the imaging abnormality condition is satisfied (step S372e: Yes), the target object OBJ may not appear in the image represented by the latest image data IMG generated by the imaging device 21. Alternatively, the target object OBJ may only appear slightly in the image represented by the latest image data IMG generated by the imaging device 21. For this reason, the control device 3 may not be able to calculate at least one of the position and orientation of the target object OBJ based on the latest image data IMG generated by the imaging device 21.
  • the control device 3 performs a process of estimating at least one of the position and orientation of the target object OBJ using another method instead of a process of directly calculating at least one of the position and orientation of the target object OBJ based on the latest image data IMG generated by the imaging device 21 (step S373e).
  • the control device 3 estimates (i.e., calculates) at least one of the position and orientation of the target object OBJ at the first time using another method.
  • the control device 3 may estimate (i.e., calculate) at least one of the position and orientation of the target object OBJ at the first time without using the image data IMG generated by the imaging device 21 at the first time.
  • control device 3 may estimate at least one of the position and orientation of the target object OBJ at the first time based on a calculation result of at least one of the position and orientation of the target object OBJ at the second time calculated based on image data IMG generated by the imaging device 21 at a second time prior to the first time.
  • the second time is, for example, a time during the above-mentioned processing period and at which the imaging abnormality condition is not satisfied.
  • the control device 3 may use at least one of the position and orientation of the target object OBJ at the second time as at least one of the position and orientation of the target object OBJ at the first time.
  • the process of estimating at least one of the position and orientation of the target object OBJ at the first time based on at least one of the position and orientation of the target object OBJ at the second time may include a process of reusing at least one of the position and orientation of the target object OBJ at the second time as at least one of the position and orientation of the target object OBJ at the first time.
  • the control device 3 may estimate at least one of the position and orientation of the target object OBJ at the first time based on the calculation result of at least one of the position and orientation of the target object OBJ at the second time and the movement mode of the target object OBJ between the second time and the first time.
  • an example of the movement mode of the target object OBJ is at least one of the movement speed of the target object OBJ, the movement direction of the target object OBJ, the movement time of the target object OBJ, and the movement distance of the target object OBJ.
  • the control device 3 may estimate at least one of the position and orientation of the target object OBJ at the first time by correcting the calculation results of at least one of the position and orientation of the target object OBJ at the second time based on the movement mode of the target object OBJ between the second time and the first time.
  • the control device 3 may estimate at least one of the position and orientation of the virtually moved target object OBJ as at least one of the position and orientation of the target object OBJ at the first time by performing a process of virtually moving the target object OBJ at the second time in the movement direction of the target object OBJ between the second time and the first time as a process of correcting the calculation results of at least one of the position and orientation of the target object OBJ at the second time.
  • control device 3 may perform a process of virtually moving the target object OBJ at the second time by the movement distance of the target object OBJ between the second time and the first time as a process of correcting a calculation result of at least one of the position and orientation of the target object OBJ at the second time, thereby estimating at least one of the position and orientation of the virtually moved target object OBJ as at least one of the and orientation of the target object OBJ at the first time.
  • control device 3 may perform a process of virtually moving the target object OBJ at the second time by the movement distance according to the movement speed of the target object OBJ between the second time and the first time as a process of correcting a calculation result of at least one of the position and orientation of the target object OBJ at the second time, thereby estimating at least one of the position and orientation of the virtually moved target object OBJ as at least one of the and orientation of the target object OBJ at the later first time.
  • the control device 3 may first estimate the movement mode of the target object OBJ between the second time and the first time. For example, the control device 3 may estimate the movement mode of the target object OBJ in at least a part of the period before the first time based on at least one calculation result of the position and the orientation of the target object OBJ at each of a plurality of different second times before the first time.
  • control device 3 may estimate the direction from the position of the target object OBJ at one time to the position of the target object OBJ at another time as the movement direction of the target object OBJ in the period between the one time and the other time.
  • control device 3 may estimate the difference between the position of the target object OBJ at one time and the position of the target object OBJ at the other time as the movement distance of the target object OBJ in the period between the one time and the other time.
  • control device 3 may estimate the moving speed of the target object OBJ in the period between the one time and the other time by dividing the difference between the position of the target object OBJ at one time and the position of the target object OBJ at another time by the time difference between the one time and the other time. Then, using the premise that the moving manner of the target object OBJ moving by being transported by the transport device does not change significantly over time, the control device 3 may estimate the moving manner of the target object OBJ between the second time and the first time based on the estimated moving manner of the target object OBJ in at least a part of the period before the first time.
  • control device 3 may estimate at least one of the position and the orientation of the target object OBJ at the first time based on the calculated result of at least one of the position and the orientation of the target object OBJ at the second time and the estimated moving manner of the target object OBJ between the second time and the first time.
  • control device 3 continues to control the end effector 5 to perform a predetermined process on the target object OBJ while performing the fine movement process (step S34).
  • the control device 3 does not perform a process of calculating at least one of the position and orientation of the target object OBJ based on the latest image data IMG generated by the imaging device 21 as the fine movement process, but generates a control signal based on the estimation result of at least one of the position and orientation of the target object OBJ in step S373e.
  • the control device 3 can generate a control signal for controlling the robot 1 to appropriately move the end effector 5.
  • the control device 3 can generate a control signal for controlling the robot 1 to make the end effector 5 follow the target object OBJ.
  • the control device 3 can generate a control signal for controlling the robot 1 to maintain the positional relationship between the end effector 5 and the target object OBJ at a predetermined target positional relationship.
  • the control device 3 can generate a control signal for controlling the robot 1 to maintain the posture relationship between the end effector 5 and the target object OBJ at a predetermined target posture relationship.
  • the control device 3 estimates at least one of the position and the orientation of the target object OBJ.
  • the control device 3 may estimate at least one of the position and the orientation of the target object OBJ. Specifically, the control device 3 may determine whether or not the image capturing abnormal condition is satisfied during the period when the fine movement process is being performed in step S32 of FIG. 18.
  • control device 3 may determine whether or not the image capturing abnormal condition is satisfied before it is determined in step S33 of FIG. 18 that the convergence condition is satisfied.
  • the control device 3 may determine whether or not the image capturing abnormal condition is satisfied before the control device 3 starts to control the end effector 5 so that the end effector 5 performs a predetermined process on the target object OBJ in step S34 of FIG. 18. Thereafter, when it is determined that the imaging abnormal condition is satisfied, the control device 3 may estimate at least one of the position and orientation of the target object OBJ. For example, when it is determined that the imaging abnormal condition is satisfied at a first time during the period in which the fine movement process is being performed in step S32 of FIG.
  • control device 3 may estimate (i.e., calculate) at least one of the position and orientation of the target object OBJ at the first time based on a calculation result of at least one of the position and orientation of the target object OBJ calculated based on image data IMG generated by the imaging device 21 capturing an image of the target object OBJ at a second time prior to the first time.
  • the control device 3 may estimate at least one of the position and orientation of the target object OBJ when an abnormal imaging condition is met. In other words, when the control device 3 starts fine movement processing without the robot control device 4 performing rough movement processing, the control device 3 may estimate at least one of the position and orientation of the target object OBJ when an abnormal imaging condition is met.
  • control device 3 may perform processing to generate a control signal based on information regarding contact between the end effector 5 and the target object OBJ when the imaging abnormality condition is satisfied as described above.
  • the robot 1 (particularly the robot arm 12) may be equipped with a force sensor for detecting contact between the end effector 5 and the target object OBJ.
  • the end effector 5 may be equipped with a force sensor for detecting contact between the end effector 5 and the target object OBJ.
  • the force sensor may include a sensor that directly detects contact between the end effector 5 and the target object OBJ.
  • the force sensor may include a sensor that indirectly detects contact between the end effector 5 and the target object OBJ by detecting stress applied to the end effector 5 due to contact between the end effector 5 and the target object OBJ.
  • the force sensor may include a sensor that indirectly detects contact between the end effector 5 and the target object OBJ by detecting stress applied to the robot 1 to which the end effector 5 is attached due to contact between the end effector 5 and the target object OBJ.
  • the force sensor may be referred to as a contact sensor.
  • control device 3 may generate a control signal based on the detection result of the force sensor. For example, the control device 3 may determine whether or not the end effector 5 and the target object OBJ are in contact with each other based on the detection result of the force sensor.
  • the control device 3 may generate a control signal for controlling the robot 1 to move the end effector 5 closer to the target object OBJ.
  • the control device 3 may generate a control signal for controlling the robot 1 so that the end effector 5 follows the target object OBJ.
  • the control device 3 can generate a control signal for controlling the robot 1 to appropriately move the end effector 5.
  • control device 3 when the control device 3 generates a control signal based on information regarding contact between the end effector 5 and the target object OBJ, it is not necessary to calculate (or estimate, the same applies hereinafter in this paragraph) at least one of the position and orientation of the target object OBJ. However, even when the control device 3 generates a control signal based on information regarding contact between the end effector 5 and the target object OBJ, it may calculate at least one of the position and orientation of the target object OBJ and generate a control signal based on the calculation result of at least one of the position and orientation of the target object OBJ.
  • control device 3 may calculate at least one of the position and orientation of the target object OBJ based on information regarding contact between the end effector 5 and the target object OBJ, and generate a control signal based on the calculation result of at least one of the position and orientation of the target object OBJ.
  • control device 3 may generate a control signal based on information regarding contact between the end effector 5 and the target object OBJ.
  • the control device 3 may generate a control signal based on information regarding contact between the end effector 5 and the target object OBJ.
  • the control device 3 performs processing to generate a control signal based on information regarding contact between the end effector 5 and the target object OBJ when the imaging abnormal condition is satisfied as described above.
  • the control device 3 may perform processing to generate a control signal based on information regarding contact between the end effector 5 and the target object OBJ even when the imaging abnormal condition is not satisfied.
  • the control device 3 can calculate the positional relationship between the end effector 5 and the target object OBJ based on information regarding contact between the end effector 5 and the target object OBJ.
  • a tool capable of performing machining processing to spindle-machine an object may be attached to the robot 1 as the end effector 5.
  • the robot 1 may machine the target object OBJ by bringing the tool into contact with the target object OBJ while rotating the tool.
  • the tool performing spindle machining as the end effector 5 may be a tool capable of machining an object.
  • the tool capable of machining may be a cutting tool or a grinding tool.
  • machining the object may be considered as a predetermined process performed by the end effector 5.
  • step S33 a flowchart showing the flow of the robot control process in the sixth modified example, after it is determined that the convergence condition is satisfied (step S33: Yes), the control device 3 may perform a process of controlling the end effector 5 (tool) to machine the target object OBJ (step S34f) as a process of controlling the end effector 5 to perform a predetermined process on the target object OBJ (step S34 in FIG. 9).
  • the control device 3 may control the processing conditions of the tool. Specifically, the control device 3 may calculate at least one of the position and orientation of the target object OBJ based on the image data IMG generated during the processing period, and control the processing conditions based on the calculation result of at least one of the position and orientation of the target object OBJ.
  • control device 3 may calculate at least one of the position and orientation of each of the target object OBJ and the tool (i.e., the end effector 5) based on the image data IMG generated during the processing period, and control the processing conditions based on the calculation result of at least one of the position and orientation of each of the target object OBJ and the tool.
  • the operation of calculating at least one of the position and orientation of the tool (i.e., the end effector 5) has already been described in the fourth modified example.
  • An example of a machining condition of the tool is the rotational speed of the tool.
  • the control device 3 may control the rotational speed of the tool based on at least one of the calculation results of the position and orientation of the target object OBJ and the tool.
  • the control device 3 may calculate the positional relationship between the tool and the target object OBJ based on at least one of the calculation results of the position and orientation of the target object OBJ and the tool, and may estimate the progress of the machining using the tool based on the calculation result of the positional relationship between the tool and the target object OBJ. Thereafter, the control device 3 may control the rotational speed of the tool based on the estimation result of the progress of the machining using the tool.
  • the control device 3 may control the rotational speed of the tool to be a relatively low first rotational speed in order to prioritize an increase in torque over a reduction in the time required for machining. For example, after a certain period of time has passed since the start of machining using the tool, the control device 3 may control the rotational speed of the tool to be a relatively high second rotational speed in order to prioritize a reduction in the time required for machining over an increase in torque. For example, the control device 3 may control the rotation speed of the tool so that the longer the time that has elapsed since the start of machining using the tool, the higher the rotation speed of the tool is in order to prioritize shortening the time required for machining over increasing the torque. As a result, the robot system SYS can efficiently machine the target object OBJ.
  • the control device 3 may control the machining conditions based on the detection result of the force sensor. In other words, the control device 3 may control the machining conditions based on information regarding the contact between the tool and the target object OBJ.
  • the control device 3 may determine that machining using the tool has started, and control the rotation speed of the tool to be a relatively low first rotation speed in order to prioritize an increase in torque over a reduction in the time required for machining. After a certain period of time has elapsed since the state of the tool switched to a state in which the tool is in contact with the target object OBJ, the control device 3 may control the rotation speed of the tool to be a relatively high second rotation speed in order to prioritize a reduction in the time required for machining over an increase in torque.
  • control device 3 may control the rotation speed of the tool so that the longer the time that has elapsed since the state of the tool was switched to a state in which the tool is in contact with the target object OBJ, the higher the rotation speed of the tool is in order to prioritize shortening the time required for machining over increasing the torque.
  • the robot system SYS can efficiently machine the target object OBJ.
  • the control device 3 may control the processing conditions based on the detection result of the audio sensor. For example, the sound generated when the tool is not in contact with the target object OBJ is different from the sound generated when the tool is in contact with the target object OBJ. For this reason, the control device 3 may determine whether or not the tool is in contact with the target object OBJ based on the detection result of the audio sensor. Thereafter, the control device 3 may control the processing conditions based on the determination result of whether or not the tool is in contact with the target object OBJ. Note that the process of controlling the processing conditions based on the determination result of whether or not the tool is in contact with the target object OBJ may be the same as the process of controlling the processing conditions based on the detection result of the force sensor.
  • an audio sensor e.g., a microphone
  • control device 3 may control the processing conditions. In other words, when the control device 3 starts fine movement processing without the robot control device 4 performing rough movement processing, the control device 3 may control the processing conditions.
  • control device 3 may inspect the target object OBJ based on image data IMG generated by the imaging device 21.
  • control device 3 may inspect the target object OBJ based on image data IMG generated by the imaging device 21 capturing an image of the target object OBJ (i.e., image data IMG indicating an image in which the target object OBJ is captured).
  • the control device 3 may inspect the target object OBJ after the convergence condition is satisfied (step S33: Yes). In other words, the control device 3 may inspect the target object OBJ after the fine movement process is completed (step S33: Yes). In this case, the control device 3 may inspect the target object OBJ based on image data IMG acquired for performing the fine movement process. The control device 3 may inspect the target object OBJ based on image data IMG acquired separately from the image data IMG acquired for performing the fine movement process. For example, the control device 3 may inspect the target object OBJ based on image data IMG acquired for inspecting the target object OBJ.
  • the target object OBJ inspected by the control device 3 may be the same as the target object OBJ that was the target of the fine movement process (i.e., the target object OBJ imaged by the imaging device 21 for the fine movement process).
  • the target object OBJ inspected by the control device 3 may be different from the target object OBJ that was the target of the fine movement process.
  • the imaging device 21 for the fine movement process may image the hole HL, while the control device 3 may inspect the workpiece W fitted into the hole HL after the convergence condition is met.
  • the control device 3 may inspect the target object OBJ before starting to control the end effector 5 to perform a predetermined process on the target object OBJ in step S34 (step S371g). In other words, the control device 3 may inspect the target object OBJ before the end effector 5 starts the predetermined process (step S371g). In this case, the control device 3 may be considered to be performing a pre-inspection of the target object OBJ.
  • step S371g the control device 3 may inspect the target object OBJ to be held by the end effector 5. In other words, in step S371g, the control device 3 may inspect the target object OBJ for which the holding process is to be performed.
  • the control device 3 may check whether the target object OBJ to be held by the end effector 5 is in a designed state (i.e., an expected state).
  • the designed state may include a state that the end effector 5 can hold.
  • the control device 3 may check whether the target object OBJ to be held by the end effector 5 is in a state that the end effector 5 can hold.
  • the state of the target object OBJ may include the shape of the target object OBJ.
  • the control device 3 may inspect whether the target object OBJ to be held by the end effector 5 has a designed shape (i.e., an expected shape).
  • the designed shape may include a shape that the end effector 5 can hold.
  • the control device 3 may inspect whether the target object OBJ to be held by the end effector 5 has a shape that the end effector 5 can hold.
  • a state in which the target object OBJ does not have the designed shape may include a state in which a part of the target object OBJ is missing or deformed.
  • the state of the target object OBJ may include the size of the target object OBJ.
  • the control device 3 may check whether the target object OBJ to be held by the end effector 5 has a designed size (i.e., an expected size).
  • the designed size may include a size that the end effector 5 can hold.
  • the control device 3 may check whether the target object OBJ to be held by the end effector 5 has a size that the end effector 5 can hold.
  • the state of the target object OBJ may include the posture of the target object OBJ.
  • the control device 3 may check whether the target object OBJ to be held by the end effector 5 has a designed posture (i.e., an expected posture).
  • the designed posture may include a posture that the end effector 5 can hold.
  • the control device 3 may check whether the target object OBJ to be held by the end effector 5 has a posture that the end effector 5 can hold.
  • step S371g the control device 3 may inspect the first target object OBJ held by the end effector 5. In other words, in step S371g, the control device 3 may inspect the first target object OBJ for which the release process is to be performed.
  • the control device 3 may inspect whether the first target object OBJ held by the end effector 5 is in a designed state (i.e., an expected state). For example, the control device 3 may inspect whether the first target object OBJ held by the end effector 5 has a designed shape (i.e., an expected shape). For example, the control device 3 may inspect whether the first target object OBJ held by the end effector 5 has a designed size (i.e., an expected size). For example, the control device 3 may inspect whether the first target object OBJ held by the end effector 5 has a designed orientation (i.e., an expected orientation).
  • step S371g the control device 3 may inspect the second target object OBJ onto which the first target object OBJ held by the end effector 5 is to be released. In other words, in step S371g, the control device 3 may inspect the second target object OBJ on which the release process is to be performed.
  • the control device 3 may check whether the second target object OBJ is in a designed state (i.e., an expected state).
  • the designed state may include a state in which the first target object OBJ can be released (e.g., a state in which the first target object OBJ can be placed, fitted, or inserted).
  • the control device 3 may check whether the second target object OBJ is in a state in which the first target object OBJ can be released (e.g., a state in which the first target object OBJ can be placed, fitted, or inserted).
  • the control device 3 may check whether the hole exists.
  • the control device 3 may inspect whether the second target object OBJ has a designed shape (i.e., an expected shape).
  • the designed shape may include a shape in which the first target object OBJ can be released (e.g., a shape in which the first target object OBJ can be placed, fitted, or inserted).
  • the control device 3 may inspect whether the second target object OBJ has a shape in which the first target object OBJ can be released (e.g., a shape in which the first target object OBJ can be placed, fitted, or inserted).
  • the control device 3 may check whether the second target object OBJ has a design size (i.e., an expected size).
  • the design size may include a size at which the first target object OBJ can be released (e.g., a size at which the first target object OBJ can be placed, fitted, or inserted).
  • the control device 3 may check whether the second target object OBJ has a size at which the first target object OBJ can be released (e.g., a size at which the first target object OBJ can be placed, fitted, or inserted).
  • the control device 3 may check whether the second target object OBJ has a design orientation (i.e., an expected orientation).
  • the design orientation may include an orientation in which the first target object OBJ can be released (e.g., an orientation in which the first target object OBJ can be placed, fitted, or inserted).
  • the control device 3 may check whether the second target object OBJ has an orientation in which the first target object OBJ can be released (e.g., an orientation in which the first target object OBJ can be placed, fitted, or inserted).
  • the control device 3 may inspect the multiple target portions of the second target object OBJ into which the end effector 5 releases the multiple first target objects OBJ.
  • the inspection of the target object OBJ may include the inspection of the target portions of the target object OBJ.
  • the inspection of the target object OBJ may include the inspection of a portion of the target object OBJ.
  • the control device 3 may inspect whether the hole HL is in a designed state (i.e., an expected state).
  • the designed state may include a state in which the first workpiece W1 can be inserted.
  • the control device 3 may inspect whether the hole HL is in a state in which the first workpiece W1 can be inserted.
  • the control device 3 may inspect the presence or absence of the hole HL. For example, the control device 3 may inspect whether the hole HL has a designed shape (i.e., an expected shape). The designed shape may include a shape in which the first workpiece W1 can be inserted. In this case, the control device 3 may inspect whether the hole HL has a shape in which the first workpiece W1 can be inserted. For example, the control device 3 may inspect whether the hole HL has a design size (i.e., an expected size). The design size may include a size into which the first workpiece W1 can be inserted. In this case, the control device 3 may inspect whether the hole HL has a size into which the first workpiece W1 can be inserted.
  • a design size i.e., an expected size
  • the design size may include a size into which the first workpiece W1 can be inserted. In this case, the control device 3 may inspect whether the hole HL has a size into which the first workpiece W1 can be inserted.
  • control device 3 may inspect whether the hole HL has a design orientation (i.e., an expected orientation).
  • the design orientation may include an orientation into which the first workpiece W1 can be inserted.
  • the control device 3 may inspect whether the hole HL has an orientation into which the first workpiece W1 can be inserted.
  • the control device 3 may determine whether or not an abnormality has occurred in the target object OBJ by inspecting the target object OBJ. For example, when the state of the target object OBJ is not the designed state, it may be determined that an abnormality has occurred in the target object OBJ. As an example, when the target object OBJ does not have the designed shape, it may be determined that an abnormality has occurred in the target object OBJ. As an example, when the target portion of the target object OBJ does not have the designed shape, it may be determined that an abnormality has occurred in the target object OBJ. As an example, when the target object OBJ does not have the designed size, it may be determined that an abnormality has occurred in the target object OBJ.
  • the target portion of the target object OBJ when the target portion of the target object OBJ does not have the designed size, it may be determined that an abnormality has occurred in the target object OBJ. As an example, when the target object OBJ does not have the designed orientation, it may be determined that an abnormality has occurred in the target object OBJ. As an example, when the target portion of the target object OBJ does not have the designed orientation, it may be determined that an abnormality has occurred in the target object OBJ.
  • the control device 3 may start processing to control the end effector 5 to perform a predetermined process on the target object OBJ.
  • the control device 3 may not start processing to control the end effector 5 to perform a predetermined process on the target object OBJ. In this case, the control device 3 may notify the user that an abnormality has occurred in the target object OBJ.
  • the control device 3 may start processing to control the end effector 5 to perform a predetermined process on the target object OBJ.
  • the control device 3 may inspect the target object OBJ after finishing controlling the end effector 5 to perform a predetermined process on the target object OBJ in step S34 (step S372g). That is, the control device 3 may inspect the target object OBJ after the end effector 5 completes the predetermined process (step S372g). In this case, the control device 3 may be considered to be performing a post-inspection of the target object OBJ. In this case, the control device 3 may inspect the target object OBJ based on image data IMG acquired after finishing controlling the end effector 5 to perform a predetermined process on the target object OBJ in step S34.
  • the control device 3 may inspect the target object OBJ held by the end effector 5. For example, the control device 3 may inspect whether the target object OBJ is being appropriately held by the end effector 5. For example, the control device 3 may inspect whether the target object OBJ held by the end effector 5 has a design orientation (i.e., an expected orientation).
  • the control device 3 may inspect at least one of the first target object OBJ released by the end effector 5 and the second target object OBJ from which the first target object OBJ has been released. For example, the control device 3 may inspect whether the first target object OBJ has been appropriately released from the end effector 5. As an example, the control device 3 may inspect the presence or absence of the first target object OBJ on the second target object OBJ. In other words, the control device 3 may inspect whether the released first target object OBJ exists on the second target object OBJ.
  • the control device 3 may inspect whether the released first target object OBJ has been released to a design position (i.e., an expected position). For example, the control device 3 may check whether the released first target object OBJ is released at a designed position (i.e., an expected position) of the second target object OBJ. For example, the control device 3 may check whether the released first target object OBJ is fitted (inserted) into an expected position of the second target object OBJ. For example, the control device 3 may check whether the released first target object OBJ has a designed orientation (i.e., an expected orientation).
  • the control device 3 may check whether the released first target object OBJ has a designed orientation (i.e., an expected orientation) with respect to the second target object OBJ. For example, the control device 3 may check whether the positional relationship between the released first target object OBJ and the second target object OBJ from which the second target object OBJ was released is a designed positional relationship (i.e., an expected positional relationship). For example, the control device 3 may check whether the orientation relationship between the released first target object OBJ and the released second target object OBJ is the designed orientation relationship (e.g., the expected orientation relationship).
  • a designed orientation i.e., an expected orientation
  • the control device 3 may inspect whether the shape of the machined target object OBJ (i.e., the shape after machining) is the designed shape (i.e., the expected shape). The control device 3 may inspect whether the size of the machined target object OBJ (i.e., the size after machining) is the designed size (i.e., the expected size). The control device 3 may inspect whether the target object OBJ has been machined in the expected machining manner. Alternatively, the control device 3 may perform a similar inspection even when the end effector 5 machines the target object OBJ by irradiating the target object OBJ with processing light.
  • the inspection in step S372g may be considered to be equivalent to an inspection of the results of a predetermined process performed by the end effector 5.
  • the process of inspecting whether the target object OBJ is being properly held by the end effector 5 and the process of inspecting whether the posture of the target object OBJ held by the end effector 5 has the designed posture (i.e., the expected posture) may each be considered to be equivalent to a process of inspecting the results of the holding process.
  • the following steps are performed: a process of inspecting whether the first target object OBJ has been appropriately released from the end effector 5; a process of inspecting whether the released first target object OBJ has been released to a designed position (i.e., an expected position); a process of inspecting whether the released first target object OBJ has been released to a designed position of the second target object OBJ (i.e., an expected position); a process of inspecting whether the released first target object OBJ has been fitted (inserted) into the expected position of the second target object OBJ;
  • a designed orientation i.e., an expected orientation
  • each of the process of inspecting whether the shape of the machined target object OBJ i.e., the shape after machining
  • the process of inspecting whether the size of the machined target object OBJ i.e., the size after machining
  • the process of inspecting whether the target object OBJ has been machined in an expected machining manner may be considered to be equivalent to a process of inspecting the result of the machining process.
  • the above-mentioned inspection may be considered to be equivalent to an inspection of the result of processing using processing light.
  • the control device 3 may determine whether the end effector 5 has normally completed the specified processing by inspecting the target object OBJ. In other words, the control device 3 may determine whether the result of the processing performed by the end effector 5 is as expected by inspecting the target object OBJ. If it is determined as a result of the inspection in step S372g that the end effector 5 has not normally completed the specified processing (in other words, the result of the processing performed by the end effector 5 is not the expected result), the control device 3 may notify the user that the end effector 5 has not normally completed the specified processing.
  • the control device 3 inspects the target object OBJ after the convergence condition is satisfied. In other words, the control device 3 inspects the target object OBJ after the fine movement process is completed.
  • the period during which the control device 3 inspects the target object OBJ is not limited to the above-mentioned period.
  • the control device 3 may inspect the target object OBJ while the fine movement process is being performed.
  • the control device 3 may inspect the target object OBJ before the fine movement process is started.
  • the control device 3 may inspect the target object OBJ after the rough movement process is completed and before the fine movement process is started.
  • the control device 3 may inspect the target object OBJ while the rough movement process is being performed.
  • the control device 3 may inspect the target object OBJ before the rough movement process is started.
  • the control device 3 inspects the target object OBJ.
  • a device other than the control device 3 may acquire image data IMG from the imaging device 21 and inspect the target object OBJ based on the acquired image data IMG.
  • the robot control device 4 may acquire image data IMG from the imaging device 21 and inspect the target object OBJ based on the acquired image data IMG.
  • the control device 3 inspects the target object OBJ based on the image data IMG generated by the imaging device 21.
  • the control device 3 may inspect the target object OBJ using a method different from the method of inspecting the target object OBJ based on the image data IMG generated by the imaging device 21.
  • the control device 3 may inspect the target object OBJ based on the measurement results of the target object OBJ by a measuring device 24g capable of measuring the target object OBJ.
  • the measuring device 24g may be a measuring device provided in the robot system SYS, or may be a measuring device independent of the robot system SYS.
  • the measuring device 24g may be capable of measuring the characteristics of the target object OBJ.
  • the characteristics of the target object OBJ include at least one of the following: the size of the target object OBJ, the position of the target object OBJ, the distance from the measuring device 24g to the target object OBJ, the shape of the target object OBJ, the optical characteristics of the target object OBJ (e.g., at least one of reflectance and transmittance), the electrical characteristics of the target object OBJ (e.g., at least one of electrical resistance and electrical conductivity), and the thermal characteristics of the target object OBJ (e.g., at least one of temperature and thermal conductivity).
  • the measuring device 24g may be any measuring device as long as it is capable of measuring the target object OBJ.
  • the measuring device 24g may be capable of measuring the target object OBJ by contacting the target object OBJ.
  • the measuring device 24g may be capable of measuring the target object OBJ using a probe that can contact the target object OBJ.
  • the measuring device 24g may be capable of measuring the target object OBJ by irradiating the target object OBJ with measuring light and detecting return light of the measuring light from the target object OBJ (e.g., at least one of reflected light, scattered light, diffracted light, and transmitted light).
  • the measuring device 24g may be an imaging device equipped with multiple monocular cameras.
  • the measuring device 24g may be a stereo camera equipped with two monocular cameras.
  • the measuring device 24g may be attached to the robot 1 (particularly, the robot arm 12) as shown in FIG. 21, which shows an example of the configuration of the robot system SYS in the eighth modified example.
  • the robot 1 may move the measuring device 24g.
  • the robot 1 may move the measuring device 24g so that the measuring device 24g moves to a position where the measuring device 24g can measure the target object OBJ.
  • the measuring device 24g may be attached to or placed on a device that is different from the robot 1 and that can move the measuring device 24g (for example, an automatic guided vehicle or another robot different from the robot 1).
  • the measuring device 24g may not be movable.
  • the measuring device 24g may be attached to a fixed member different from the robot 1.
  • the measuring device 24g may be attached to a member that is fixed in a position where the target object OBJ can be overlooked.
  • the control device 3 may inspect the target object OBJ. That is, when the control device 3 starts the fine movement process without the robot control device 4 performing the rough movement process, the control device 3 may inspect the target object OBJ. Alternatively, even if the control device 3 does not perform the fine movement process, the control device 3 may inspect the target object OBJ. That is, when the robot control device 4 starts the rough movement process without the control device 3 performing the fine movement process, the control device 3 may inspect the target object OBJ. Alternatively, the control device 3 may inspect the target object OBJ independently of the control of the robot 1. That is, the control device 3 may inspect the target object OBJ independently of the fine movement process by the control device 3 and the rough movement process by the robot control device 4.
  • the control device 3 may perform the fine movement process under the condition that the target object OBJ is moving (transported) by the transport device.
  • the control device 3 usually waits for a certain period of time until the target object OBJ moving (transported) by the transport device moves from outside the imaging field of the imaging device 21 into the imaging field of the imaging device 21.
  • the control device 3 After that, after the target object OBJ moving (transported) by the transport device moves into the imaging field of the imaging device 21, the control device 3 generates a control signal for controlling the robot 1 so that the end effector 5 follows the target object OBJ based on the image data IMG of the target object OBJ that has moved into the imaging field of the imaging device 21.
  • the target object OBJ which would have been transported to move into the imaging field of view of the imaging device 21 if the transport device had not stopped, may stop outside the imaging field of view of the imaging device 21.
  • the target object OBJ which would normally move into the imaging field of view of the imaging device 21 within a certain period of time, may not move into the imaging field of view of the imaging device 21 even after the certain period of time has passed.
  • the control device 3 cannot calculate at least one of the position and orientation of the target object OBJ as long as the transport device continues to stop.
  • the control device 3 may perform, as at least part of the fine movement process, a process of generating a control signal for controlling the robot 1 to move the imaging device 21 relative to the stopped target object OBJ (i.e., to move the imaging device 21 together with the end effector 5).
  • the control device 3 may perform, as at least part of the fine movement process, a process of moving the imaging device 21 so that the stopped target object OBJ is positioned within the imaging field of the imaging device 21.
  • the control device 3 may perform, as at least part of the fine movement process, a process of generating a control signal for controlling the robot 1 so that the stopped target object OBJ is positioned within the imaging field of the moving imaging device 21.
  • the control device 3 may not be able to generate a control signal for controlling the robot 1 to move the imaging device 21 toward the stopped target object OBJ based on the image data IMG.
  • the control device 3 may not be able to generate a control signal for controlling the robot 1 to move the imaging device 21 toward the stopped target object OBJ so that the stopped target object OBJ is located within the imaging field of view of the imaging device 21.
  • the control device 3 performs a process of generating a control signal for controlling the robot 1 to move the imaging device 21 toward the stopped target object OBJ, based on the state of the target object OBJ being transported by the transport device, instead of the image data IMG.
  • the fine movement process of the eighth modified example which includes a process of generating a control signal based on the state of the target object OBJ being transported by the transport device, will be described.
  • FIG. 22 is a flowchart showing the flow of the fine movement process in the eighth modified example. Note that the transport state may also be referred to as the movement state of the target object OBJ by the transport device.
  • control device 3 also acquires image data IMG from the imaging device 21 (step S321), and calculates at least one of the position and orientation of the target object OBJ based on the image data IMG (step S322).
  • control device 3 determines whether or not at least one of the position and orientation of the target object OBJ was able to be calculated in step S322 (step S325h).
  • the signal generating unit 312 determines whether or not the position and orientation information POI was able to be generated in step S322 (step S325h).
  • step S325h If it is determined in step S325h that at least one of the position and orientation of the target object OBJ could be calculated (i.e., the position and orientation information POI could be calculated) (step S325h: Yes), it is unlikely that the target object OBJ is not reflected in the image indicated by the image data IMG used to calculate at least one of the position and orientation of the target object OBJ. In other words, it is unlikely that the target object OBJ is located outside the imaging field of view of the imaging device 21. Therefore, in this case, the control device 3 does not need to perform a process of generating a control signal for controlling the robot 1 to move the imaging device 21 toward the stopped target object OBJ based on the transportation state of the target object OBJ by the transportation device. Therefore, the signal generation unit 312 may generate a control signal using the position and orientation information POI generated in step S322 (step S323) and output the generated control signal (step S324).
  • step S325h if the result of the judgment in step S325h is that at least one of the position and orientation of the target object OBJ could not be calculated (i.e., the position and orientation information POI could not be calculated) (step S325h: No), it is highly likely that the target object OBJ is not reflected in the image indicated by the image data IMG used to calculate at least one of the position and orientation of the target object OBJ. In other words, it is highly likely that the target object OBJ is located outside the imaging field of view of the imaging device 21.
  • the control device 3 performs a process of generating a control signal for controlling the robot 1 to move the imaging device 21 toward the stopped target object OBJ, based on the transport state of the target object OBJ by the transport device (step S326h). More specifically, the control device 3 performs a process of generating a control signal for controlling the robot 1 to move the imaging device 21 toward the stopped target object OBJ based on the state of transportation of the target object OBJ by the transportation device before the transportation device stopped (i.e., before it was determined that at least one of the position and orientation of the target object OBJ cannot be calculated) (step S326h). Furthermore, the control device 3 outputs the generated control signal to the robot control device 4 or the robot 1 (step S326h). As a result, the robot 1 moves the imaging device 21.
  • step S326h the signal generation unit 312 performs processing to generate a control signal for controlling the robot 1 so that the stopped target object OBJ is positioned within the imaging field of the moving imaging device 21.
  • the control device 3 may be considered to be performing processing to generate a control signal for controlling the robot 1 so that the imaging device 21 searches for the stopped target object OBJ.
  • the processing performed in step S326h is referred to as a search movement processing.
  • the fine movement processing may be considered to include the search movement processing.
  • the control device 3 may be considered to be performing the search movement processing as part of the fine movement processing.
  • the signal generating unit 312 may use the manner in which the target object OBJ is transported by the transport device as the transport state of the target object OBJ. In this case, the signal generating unit 312 may acquire transport information related to the manner in which the target object OBJ is transported by the transport device. For example, the signal generating unit 312 may acquire the transport information from the transport device or from a control device that controls the transport device. Then, based on the acquired transport information, the signal generating unit 312 may generate a control signal for controlling the robot 1 to position the robot 1 within the imaging field of the moving imaging device 21.
  • the manner in which the target object OBJ is transported by the transport device may include the direction in which the target object OBJ is transported by the transport device.
  • the signal generating unit 312 may generate a control signal based on the direction in which the target object OBJ is transported by the transport device. Specifically, the signal generating unit 312 may generate a control signal for controlling the robot 1 so that the imaging device 21 moves in a moving direction determined according to the direction in which the target object OBJ is transported. For example, the signal generating unit 312 may generate a control signal for controlling the robot 1 so that the imaging device 21 moves in a moving direction opposite to the direction in which the target object OBJ is transported.
  • the control device 3 may generate a control signal for controlling the robot 1 so that the imaging device 21 moves toward the -X side along the X-axis direction.
  • the control device 3 may generate a control signal for controlling the robot 1 so that the imaging device 21 moves toward the +X side along the X-axis direction.
  • the control device 3 may generate a control signal for controlling the robot 1 to move the imaging device 21 toward the -Y side along the X-axis direction.
  • the control device 3 may generate a control signal for controlling the robot 1 to move the imaging device 21 toward the +Y side along the X-axis direction.
  • the transport direction may be referred to as the movement direction of the target object OBJ.
  • the transport mode may include the movement direction.
  • the target object OBJ that has stopped is positioned within the imaging field of view of the imaging device 21 at a certain timing during the period in which the robot 1 moves the imaging device 21.
  • the imaging device 21 can capture an image of the target object OBJ and generate image data IMG that shows an image in which the target object OBJ is captured (step S321).
  • the control device 3 can calculate at least one of the position and orientation of the target object OBJ (step S322), and generate a control signal based on the calculation result of at least one of the position and orientation of the target object OBJ (step S323).
  • the robot 1 may not be able to perform the specified processing using the end effector 5. This is because the robot 1 is configured to perform the specified processing using the end effector 5 while the target object OBJ is moving. For this reason, the control device 3 may generate a control signal after the stopped target object OBJ is now located within the imaging field of view of the imaging device 21 and the stopped transport device has resumed operation. In other words, the control device 3 may start moving the end effector 5 after the stopped target object OBJ is now located within the imaging field of view of the imaging device 21 and the stopped transport device has resumed operation.
  • the manner in which the target object OBJ is transported by the transport device may include, in addition to or instead of the direction in which the target object OBJ is transported by the transport device, the distance the target object OBJ is transported by the transport device.
  • the distance the target object OBJ is transported by the transport device may mean the distance required for the transport device to transport the target object OBJ so that the target object OBJ moves into the imaging field of view of the imaging device 21, assuming that the transport device is not stopped.
  • control device 3 may repeat, while changing the desired direction, the process of generating a control signal for controlling the robot 1 to move the imaging device 21 in the desired direction by a movement distance equal to the transport distance of the target object OBJ from the current position of the imaging device 21 (specifically, the position of the imaging device 21 at the time when it is determined that at least one of the position and orientation of the target object OBJ cannot be calculated).
  • control device 3 may perform the following processes: generating a control signal for controlling the robot 1 to move the imaging device 21 from the current position of the imaging device 21 toward the +X side along the X-axis direction by a movement distance equal to the transport distance of the target object OBJ; generating a control signal for controlling the robot 1 to move the imaging device 21 from the current position of the imaging device 21 toward the -X side along the X-axis direction by a movement distance equal to the transport distance of the target object OBJ; generating a control signal for controlling the robot 1 to move the imaging device 21 from the current position of the imaging device 21 toward the +Y side along the Y-axis direction by a movement distance equal to the transport distance of the target object OBJ; and generating a control signal for controlling the robot 1 to move the imaging device 21 from the current position of the imaging device 21 toward the -Y side along the Y-axis direction by a movement distance equal to the transport distance of the target object OBJ.
  • the transport distance may be referred to as the travel
  • the stopped target object OBJ is positioned within the imaging field of view of the imaging device 21 at the timing when the imaging device 21 moves along one of the multiple moving directions in which the imaging device 21 moves (specifically, the direction from the current position of the imaging device 21 toward the stopped target object OBJ).
  • the imaging device 21 can capture an image of the target object OBJ and generate image data IMG showing an image in which the target object OBJ is captured (step S321).
  • the control device 3 can calculate at least one of the position and orientation of the target object OBJ (step S322), and generate a control signal based on the calculation result of at least one of the position and orientation of the target object OBJ (step S323).
  • the robot 1 may not be able to perform the specified processing using the end effector 5. This is because the robot 1 is configured to perform the specified processing using the end effector 5 in a situation in which the target object OBJ is moving. For this reason, the control device 3 may generate a control signal after the stopped target object OBJ is positioned within the imaging field of view of the imaging device 21 and the stopped transport device has resumed operation. In other words, the control device 3 may start moving the end effector 5 after the stopped target object OBJ is positioned within the imaging field of view of the imaging device 21 and the stopped transport device has resumed operation.
  • the transport mode of the target object OBJ by the transport device may include the transport speed of the target object OBJ by the transport device in addition to or instead of at least one of the transport direction and transport distance of the target object OBJ by the transport device.
  • the signal generating unit 312 may generate a control signal based on the transport speed of the target object OBJ by the transport device. Specifically, the signal generating unit 312 may estimate a transport distance required for the transport device to transport the target object OBJ so that the target object OBJ moves into the imaging field of view of the imaging device 21 on the assumption that the transport device is not stopped, based on the transport speed of the target object OBJ.
  • control device 3 may repeat the process of generating a control signal for controlling the robot 1 to move the imaging device 21 in the desired direction by the same travel distance as the estimated travel distance from the current position of the imaging device 21, while changing the desired direction.
  • the transport speed may be referred to as the travel speed of the target object OBJ.
  • the transport mode may include the travel speed.
  • the control device 3 may generate a control signal based on both the direction of transport of the target object OBJ by the transport device and the distance (or transport speed) of the target object OBJ by the transport device.
  • the control device 3 can specify the moving direction in which the imaging device 21 should be moved so that the stopped target object OBJ is positioned within the imaging field of the imaging device 21, based on the direction of transport of the target object OBJ by the transport device.
  • the control device 3 can specify the moving distance in which the imaging device 21 should be moved so that the stopped target object OBJ is positioned within the imaging field of the imaging device 21, based on the distance (or transport speed) of the target object OBJ by the transport device.
  • the control device 3 may perform processing to generate a control signal for controlling the robot 1 to move the imaging device 21 from the current position of the imaging device 21 by a moving distance determined according to the transport distance of the target object OBJ, toward a moving direction determined according to the transport direction of the target object OBJ.
  • the control device 3 may perform processing to generate a control signal for controlling the robot 1 to move the imaging device 21 from the current position of the imaging device 21 in a direction opposite to the direction of transport of the target object OBJ by a distance equal to the transport distance of the target object OBJ.
  • control device 3 may generate a control signal for controlling the robot 1 to move the imaging device 21 with pinpoint precision so that the stopped target object OBJ is positioned within the imaging field of view of the imaging device 21.
  • the time required for the stopped target object OBJ to be positioned within the imaging field of view of the imaging device 21 is reduced compared to when a control signal is generated based on either the transport direction of the target object OBJ by the transport device or the transport distance (or transport speed) of the target object OBJ by the transport device.
  • the signal generating unit 312 may specify (in this case, estimate) the transport state of the target object OBJ without acquiring transport information from the transport device or the control device that controls the transport device.
  • the signal generating unit 312 may specify at least one time change amount of the position and posture of the target object OBJ in the past based on at least one calculation result of the position and posture of the target object OBJ calculated in the past.
  • the control device 3 may specify the transport state of the target object OBJ in the past based on at least one time change amount of the position and posture of the target object OBJ in the past.
  • control device 3 may specify the direction from the position of the target object OBJ at one time in the past to the position of the target object OBJ at another time in the past as the moving direction of the target object OBJ in the period between the one time and the other time (i.e., the transport direction).
  • control device 3 may specify the difference between the position of the target object OBJ at one time in the past and the position of the target object OBJ at another time in the past as the moving distance of the target object OBJ in the period between the one time and the other time (i.e., the transport distance).
  • control device 3 may determine the movement speed (i.e., the transport speed) of the target object OBJ in the period between the one time and the other time by dividing the difference between the position of the target object OBJ at a past time and the position of the target object OBJ at a past time by the time difference between the one time and the other time.
  • the control device 3 may inspect the target object OBJ. In other words, when the control device 3 starts the fine movement process without the robot control device 4 performing the rough movement process, the control device 3 may generate a control signal for controlling the robot 1 to move the imaging device 21 toward the stopped target object OBJ, based on the transportation state of the target object OBJ by the transportation device.
  • control device 3 performs a search process to generate a control signal for controlling the robot 1 to move the imaging device 21 to the stopped target object OBJ when the target object OBJ stops moving from outside the imaging field of the imaging device 21 into the imaging field of the imaging device 21 due to the transport device being stopped.
  • control device 3 may perform a search process to generate a control signal for controlling the robot 1 to move the imaging device 21 to the stopped target object OBJ when the target object OBJ, which was located in the imaging field of the imaging device 21, moves out of the imaging field of the imaging device 21 due to the transport device being stopped.
  • the robot 1 is controlled to stop the imaging device 21 in accordance with the stop of the transport device, the target object OBJ, which was located in the imaging field of the imaging device 21, will not move out of the imaging field of the imaging device 21 even if the transport device is stopped.
  • the imaging device 21 may move after the transport device is stopped.
  • the control device 3 may perform a search process to control the robot 1 to move the imaging device 21 so that the target object OBJ that has moved out of the imaging field of view moves into the imaging field of view.
  • control device 3 may perform a search process to generate a control signal for controlling the robot 1 to move the imaging device 21 relative to the target object OBJ, regardless of whether the transport device is stopped.
  • the control device 3 may perform a search process to control the robot 1 to move the imaging device 21 so that the target object OBJ moves into the imaging field of view of the imaging device 21, regardless of whether the transport device is stopped. For example, if the target object OBJ cannot be properly detected by the matching process, the target object OBJ may not be included in the imaging field of view of the imaging device 21. For this reason, the control device 3 may perform a search process when the target object OBJ cannot be properly detected by the matching process.
  • the control device 3 may perform a search process when an object other than the target object OBJ is erroneously detected as the target object OBJ by the matching process.
  • the end effector 5 e.g., a hand clipper or a vacuum clipper
  • the end effector 5 is not limited to a device that performs at least one of a holding process and a release process, and may be a device that performs other processes on the target object OBJ.
  • the robot arm 12 may be equipped with a processing device for processing the target object OBJ, which is an example of the end effector 5.
  • the processing device may perform at least one of an additive process for adding a new object to the target object OBJ, a subtractive process for removing a part of the target object OBJ, a welding process for joining two target objects OBJ, and a cutting process for cutting the target object OBJ.
  • the processing device may process the target object OBJ using a tool. In this case, the processing device including the tool may be attached to the robot arm 12.
  • the processing device may process the target object OBJ by irradiating an energy beam (e.g., light, electromagnetic waves, and charged particle beams) to the target object OBJ.
  • the processing device including an irradiation device for irradiating the target object OBJ with an energy beam may be attached to the robot arm 12.
  • the processing device may perform soldering processing to solder a component to the target object OBJ.
  • the processing device may use a soldering iron to solder the component to the target object OBJ.
  • the processing device including the soldering iron may be attached to the robot arm 12.
  • the processing device may solder the component to the target object OBJ by irradiating the solder with an energy beam (e.g., light, electromagnetic waves, and charged particle beams).
  • the processing device including an irradiation device that irradiates the target object OBJ with an energy beam may be attached to the robot arm 12.
  • the robot arm 12 may be equipped with a measuring device for measuring the target object OBJ, which is an example of the end effector 5.
  • the measuring device may be capable of measuring the characteristics of the target object OBJ. Examples of the characteristics of the target object OBJ include at least one of the shape of the target object OBJ, the size of the target object OBJ, the distance from the measuring device to the target object OBJ, the reflectance of the target object OBJ, the transmittance of the target object OBJ, and the temperature of the target object OBJ.
  • the measuring device may measure the target object OBJ using a touch probe. In this case, a measuring device including a touch probe may be attached to the robot arm 12.
  • the measuring device may measure the target object OBJ by irradiating an energy beam (e.g., light, electromagnetic waves, and charged particle beams) onto the target object OBJ.
  • an energy beam e.g., light, electromagnetic waves, and charged particle beams
  • a measuring device including an irradiation device that irradiates the target object OBJ with an energy beam may be attached to the robot arm 12.
  • the measuring device may be an imaging device equipped with one or more monocular cameras.
  • the measurement device may be a stereo camera equipped with two monocular cameras.
  • the control device 3 may generate a control signal for controlling the operation of at least one of the processing device and the measuring device.
  • the control device 3 may generate a control signal for controlling the rotation of a tool provided in the processing device.
  • the control device 3 may generate a control signal for controlling the on/off of an energy beam provided by an irradiation device provided in at least one of the processing device and the measuring device.
  • the robot arm 12 may be equipped with a discharge device as an example of the end effector 5.
  • the discharge device may discharge at least one of adhesive, sealant, paint, and solder.
  • the discharge device may discharge at least one of adhesive, sealant, paint, and solder toward the target object OBJ.
  • the control device 3 may generate a control signal for controlling at least one of turning on and off the discharge from the discharge device and the discharge amount. Note that, because discharging at least one of adhesive, sealant, paint, and solder onto the target object OBJ can be said to process the target object OBJ, the discharge device may also be called a processing device.
  • the imaging unit 2 includes the lighting device 23.
  • the lighting device 23 may be integrated with the imaging device 21, or may be a device separate from the imaging device 21.
  • the robot 1 may include the lighting device 23, rather than the imaging unit 2. In other words, the lighting device 23 may be attached to the robot 1 independently of the imaging unit 2.
  • the imaging device 21 provided in the imaging unit 2 described above is a monocular camera.
  • the imaging device 21 is not limited to a monocular camera.
  • the imaging device 21 may be a stereo camera capable of capturing an image of the target object OBJ using two monocular cameras.
  • the imaging device 21 may include two monocular cameras.
  • the imaging device 21 may be a light field camera.
  • the imaging device 21 may be a plenoptic camera.
  • the imaging device 21 may be a multispectral camera.
  • the imaging device 21 may be equipped with three or more monocular cameras.
  • the control device 3 may generate three-dimensional position data based on the image data IMG generated by the imaging device 21.
  • the three-dimensional position data may be data indicating the three-dimensional position of the target object OBJ reflected in the image data IMG.
  • the three-dimensional position data may be data indicating the three-dimensional position of each of a plurality of points of the target object OBJ.
  • the three-dimensional position data may be data indicating the three-dimensional shape of the target object OBJ reflected in the image data IMG.
  • An example of the three-dimensional position data is at least one of depth image data and point cloud data.
  • control device 3 may calculate at least one of the position and the orientation of the target object OBJ by performing a matching process using the three-dimensional position data and the three-dimensional model data instead of the matching process using the image data IMG and the above-mentioned model data MDL.
  • the three-dimensional model data is data indicating a three-dimensional model of the target object OBJ.
  • the control device 3 may generate three-dimensional position data based on image data IMG generated by the imaging device 21.
  • the illumination device 23 may project a desired projection pattern onto the target object OBJ by irradiating the target object OBJ with illumination light.
  • the illumination device 23 and the illumination light may be referred to as a projection device and a projection light, respectively.
  • the desired projection pattern may include, for example, a random pattern.
  • the random pattern may be a projection pattern having a different pattern for each unit irradiation area.
  • the random pattern may include a random dot pattern.
  • the desired projection pattern may include, for example, a one-dimensional or two-dimensional grid pattern.
  • the desired projection pattern may include other projection patterns.
  • the imaging device 21 may generate image data IMG by capturing an image of the target object OBJ onto which the projection pattern from the illumination device 23 is projected.
  • the projection pattern reflected in the image represented by the image data IMG reflects the three-dimensional shape of at least a part of the surface of the target object OBJ onto which the projection pattern is projected. Therefore, the control device 3 can calculate the three-dimensional shape of at least a part of the surface of the target object OBJ based on the projection pattern reflected in the image represented by the image data IMG.
  • the three-dimensional shape of at least a part of the surface of the target object OBJ essentially indicates the three-dimensional position of each of the multiple points of the target object OBJ. This is because each of the multiple points of the target object OBJ is included in the surface of the target object OBJ. Therefore, the process of calculating the three-dimensional shape of at least a part of the surface of the target object OBJ may be considered to be substantially equivalent to the process of calculating the three-dimensional position of each of the multiple points of the target object OBJ. Therefore, the control device 3 can generate three-dimensional position data WSD based on the image data IMG generated by the imaging device 21, which is a monocular camera. Thereafter, the control device 3 may calculate at least one of the position and the orientation of the target object OBJ by performing a matching process using the three-dimensional position data and the three-dimensional model data.
  • the lighting device 23 may project a desired projection pattern onto the target object OBJ by irradiating the target object OBJ with illumination light. Thereafter, the control device 3 may generate three-dimensional position data WSD based on the image data IMG generated by the imaging device 21, which is a stereo camera.
  • the imaging unit 2 includes a single imaging device 21.
  • the imaging unit 2 may include multiple imaging devices 21.
  • the imaging unit 2 may include a first imaging device 21 that is a monocular camera and a second imaging device 21 that is a stereo camera.
  • the control device 3 may calculate at least one of the position and orientation of the target object OBJ by performing at least one of a matching process using image data IMG generated by the first imaging device 21 and a matching process using three-dimensional position data generated based on the image data IMG generated by the second imaging device 21.
  • the control device 3 may calculate at least one first provisional value of the position and orientation of the target object OBJ by performing a matching process using the image data IMG generated by the first imaging device 21. Furthermore, the control device 3 may calculate at least one second provisional value of the position and orientation of the target object OBJ by performing a matching process using three-dimensional position data generated based on the image data IMG generated by the second imaging device 21. Thereafter, the control device 3 may calculate at least one final value of the position and orientation of the target object OBJ by performing an arithmetic process using the first provisional value and the second provisional value. In this case, the control device 3 may generate a control signal based on at least one final value of the position and orientation of the target object OBJ.
  • An example of a calculation process for calculating a final value using the first provisional value and the second provisional value is a calculation process for calculating the average value of the first provisional value and the second provisional value as the final value.
  • An example of a calculation process for calculating a final value using the first provisional value and the second provisional value is a calculation process for selecting, as the final value, one of the first provisional value and the second provisional value that corresponds to a higher matching similarity.
  • An example of a calculation process for calculating a final value using the first provisional value and the second provisional value is a calculation process for selecting, as the final value, one of the first provisional value and the second provisional value that corresponds to a matching similarity that exceeds the matching judgment threshold.
  • the control device 3 may calculate at least one of the position and orientation of the target object OBJ by performing at least one of a matching process using image data IMG generated by the imaging device 21 and a matching process using three-dimensional position data generated based on the image data IMG generated by the imaging device 21. Specifically, the control device 3 may calculate at least one of the position and orientation of the target object OBJ by performing a matching process using image data IMG generated by one monocular camera equipped in the imaging device 21.
  • control device 3 may calculate at least one of the position and orientation of the target object OBJ by generating three-dimensional position data based on image data IMG generated by two monocular cameras equipped in the imaging device 21 and performing a matching process using the three-dimensional position data.
  • both the imaging unit 2 and the end effector 5 are attached to the same robot 1.
  • the imaging unit 2 may be attached to a first robot 1, and the end effector 5 may be attached to a second robot 1 different from the first robot 1.
  • the imaging unit 2 may be attached to a robot arm 12 provided in the first robot 1, and the end effector 5 may be attached to a robot arm 12 provided in the second robot 1.
  • the imaging unit 2 and the end effector 5 may be attached to two different robots 1, respectively.
  • both the imaging unit 2 and the end effector 5 are attached to the same robot arm 12. That is, the robot 1 has a robot arm 12 to which both the imaging unit 2 and the end effector 5 are attached.
  • the imaging unit 2 may be attached to a first robot arm 12, and the end effector 5 may be attached to a second robot arm 12 different from the first robot arm 12. That is, the imaging unit 2 and the end effector 5 may be attached to two different robot arms 12, respectively.
  • the robot 1 may have a first robot arm 12 to which the imaging unit 2 is attached, and a second robot arm 12 to which both the end effector 5 are attached.
  • the imaging unit 2 may not be attached to the robot 1.
  • at least one of the imaging device 21 and the lighting device 23 may not be attached to the robot 1.
  • the imaging device 21 may be disposed at any position where the imaging device 21 can image the target object OBJ.
  • the lighting device 23 may be disposed at any position where the lighting device 23 can illuminate the target object OBJ with illumination light.
  • at least one of the imaging device 21 and the lighting device 23 may be disposed on the ceiling of a building in which the robot system SYS is disposed.
  • at least one of the imaging device 21 and the lighting device 23 may be attached to a support member that is a member different from the robot 1 and can support the imaging unit 2.
  • the support member may be a member that can suspend at least one of the imaging device 21 and the lighting device 23 from above the target object OBJ.
  • the support member may include a plurality of leg members extending upward from the support surface S, and a beam member that connects the plurality of leg members via the upper ends or the vicinity thereof of the plurality of leg members. Furthermore, if the imaging unit 2 is attached to a support member, the robot 1 may be able to move the support member.
  • a control system that generates control signals for controlling a robot, The robot is equipped with a processing device that processes the target object, The robot moves the processing device, The control system is A control device is provided that generates and outputs the control signal based on image data generated by the imaging device capturing an image of the target object, The control device performs a second movement process after the first movement process is started, The first movement process includes a process of controlling the robot to move the processing device without using the image data, The second movement process includes a process of generating and outputting the control signal for controlling the robot to move the processing device based on at least one of the calculation results of the position and the orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object after the first movement process is started.
  • the first movement process includes a process of controlling the robot to move the processing device according to movement instruction information without using the image data.
  • Appendix A3 A control system that generates control signals for controlling a robot, The robot is equipped with a processing device that processes the target object, The robot moves the processing device, The control system is A control device is provided that generates and outputs the control signal based on image data generated by the imaging device capturing an image of the target object, The control device performs a second movement process after the first movement process is started,
  • the first movement process includes a process of controlling the robot to move the processing device according to movement instruction information,
  • the second movement process includes a process of generating and outputting the control signal for controlling the robot to move the processing device based on at least one of the calculation results of the position and the orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object after the first movement process is started.
  • the first movement process includes a process of controlling the robot to move the processing device according to the movement instruction information without using the image data.
  • a part of the first movement process is performed during the period in which the second movement process is performed.
  • the control device performs a part of the second movement process during the period in which the first movement process is being performed.
  • the control device performs the second movement process after the first movement process.
  • the control device starts the second movement process after the first movement process.
  • the second movement process includes a process of acquiring the image data generated by the imaging device capturing an image of the target object after the first movement process is started.
  • the second movement process includes a process of generating the image data by capturing an image of the target object using the imaging device after the first movement process is started.
  • the second movement process includes a process of calculating at least one of the position and the orientation of the target object based on the image data generated by the imaging device capturing an image of the target object after the first movement process is started.
  • the second movement process includes a process of generating and outputting the control signal for controlling the robot to move the processing device based on at least one of the position and orientation of the target object calculated by the second movement process.
  • the control device performs the first movement process, and after performing the first movement process, performs the second movement process.
  • Appendix A14 When the control device is the second control device, The first movement process is performed by a first control device that is different from the second control device. A control system described in any one of Appendices A1 to A13. [Appendix A15] The second control device receives from the first control device an authorization signal that authorizes the second control device to control the robot to move the processing device by performing the second movement process after the first movement process is completed by the first control device, The second control device starts at least a part of the second movement process after receiving the permission signal from the first control device. The control system described in Appendix A14.
  • the second control device After the first control device has completed the first movement process, the second control device receives from the first control device an authorization signal that authorizes the second control device to reflect the control signal generated by the second movement process in the control of the robot, The second control device starts at least a part of the second movement process after receiving the permission signal from the first control device.
  • the second control device receives from the first control device an authorization signal that authorizes the second control device to control the robot to move the processing device by performing the second movement process after the first movement process is completed by the first control device, The second control device starts a process of generating the control signal as part of the second movement process before receiving the permission signal from the first control device, After receiving the permission signal from the first control device, the second control device starts a process of outputting the control signal as part of the second movement process.
  • the second control device After the first control device has completed the first movement process, the second control device receives from the first control device an authorization signal that authorizes the second control device to reflect the control signal generated by the second movement process in the control of the robot, The second control device starts a process of generating the control signal as part of the second movement process before receiving the permission signal from the first control device, After receiving the permission signal from the first control device, the second control device starts a process of outputting the control signal as part of the second movement process.
  • the second control device repeats at least a part of the second movement process multiple times after the second control device receives the permission signal.
  • the second control device controls the robot to move the processing device each time the second control device performs at least a portion of the second movement process after receiving the permission signal.
  • the second control device repeats at least a part of the second movement process multiple times between the time when the first control device completes the first movement process and the time when the first control device newly performs the first movement process.
  • the second control device repeats at least a part of the second movement process multiple times even if the second control device does not receive a signal transmitted from the first control device after the second control device receives the permission signal.
  • the second control device repeats at least a part of the second movement process until the positional relationship between the processing device and the target object becomes a predetermined positional relationship between the second control device and the first control device, after the second control device receives the permission signal and before the first control device newly performs the first movement process.
  • the second control device repeats at least a part of the second movement process until the attitude relationship between the processing device and the target object becomes a predetermined attitude relationship during the period from when the second control device receives the permission signal until when the first control device newly performs the first movement process.
  • the processing device is capable of holding an object, The second control device repeats at least a part of the second movement process until the positional relationship between the object and the target object held by the processing device becomes a predetermined positional relationship between the second control device and the first control device after the second control device receives the permission signal and before the first control device newly performs the first movement process.
  • the processing device is capable of holding an object, The second control device repeats at least a part of the second movement process until the attitude relationship between the object and the target object held by the processing device becomes a predetermined attitude relationship during the period from when the second control device receives the permission signal until when the first control device newly performs the first movement process.
  • the second control device After completing the second movement process, transmits a notification signal to the first control device to notify the first control device that the second control device has completed the second movement process, The first control device starts the first movement process after receiving the notification signal transmitted by the second control device.
  • the second control device repeats at least a part of the second movement process until the positional relationship between the processing device and the target object becomes a predetermined positional relationship, and then transmits the notification signal to the first control device; The first control device starts the first movement process after receiving the notification signal transmitted by the second control device.
  • the second control device repeats at least a portion of the second movement process until the attitude relationship between the processing device and the target object becomes a predetermined attitude relationship, and then transmits the notification signal to the first control device; The first control device starts the first movement process after receiving the notification signal transmitted by the second control device.
  • the control device generates and outputs a signal for controlling the processing device to perform processing on the target object after the attitude relationship between the processing device and the target object becomes a predetermined attitude relationship.
  • the control device as at least a part of the second movement process, performs a process of generating the control signal for controlling the robot by moving the processing device through the second movement process so that the positional relationship between the processing device and the target object becomes a predetermined positional relationship.
  • the predetermined positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the target object.
  • a control system as described in Appendix A24, A29, A33 or A35.
  • the control device as at least a part of the second movement process, performs a process of generating the control signal for controlling the robot so that the posture relationship between the processing device and the target object becomes a predetermined posture relationship.
  • the predetermined attitude relationship includes an attitude relationship in which the processing device assumes a target processing attitude that the processing device should assume when performing processing on the target object.
  • the processing device is capable of holding an object
  • the second movement process includes a process of generating the control signal for controlling the robot so that the positional relationship between the object held by the processing device and the target object becomes a predetermined positional relationship by moving the processing device by the second movement process.
  • the processing device is capable of holding an object,
  • the second movement process includes a process of generating the control signal for controlling the robot so that the posture relationship between the object held by the processing device and the target object becomes a predetermined posture relationship by moving the processing device by the second movement process.
  • a control system described in any one of Appendices A1 to A39 The control device performs the second movement process so that the movement distance of the processing device by the second movement process is shorter than the movement distance of the processing device by the first movement process.
  • a control system described in any one of Appendices A1 to A40 The control device performs the second movement process so that the maximum movement speed of the processing device by the second movement process is slower than the maximum movement speed of the processing device by the first movement process.
  • control device performs the second movement process so that the average movement speed of the processing device by the second movement process is slower than the average movement speed of the processing device by the first movement process.
  • control device performs a process of generating the control signal for controlling the robot such that, by moving the processing device by the second movement process, the difference between the position of the processing device at the time when the second movement process is completed and the target processing position where the processing device should be located when the processing device processes the target member is smaller than the difference between the position of the processing device at the time when the first movement process is completed and the target processing position where the processing device should be located when the processing device processes the target member.
  • the control device performs a process of generating the control signal for controlling the robot such that, by moving the processing device by the second movement process, the difference between the attitude of the processing device at the time when the second movement process is completed and the target processing attitude that the processing device should take when the processing device processes the target member is smaller than the difference between the attitude of the processing device at the time when the first movement process is completed and the target processing attitude that the processing device should take when processing the target member.
  • the target object has a first portion that is processed by the processing device, and a second portion that is different from the first portion and is also processed by the processing device,
  • the first movement process includes the first movement process for the first part and the first movement process for the second part
  • the second movement process includes the second movement process for the first part and the second movement process for the second part
  • the first movement process for the first part includes a process of controlling the robot so that the processing device approaches the first part
  • the second movement process for the first part includes, after the first movement process for the first part is started, a process of generating and outputting a first control signal as the control signal for controlling the robot to move the processing device based on a calculation result of at least one of the position and orientation of the first part calculated based on the image data generated by the imaging device capturing an image of the first part
  • the first movement process for the second part includes a process in which the processing device controls the robot to move from the first part to the second part after
  • the target object includes a first target object to be processed by the processing device, and a second target object that is different from the first target object and is to be processed by the processing device,
  • the first movement process includes the first movement process for the first target object and the first movement process for the second target object
  • the second movement process includes the second movement process for the first target object and the second movement process for the second target object
  • the first movement process for the first target object includes a process in which the processing device controls the robot to approach the first target object
  • the second movement process for the first target object includes, after the first movement process for the first target object is started, generating and outputting a third control signal as the control signal for controlling the robot to move the processing device based on a calculation result of at least one of the position and orientation of the first target object calculated based on the image data generated by the imaging device capturing an image of the first target object
  • the first movement process for the second target object includes, after the second movement process for the first target object
  • the target object has a first portion that is processed by the processing device, and a second portion that is different from the first portion and is also processed by the processing device,
  • the first movement process includes the first movement process for the first part and the first movement process for the second part
  • the control device performs, as at least a part of the second movement process, the second movement process for the first part and the second movement process for the second part
  • the first movement process for the first part includes a process of controlling the robot so that the processing device approaches the first part
  • the control device as at least a part of the second movement process for the first part, performs a process of generating and outputting a first control signal as the control signal for controlling the robot to move the processing device based on a calculation result of at least one of the position and orientation of the first part calculated based on the image data generated by the imaging device capturing an image of the first part after the first movement process for the first part has been started;
  • the control device repeats the second movement process for the first part until the positional relationship between the processing device and the first part becomes the first positional relationship between the processing device and the first part and the first movement process for the second part is started after the first movement process for the first part is completed.
  • the first positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the first portion. The control system described in Appendix A49.
  • the control device repeats the second movement process for the first part until the attitude relationship between the processing device and the first part becomes the first attitude relationship between the completion of the first movement process for the first part and the start of the first movement process for the second part.
  • the first attitude relationship includes an attitude relationship in which the processing device assumes a target processing attitude that the processing device should assume when performing processing on the first portion. The control system described in Appendix A51.
  • the control device repeats the second movement process for the second part until the positional relationship between the processing device and the second part becomes the second positional relationship.
  • the second positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the second portion.
  • the control device After the first movement process for the second part is started, the control device repeats the second movement process for the second part until the attitude relationship between the processing device and the second part becomes the second attitude relationship.
  • the second attitude relationship includes an attitude relationship in which the processing device is taking a target processing attitude that the processing device should take when performing processing on the second part.
  • a control system described in any one of Appendices A48 to A57 The control device repeats the second movement process for the first part during a first period from when the first movement process for the first part is completed to when the first movement process for the second part is started, until the attitude relationship between the processing device and the first part becomes a first attitude relationship; The control device repeats the second movement process for the second part during a second period after the first movement process for the second part is started, until the attitude relationship between the processing device and the second part becomes a second attitude relationship; The control device generates the second control signal in the second period based on the first control signal generated in the first period.
  • a control system described in any one of Appendices A48 to A58 The control device repeats the second movement process for the first part during a first period from when the first movement process for the first part is completed to when the first movement process for the second part is started, until the attitude relationship between the processing device and the first part becomes a first attitude relationship; The control device repeats the second movement process for the second part during a second period after
  • the control device generates the second control signal that is generated first in the second period based on the first control signal that is generated last in the first period.
  • the control device generates the second control signal by using the first control signal as an initial signal for the second control signal.
  • the control device repeats the second movement process for the first part during a first period from when the first movement process for the first part is completed to when the first movement process for the second part is started, until the positional relationship between the processing device and the first part becomes a first positional relationship;
  • the control device repeats the second movement process for the second part during a second period after the first movement process for the second part is started, until the positional relationship between the processing device and the second part becomes a second positional relationship;
  • the control device generates the second control signal during the second period by using the first control signal generated during the first period as an initial signal for the second control signal.
  • the control device repeats the second movement process for the first part during a first period from when the first movement process for the first part is completed to when the first movement process for the second part is started, until the attitude relationship between the processing device and the first part becomes a first attitude relationship;
  • the control device repeats the second movement process for the second part during a second period after the first movement process for the second part is started, until the attitude relationship between the processing device and the second part becomes a second attitude relationship;
  • the control device generates the second control signal during the second period by using the first control signal generated during the first period as an initial signal for the second control signal.
  • the control device generates the second control signal that is generated first in the second period by using the first control signal that is generated last in the first period as the initial signal of the second control signal.
  • the target object includes a first target object to be processed by the processing device, and a second target object that is different from the first target object and is to be processed by the processing device,
  • the first movement process includes the first movement process for the first target object and the first movement process for the second target object,
  • the control device performs, as at least a part of the second movement process, the second movement process for the first target object and the second movement process for the second target object,
  • the first movement process for the first target object includes a process in which the processing device controls the robot to approach the first target object,
  • the control device as at least a part of the second movement process for the first target object, performs a process of generating and outputting a third control signal as the control signal for controlling the robot to move the processing device based on
  • the control device repeats the second movement process for the first target object until the positional relationship between the processing device and the first target object becomes a third positional relationship between the processing device and the first target object after completing the first movement process for the first target object and before starting the first movement process for the second target object.
  • the third positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the first target object.
  • the control device repeats the second movement process for the first target object until the attitude relationship between the processing device and the first target object becomes a third attitude relationship between the processing device and the first target object and the first movement process for the second target object is started after the first movement process for the first target object is completed.
  • the third attitude relationship includes an attitude relationship in which the processing device assumes a target processing attitude that the processing device should assume when performing processing on the first target object. The control system described in Appendix A68.
  • the fourth attitude relationship includes an attitude relationship in which the processing device assumes a target processing attitude that the processing device should assume when performing processing on the second target object.
  • the control device generates the fourth control signal based on the third control signal.
  • the control device repeats the second movement process for the first target object during a third period from when the first movement process for the first target object is completed to when the first movement process for the second target object is started, until the positional relationship between the processing device and the first target object becomes a third positional relationship;
  • the control device repeats the second movement process for the second target object during a fourth period after the first movement process for the second target object is started, until the positional relationship between the processing device and the second target object becomes a fourth positional relationship;
  • the control device generates the fourth control signal in the fourth period based on the third control signal generated in the third period.
  • the control device repeats the second movement process for the first target object during a third period from the completion of the first movement process for the first target object to the start of the first movement process for the second target object until the attitude relationship between the processing device and the first target object becomes a third attitude relationship;
  • the control device repeats the second movement process for the second target object during a fourth period after the first movement process for the second target object is started, until the attitude relationship between the processing device and the second target object becomes a fourth attitude relationship;
  • the control device generates the fourth control signal in the fourth period based on the third control signal generated in the third period.
  • the control device generates the fourth control signal that is generated first in the fourth period based on the third control signal that is generated last in the third period.
  • the control device generates the fourth control signal by using the third control signal as an initial signal for the fourth control signal.
  • the control device repeats the second movement process for the first target object during a third period from when the first movement process for the first target object is completed to when the first movement process for the second target object is started, until the positional relationship between the processing device and the first target object becomes a third positional relationship;
  • the control device repeats the second movement process for the second target object during a fourth period after the first movement process for the second target object is started, until the positional relationship between the processing device and the second target object becomes a fourth positional relationship;
  • the control device generates the fourth control signal in the fourth period by using the third control signal generated in the third period as an initial signal for the fourth control signal.
  • the control device repeats the second movement process for the first target object during a third period from the completion of the first movement process for the first target object to the start of the first movement process for the second target object until the attitude relationship between the processing device and the first target object becomes a third attitude relationship;
  • the control device repeats the second movement process for the second target object during a fourth period after the first movement process for the second target object is started, until the attitude relationship between the processing device and the second target object becomes a fourth attitude relationship;
  • the control device generates the fourth control signal during the fourth period by using the third control signal generated during the third period as an initial signal for the fourth control signal.
  • the control device generates the fourth control signal that is generated first in the fourth period by using the third control signal that is generated last in the third period as the initial signal of the fourth control signal.
  • the control device is a second control device
  • the first movement process is performed by a first control device different from the second control device
  • the target object has a first portion that is processed by the processing device, and a second portion that is different from the first portion and is also processed by the processing device
  • the first control device performs, as at least a part of the first movement process, the first movement process for the first part and the first movement process for the second part;
  • the second control device performs, as at least a part of the second movement process, the second movement process for the first part and the second movement process for the second part;
  • the first control device performs a process of controlling the robot so that the processing device approaches the first part as at least a part of the first movement process for the first part,
  • the second control device as at
  • the second control device performs the first movement process for the second part.
  • the second control device, as at least a part of the first movement process for the second part performs a process of controlling the robot so that the processing device approaches the second part from the first part, even if the second control device does not receive a signal transmitted from the first control device after the second movement process for the first part.
  • the second control device repeats the second movement process for the first part until the positional relationship between the processing device and the first part becomes the first positional relationship between the first movement process for the first part and the start of the first movement process for the second part.
  • the first positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the first portion. The control system described in Appendix A85.
  • the second control device repeats the second movement process for the first part until the attitude relationship between the processing device and the first part becomes the first attitude relationship between the completion of the first movement process for the first part and the start of the first movement process for the second part.
  • the first attitude relationship includes an attitude relationship in which the processing device assumes a target processing attitude that the processing device should assume when performing processing on the first portion. The control system described in Appendix A87.
  • the second control device repeats the second movement process for the second part until the positional relationship between the processing device and the second part becomes the second positional relationship.
  • the second positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the second portion.
  • the second control device After the first movement process for the second part is started, the second control device repeats the second movement process for the second part until the attitude relationship between the processing device and the second part becomes the second attitude relationship.
  • the second attitude relationship includes an attitude relationship in which the processing device is taking a target processing attitude that the processing device should take when performing processing on the second part.
  • the second control device generates the second control signal based on the first control signal.
  • the second control device repeats the second movement process for the first part during a first period from when the first movement process for the first part is completed to when the first movement process for the second part is started, until the positional relationship between the processing device and the first part becomes a first positional relationship;
  • the second control device repeats the second movement process for the second part during a second period after the first movement process for the second part is started, until the positional relationship between the processing device and the second part becomes a second positional relationship;
  • the second control device generates the second control signal during the second period based on the first control signal generated during the first period.
  • the second control device repeats the second movement process for the first part during a first period from when the first movement process for the first part is completed to when the first movement process for the second part is started, until the attitude relationship between the processing device and the first part becomes a first attitude relationship;
  • the second control device repeats the second movement process for the second part during a second period after the first movement process for the second part is started, until the attitude relationship between the processing device and the second part becomes a second attitude relationship;
  • the second control device generates the second control signal during the second period based on the first control signal generated during the first period.
  • the second control device generates the second control signal that is generated first in the second period based on the first control signal that is generated last in the first period.
  • the second control device generates the second control signal by using the first control signal as an initial signal for the second control signal.
  • the second control device repeats the second movement process for the first part during a first period from when the first movement process for the first part is completed to when the first movement process for the second part is started, until the positional relationship between the processing device and the first part becomes a first positional relationship;
  • the second control device repeats the second movement process for the second part during a second period after the first movement process for the second part is started, until the positional relationship between the processing device and the second part becomes a second positional relationship;
  • the second control device generates the second control signal during the second period by using the first control signal generated during the first period as an initial signal for the second control signal.
  • the second control device repeats the second movement process for the first part during a first period from when the first movement process for the first part is completed to when the first movement process for the second part is started, until the attitude relationship between the processing device and the first part becomes a first attitude relationship;
  • the second control device repeats the second movement process for the second part during a second period after the first movement process for the second part is started, until the attitude relationship between the processing device and the second part becomes a second attitude relationship;
  • the second control device generates the second control signal during the second period by using the first control signal generated during the first period as an initial signal for the second control signal.
  • the second control device generates the second control signal that is generated first in the second period by using the first control signal that is generated last in the first period as the initial signal of the second control signal.
  • the first movement process is performed by a first control device different from the second control device
  • the target object includes a first target object to be processed by the processing device, and a second target object that is different from the first target object and is to be processed by the processing device
  • the first control device performs the first movement process for the first target object and the first movement process for the second target object as at least a part of the first movement process
  • the second control device performs, as at least a part of the second movement process, the second movement process for the first target object and the second movement process for the second target object
  • the first control device performs a process of controlling the robot so that the processing device approaches the first target object as at least a part of the first movement process for the first target
  • the second control device instead of the first control device, performs the first movement process for the second target object.
  • the second control device as at least a part of the first movement process for the second target object, performs a process of controlling the robot so that the processing device moves from the first target object toward the second target object even if the second control device does not receive a signal transmitted from the first control device after the second movement process for the first target object.
  • the control system described in Appendix A102 The control system described in Appendix A102.
  • the second control device repeats the second movement process for the first target object until the positional relationship between the processing device and the first target object becomes a third positional relationship between the first movement process for the first target object and the start of the first movement process for the second target object.
  • the third positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the first target object. The control system described in Appendix A104.
  • the second control device repeats the second movement process for the first target object until the attitude relationship between the processing device and the first target object becomes a third attitude relationship between the first movement process for the first target object and the start of the first movement process for the second target object.
  • the third attitude relationship includes an attitude relationship in which the processing device assumes a target processing attitude that the processing device should assume when performing processing on the first target object. The control system described in Appendix A106.
  • the second control device repeats the second movement process for the second target object until the positional relationship between the processing device and the second target object becomes a fourth positional relationship.
  • the fourth positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the second target object.
  • the second control device repeats the second movement process for the second target object until the attitude relationship between the processing device and the second target object becomes a fourth attitude relationship.
  • the fourth attitude relationship includes an attitude relationship in which the processing device is taking a target processing attitude that the processing device should take when performing processing on the second target object.
  • the second control device generates the fourth control signal based on the third control signal.
  • the second control device repeats the second movement process for the first target object during a third period from the completion of the first movement process for the first target object to the start of the first movement process for the second target object until the positional relationship between the processing device and the first target object becomes a third positional relationship;
  • the second control device repeats the second movement process for the second target object during a fourth period after the first movement process for the second target object is started, until the positional relationship between the processing device and the second target object becomes a fourth positional relationship;
  • the second control device generates the fourth control signal in the fourth period based on the third control signal generated in the third period.
  • the second control device repeats the second movement process for the first target object during a third period from when the first movement process for the first target object is completed to when the first movement process for the second target object is started, until the attitude relationship between the processing device and the first target object becomes a third attitude relationship;
  • the second control device repeats the second movement process for the second target object during a fourth period after the first movement process for the second target object is started, until the attitude relationship between the processing device and the second target object becomes a fourth attitude relationship;
  • the second control device generates the fourth control signal in the fourth period based on the third control signal generated in the third period.
  • the second control device generates the fourth control signal that is generated first in the fourth period based on the third control signal that is generated last in the third period.
  • the second control device generates the fourth control signal by using the third control signal as an initial signal for the fourth control signal.
  • the second control device repeats the second movement process for the first target object during a third period from the completion of the first movement process for the first target object to the start of the first movement process for the second target object until the positional relationship between the processing device and the first target object becomes a third positional relationship; the second control device repeats the second movement process for the second target object during a fourth period after the first movement process for the second target object is started, until the positional relationship between the processing device and the second target object becomes a fourth positional relationship; The second control device generates the fourth control signal in the fourth period by using the third control signal generated in the third period as an initial signal for the fourth control signal.
  • the second control device repeats the second movement process for the first target object during a third period from when the first movement process for the first target object is completed to when the first movement process for the second target object is started, until the attitude relationship between the processing device and the first target object becomes a third attitude relationship;
  • the second control device repeats the second movement process for the second target object during a fourth period after the first movement process for the second target object is started, until the attitude relationship between the processing device and the second target object becomes a fourth attitude relationship;
  • the second control device generates the fourth control signal during the fourth period by using the third control signal generated during the third period as an initial signal for the fourth control signal.
  • the second control device generates the fourth control signal that is generated first in the fourth period by using the third control signal that is generated last in the third period as the initial signal of the fourth control signal.
  • the first movement process includes a process of controlling the robot to move the processing device according to movement instruction information.
  • the movement instruction information is set in advance by the user inputting the movement instruction information. The control system described in Appendix A120.
  • the movement instruction information includes at least one of a target movement position indicating a position to which the processing device should move, a target movement distance indicating a distance to which the processing device should move, and a target movement direction indicating a direction to which the processing device should move.
  • the target movement position is fixed to a preset position during the period in which the first movement process is being performed, the target movement distance is fixed to a preset distance during the period in which the first movement process is being performed, and the target movement direction is fixed to a preset direction during the period in which the first movement process is being performed.
  • the control system described in Appendix A122 The control system described in Appendix A122.
  • the target object is moved by a transport device capable of transporting the target object, At least one of the target movement position, the target movement distance, and the target movement direction is preset based on at least one of the conveying speed of the target member by the conveying device and the conveying direction of the target member by the conveying device.
  • the target object has a first portion that is processed by the processing device, and a second portion that is different from the first portion and is also processed by the processing device,
  • the processing device processes the first portion, and then processes the second portion,
  • At least one of the target movement position, the target movement distance, and the target movement direction is preset based on the positional relationship between the first part and the second part.
  • the target object includes a first target object to be processed by the processing device, and a second target object that is different from the first target object and is to be processed by the processing device, The processing device performs processing on the first target object, and then performs processing on the second target object, At least one of the target movement position, the target movement distance, and the target movement direction is preset based on the positional relationship between the first target object and the second target object.
  • the first movement process is based on open loop control.
  • the second movement process is a process based on closed-loop control.
  • the closed loop control is at least one of P (Proportional) control, PI (Proportional-Integral) control, and PID (Proportional-Integral-Derivative) control.
  • the target object is transported by a transport device capable of transporting the target object, The first movement process and the second movement process are performed on the target object being moved by the transport device.
  • the second movement process includes a process of calculating at least one of the position and the orientation of the target object by performing a matching process using the image data generated by the imaging device capturing an image of the target object after the first movement process is started and model data indicating an object model that is a model of the target object.
  • the object model is a model of at least a portion of the edge of the target object.
  • the model data is data that indicates the edge of at least a portion of the target object or the sample in an image obtained by imaging the target object or a sample object having the same shape as the target object in advance.
  • the second movement process includes a process of generating and outputting the control signal for controlling the robot to move the processing device based on at least one of the calculation results of the position and orientation of the target object and at least one of the calculation results of the position and orientation of the processing device, which are calculated based on the image data generated by the imaging device capturing an image of the target object and the processing device after the first movement process is started.
  • the second movement process includes a process of calculating at least one of the position and orientation of the target object and at least one of the position and orientation of the processing device by performing a matching process using the image data generated by the imaging device capturing an image of the target object and the processing device after the first movement process is started, and model data indicating an object model that is a model of the target object and a device model that is a model of the processing device.
  • the object model is a model of at least a portion of the edge of the target object
  • the device model is a model of at least a portion of the edge of the processing device.
  • the model data indicates the object model and the device model aligned with each other
  • the second movement process includes a process of generating the control signal for controlling the robot by moving the processing device so that the difference between the positional relationship between the target object and the processing device and the positional relationship between the object model and the device model that are aligned with each other is within a threshold value, based on at least one of the calculation results of the position and orientation of the target object and at least one of the calculation results of the position and orientation of the processing device.
  • the model data indicates the object model and the device model aligned with each other
  • the second movement process includes a process of generating the control signal for controlling the robot so that the difference between the posture relationship between the target object and the processing device and the posture relationship between the object model and the device model that are aligned with each other is within a threshold value by moving the processing device, based on at least one of the calculation results of the position and posture of the target object and at least one of the calculation results of the position and posture of the processing device.
  • the model data indicates the object model and the device model aligned with each other
  • the second movement process includes a process of selecting, when a plurality of target objects are detected by the matching process, one target object that satisfies the condition that the positional relationship between the target object and the processing device is closest to the positional relationship between the object model and the device model that are aligned with each other, as the target object to be processed by the processing device.
  • the control system performs a preparatory process for carrying out the second movement process during the period in which the first movement process is being carried out and before the second movement process is started.
  • the control device Before starting the second movement process, the control device performs a preparation process for the second movement process.
  • the control device performs a preparatory process for the second movement process during the period in which the first movement process is being performed and before the second movement process is started.
  • the preparation process includes a process of acquiring information used to perform the second movement process.
  • the information used to perform the second movement processing includes target object information regarding the target object on which the processing device performs the predetermined processing.
  • the target object information includes information regarding the type of the target object.
  • the information used to perform the second movement processing includes peripheral object information regarding peripheral objects located in the vicinity of the target object on which the processing device performs the predetermined processing.
  • the surrounding object information includes information regarding at least one of the position of the surrounding object, the orientation of the surrounding object, the shape of the surrounding object, and the size of the surrounding object.
  • the second movement process includes a process of calculating at least one of the position and the orientation of the target object by performing a matching process using the image data generated by the imaging device capturing an image of the target object after the first movement process is started and model data indicating an object model that is a model of the target object,
  • the information used to perform the second movement process includes matching information, which is information used in the matching process.
  • the matching information includes at least one of information specifying the model data, information regarding a matching judgment threshold used in the matching process, and information regarding a convergence condition that must be satisfied to terminate the second movement process.
  • the information used to perform the second movement process includes imaging condition information, which is information regarding the imaging conditions of the imaging device.
  • imaging condition information includes at least one of information regarding exposure time and information regarding the intensity of illumination light.
  • the information used to perform the second movement process includes the image data generated by the imaging device capturing an image of the target object after the first movement process is started. A control system described in any one of Appendices A143 to A151.
  • the information used to perform the second movement process includes information regarding the calculation result of at least one of the position and orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object after the first movement process is started.
  • the preparation process includes a process of acquiring the image data generated by the imaging device capturing an image of the target object after the first movement process is started.
  • the preparation process includes a process of generating the image data by capturing an image of the target object using the imaging device after the first movement process is started.
  • the preparation process includes a process of calculating at least one of the position and the orientation of the target object based on the image data generated by the imaging device capturing an image of the target object after the first movement process is started.
  • the preparation process includes a process for adjusting the exposure conditions of the imaging device.
  • the imaging device is provided with an illumination device capable of illuminating the target object with illumination light
  • the preparation process includes a process of causing the lighting device to start illuminating the target object with the illumination light.
  • the second movement process includes, when it becomes impossible to calculate at least one of the position and orientation of the target object based on image data generated by imaging by the imaging device at a first time during the period in which the second movement process is being performed, a process of calculating at least one of the position and orientation of the target object at the first time based on at least one of the position and orientation of the target object calculated based on the image data generated by imaging the target object by the imaging device at a second time prior to the first time.
  • the second movement process includes a process of generating and outputting a control signal for controlling the robot to move the processing device based on information regarding contact between the processing device and the target object when it becomes impossible to calculate at least one of the position and orientation of the target object based on the image data generated by imaging by the imaging device during the period in which the second movement process is being performed.
  • the control system inspects the target object based on the image data.
  • the control system inspects the target object based on the measurement results of the target object by a measuring device capable of measuring the target object.
  • the control system inspects the target object before the processing device processes the target object.
  • the control system inspects the target object after the processing device processes the target object.
  • the control system inspects the results of the processing performed by the processing device on the target object based on the image data or the measurement results of the measuring device.
  • the control device inspects the target object based on the image data.
  • a control system described in any one of Appendices A1 to A165 A control system described in any one of Appendices A1 to A165.
  • the control device inspects the target object based on the measurement results of the target object by a measuring device capable of measuring the target object.
  • a control system described in any one of Appendices A1 to A166 A control system described in any one of Appendices A1 to A166.
  • the control device inspects the target object before the processing device processes the target object.
  • the control device inspects the target object after the processing device processes the target object.
  • the control device inspects the results of the processing performed by the processing device on the target object based on the image data or the measurement results of the measuring device.
  • the target object is moved and transported by a transport device capable of transporting the target object, The control system performs the second movement process while the transport device is moving the target object, The second movement process includes a process of generating a control signal for controlling the robot to move the processing device based on a transport state of the target object when it becomes impossible to calculate at least one of the position and the orientation of the target object based on image data generated by imaging by the imaging device at a first time during the period in which the second movement process is being performed.
  • the target object is moved by a transport device capable of transporting the target object,
  • the control device performs the second movement process while the transport device is moving the target object,
  • the second movement process includes a process of generating a control signal for controlling the robot to move the processing device based on a transport state of the target object when it becomes impossible to calculate at least one of the position and the orientation of the target object based on image data generated by imaging by the imaging device at a first time during the period in which the second movement process is being performed.
  • the transport state of the target object includes at least one of the transport direction of the target object, the transport distance of the target object, and the transport speed of the target object.
  • the transport state of the target object includes the transport direction of the target object,
  • the second movement process includes a process of generating a control signal for controlling the robot to move the processing device in a movement direction determined according to the transport direction of the target object.
  • the transport state of the target object includes the transport distance of the target object
  • the second movement process includes a process of generating a control signal for controlling the robot to move the processing device by a movement distance determined according to the transport distance of the target object.
  • the transport state of the target object includes the transport speed of the target object
  • the second movement process includes a process of estimating a transport distance of the target object based on the transport speed of the target object, and a process of generating a control signal for controlling the robot to move the processing device by a travel distance determined according to the estimated result of the transport distance of the target object.
  • the transport state of the target object includes the transport direction of the target object and the transport distance of the target object
  • the second movement process includes a process of generating a control signal for controlling the robot to move the processing device in a movement direction determined according to the transport direction of the target object by a movement distance determined according to the transport distance of the target object.
  • the transport state of the target object includes the transport direction of the target object and the transport speed of the target object
  • the second movement process includes a process of estimating a transport distance of the target object based on the transport speed of the target object, and a process of generating a control signal for controlling the robot to move the processing device by a travel distance determined according to the estimated result of the transport distance of the target object in a travel direction determined according to the transport direction of the target object.
  • the second movement process includes a process of acquiring information about the transport state of the target object from the transport device or a device that controls the transport device, and a process of generating a control signal for controlling the robot to move the processing device based on the acquired information.
  • the second movement process includes a process of estimating a transport state of the target object based on a calculation result of at least one of the position and the orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object during a period prior to the first time, and a process of generating a control signal for controlling the robot to move the processing device based on the estimation result of the transport state of the target object.
  • the control system further includes the imaging device.
  • the control device When the control device is a second control device, the first movement process is performed by a first control device different from the second control device, The control system further includes the first control device.
  • the imaging device is attached to the robot.
  • the robot is a first robot, The imaging device is attached to a second robot that is different from the first robot.
  • the imaging device is attached to a support member that is different from the robot and can support the imaging device.
  • Appendix A186 A control system according to any one of appendices A1 to A185; The robot and A robot system equipped with [Appendix A187] Further comprising the processing device that performs the processing on the target object A robot system as described in Appendix A186.
  • a control method for generating a control signal for controlling a robot comprising: The robot is equipped with a processing device that processes the target object, The robot moves the processing device, The control method includes generating and outputting the control signal based on image data generated by an imaging device capturing an image of the target object, The control method includes performing a second movement process after the first movement process is started, The first movement process includes a process of controlling the robot to move the processing device without using the image data, The second movement process includes a process of generating and outputting the control signal for controlling the robot to move the processing device based on at least one of the calculation results of the position and the orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object after the first movement process is started.
  • Control method [Appendix B2]
  • the first movement process includes a process of controlling the robot to move the processing device according to movement instruction information without using the image data.
  • a control method for generating a control signal for controlling a robot comprising: The robot is equipped with a processing device that processes the target object, The robot moves the processing device, The control method includes generating and outputting the control signal based on image data generated by an imaging device capturing an image of the target object, The control method includes performing a second movement process after the first movement process is started, The first movement process includes a process of controlling the robot to move the processing device according to movement instruction information, The second movement process includes a process of generating and outputting the control signal for controlling the robot to move the processing device based on at least one of the calculation results of the position and the orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object after the first movement process is started.
  • the first movement process includes a process of controlling the robot to move the processing device according to the movement instruction information without using the image data.
  • a part of the first movement process is performed during the period in which the second movement process is performed.
  • the control method includes performing a part of the second movement process during a period in which the first movement process is being performed.
  • the control method includes performing the second movement process after the first movement process.
  • the control method includes starting the second movement process after the first movement process.
  • the second movement process includes a process of acquiring the image data generated by the imaging device capturing an image of the target object after the first movement process is started.
  • the second movement process includes a process of generating the image data by capturing an image of the target object using the imaging device after the first movement process is started.
  • the second movement process includes a process of calculating at least one of the position and the orientation of the target object based on the image data generated by the imaging device capturing an image of the target object after the first movement process is started.
  • the second movement process includes a process of generating and outputting the control signal for controlling the robot to move the processing device based on at least one of the position and orientation of the target object calculated by the second movement process.
  • the control method includes using a control device to perform the first movement process and, after performing the first movement process, to perform the second movement process.
  • the control method includes performing the second movement process using a second control device, The first movement process is performed by a first control device that is different from the second control device.
  • the control method is using the second control device, after the first control device has completed the first movement process, receiving from the first control device an authorization signal that authorizes the second control device to control the robot to move the processing device by performing the second movement process; Starting at least a part of the second movement process using the second control device after receiving the permission signal from the first control device;
  • a control method described in Appendix B14, including: [Appendix B16] The control method is using the second control device, after the first control device has completed the first movement process, receiving from the first control device an authorization signal that authorizes the second control device to reflect the control signal generated by the second movement process in the control of the robot; Starting at least a part of the second movement process using the second control device after receiving the permission signal from the first control device;
  • a control method described in Appendix B14 or B15, including: [Appendix B17] The control method is using the second control device, after the first control device has completed the first movement process, receiving from the first control device an authorization signal that authorizes the second control device to control the
  • a control method described in any one of Appendices B14 to B16 including: [Appendix B18]
  • the control method is using the second control device, after the first control device has completed the first movement process, receiving from the first control device an authorization signal that authorizes the second control device to reflect the control signal generated by the second movement process in the control of the robot; using the second control device, before receiving the permission signal from the first control device, initiating a process of generating the control signal as part of the second movement process; After receiving the permission signal from the first control device, the second control device starts a process of outputting the control signal as part of the second movement process.
  • a control method described in any one of Appendices B14 to B17 is described in any one of Appendices B14 to B17.
  • the control method includes, by using the second control device, not starting at least a part of the second movement process if the permission signal transmitted by the first control device is not received.
  • the control method includes repeating, by using the second control device, at least a portion of the second movement process multiple times after the second control device receives the permission signal.
  • the control method includes controlling the robot to move the processing device using the second control device each time the second control device performs at least a portion of the second movement process after receiving the permission signal.
  • the control method includes repeating at least a part of the second movement process multiple times using the second control device during the period from when the first control device completes the first movement process until when the first control device newly performs the first movement process.
  • the control method includes repeating, by using the second control device, at least a portion of the second movement process multiple times even if the second control device does not receive a signal transmitted from the first control device after the second control device receives the permission signal.
  • the control method includes repeating, by using the second control device, at least a part of the second movement process until a positional relationship between the processing device and the target object becomes a predetermined positional relationship during the period from when the second control device receives the permission signal until when the first control device newly performs the first movement process.
  • the control method includes repeating, by using the second control device, at least a part of the second movement process until a posture relationship between the processing device and the target object becomes a predetermined posture relationship during the period from when the second control device receives the permission signal until when the first control device newly performs the first movement process.
  • the processing device is capable of holding an object,
  • the control method includes repeating, by using the second control device, at least a part of the second movement process until the positional relationship between the object and the target object held by the processing device becomes a predetermined positional relationship during the period from when the second control device receives the permission signal until when the first control device newly performs the first movement process.
  • the processing device is capable of holding an object
  • the control method includes repeating, by using the second control device, at least a part of the second movement process until the attitude relationship between the object and the target object held by the processing device becomes a predetermined attitude relationship during the period from when the second control device receives the permission signal until when the first control device newly performs the first movement process.
  • Appendix B28 The control method is After completing the second movement process using the second control device, transmitting a notification signal to the first control device to notify the first control device that the second control device has completed the second movement process; Starting the first movement process after receiving the notification signal transmitted by the second control device using the first control device; A control method described in any one of Appendices B14 to B27.
  • the control method is using the second control device, repeating at least a portion of the second movement process until a positional relationship between the processing device and the target object becomes a predetermined positional relationship, and then transmitting the notification signal to the first control device; Starting the first movement process after receiving the notification signal transmitted by the second control device using the first control device; A control method described in Appendix B28, including: [Appendix B30] The control method is using the second control device, repeating at least a portion of the second movement process until the attitude relationship between the processing device and the target object becomes a predetermined attitude relationship, and then transmitting the notification signal to the first control device; Starting the first movement process after receiving the notification signal transmitted by the second control device using the first control device; A control method described in Appendix B28 or B29, including: [Appendix B31] The control method includes not starting the first movement process when the notification signal transmitted by the second control device is not received using the first control device.
  • the control method includes generating and outputting a signal for controlling the processing device to perform processing on the target object after performing the second movement processing.
  • the control method includes generating and outputting a signal for controlling the processing device to perform processing on the target object after the positional relationship between the processing device and the target object becomes a predetermined positional relationship.
  • the control method includes generating and outputting a signal for controlling the processing device to perform processing on the target object after the attitude relationship between the processing device and the target object becomes a predetermined attitude relationship.
  • the control method includes, as at least a part of the second movement process, performing a process of generating the control signal for controlling the robot so that the positional relationship between the processing device and the target object becomes a predetermined positional relationship by moving the processing device by the second movement process.
  • the predetermined positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the target object.
  • the control method includes, as at least a part of the second movement process, performing a process of generating the control signal for controlling the robot so that the posture relationship between the processing device and the target object becomes a predetermined posture relationship.
  • the predetermined attitude relationship includes an attitude relationship in which the processing device assumes a target processing attitude that the processing device should assume when performing processing on the target object.
  • the processing device is capable of holding an object
  • the second movement process includes a process of generating the control signal for controlling the robot so that the positional relationship between the object held by the processing device and the target object becomes a predetermined positional relationship by moving the processing device by the second movement process.
  • the processing device is capable of holding an object,
  • the second movement process includes a process of generating the control signal for controlling the robot so that the posture relationship between the object held by the processing device and the target object becomes a predetermined posture relationship by moving the processing device by the second movement process.
  • the control method includes performing the second movement process so that the movement distance of the processing device by the second movement process is shorter than the movement distance of the processing device by the first movement process.
  • the control method includes performing the second movement process so that the maximum movement speed of the processing device by the second movement process is slower than the maximum movement speed of the processing device by the first movement process.
  • the control method includes performing the second movement process so that the average movement speed of the processing device by the second movement process is slower than the average movement speed of the processing device by the first movement process.
  • the control method includes, as at least a part of the second movement process, performing a process of generating the control signal for controlling the robot such that the difference between the position of the processing device at the time when the second movement process is completed and the target processing position is smaller than the difference between the position of the processing device at the time when the first movement process is completed and the target processing position where the processing device should be located when the processing device performs processing on the target member by moving the processing device by the second movement process.
  • the control method includes, as at least a part of the second movement process, performing a process of generating the control signal for controlling the robot such that the difference between the attitude of the processing device at the time when the second movement process is completed and the target processing attitude that the processing device should take when the processing device processes the target member is smaller than the difference between the attitude of the processing device at the time when the first movement process is completed and the target processing attitude that the processing device should take when the processing device processes the target member by moving the processing device by the second movement process.
  • the target object has a first portion that is processed by the processing device, and a second portion that is different from the first portion and is also processed by the processing device,
  • the first movement process includes the first movement process for the first part and the first movement process for the second part
  • the second movement process includes the second movement process for the first part and the second movement process for the second part
  • the first movement process for the first part includes a process of controlling the robot so that the processing device approaches the first part
  • the second movement process for the first part includes, after the first movement process for the first part is started, a process of generating and outputting a first control signal as the control signal for controlling the robot to move the processing device based on a calculation result of at least one of the position and orientation of the first part calculated based on the image data generated by the imaging device capturing an image of
  • the target object includes a first target object to be processed by the processing device, and a second target object that is different from the first target object and is to be processed by the processing device,
  • the first movement process includes the first movement process for the first target object and the first movement process for the second target object
  • the second movement process includes the second movement process for the first target object and the second movement process for the second target object
  • the first movement process for the first target object includes a process in which the processing device controls the robot to approach the first target object
  • the second movement process for the first target object includes, after the first movement process for the first target object is started, generating and outputting a third control signal as the control signal for controlling the robot to move the processing device based on a calculation result of at least one of the position and orientation of the first target object calculated based on the image data generated by the imaging device capturing an image of the first target object
  • the first movement process for the second target object includes, after the second movement process for the first target object
  • the target object has a first portion that is processed by the processing device, and a second portion that is different from the first portion and is also processed by the processing device,
  • the first movement process includes the first movement process for the first part and the first movement process for the second part
  • the control method includes, as at least a part of the second movement process, performing the second movement process on the first part and the second movement process on the second part;
  • the first movement process for the first part includes a process of controlling the robot so that the processing device approaches the first part,
  • the control method includes, as at least a part of the second movement process for the first part, performing a process of generating and outputting a first control signal as the control signal for controlling the robot to move the processing device based on a calculation result of at least one of the position and orientation of the first part calculated based on the image data generated by the imaging device capturing an image of the first part after the first movement process for the first part has been started;
  • the control method includes repeating the second movement process for the first part until the positional relationship between the processing device and the first part becomes a first positional relationship between the first movement process for the first part and the start of the first movement process for the second part.
  • the first positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the first portion. Control method described in Appendix B49.
  • the control method includes repeating the second movement process for the first part until the attitude relationship between the processing device and the first part becomes a first attitude relationship between the completion of the first movement process for the first part and the start of the first movement process for the second part.
  • the first attitude relationship includes an attitude relationship in which the processing device assumes a target processing attitude that the processing device should assume when performing processing on the first portion.
  • the control method includes repeating the second movement process for the second part after the first movement process for the second part is started until the positional relationship between the processing device and the second part becomes a second positional relationship.
  • the second positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the second portion.
  • the control method includes repeating the second movement process for the second part after the first movement process for the second part is started until the attitude relationship between the processing device and the second part becomes the second attitude relationship.
  • the second attitude relationship includes an attitude relationship in which the processing device is taking a target processing attitude that the processing device should take when performing processing on the second part.
  • the control method includes generating the second control signal based on the first control signal.
  • the control method is repeat the second movement process for the first part during a first period from the completion of the first movement process for the first part to the start of the first movement process for the second part until the positional relationship between the processing device and the first part becomes a first positional relationship; repeat the second movement process for the second part during a second period after the first movement process for the second part is started, until the positional relationship between the processing device and the second part becomes a second positional relationship; In the second period, the second control signal is generated based on the first control signal generated in the first period.
  • a control method described in any one of Appendices B48 to B57 is repeat the second movement process for the first part during a first period from the completion of the first movement process for the first part to the start of the first movement process for the second part until the attitude relationship between the processing device and the first part becomes a first attitude relationship; repeat the second movement process for the second part during a second period after the first movement process for the second part is started, until the attitude relationship between the processing device and the second part becomes a second attitude relationship; In the second period, the second control signal is generated based on the first control signal generated in the first period.
  • a control method described in any one of Appendices B48 to B58 is repeat the second movement process for the first part during a first period from the completion of the first movement process for the first part to the start of the first movement process for the second part until the attitude relationship between the processing device and the first part becomes a first attitude relationship; repeat the second movement process for the second part during a second period after the first movement process for the second part is started, until the attitude relationship between the processing
  • the control method includes generating the second control signal that is generated first in the second period based on the first control signal that is generated last in the first period. Control method described in Appendix B58 or B59. [Appendix B61] The control method includes generating the second control signal by using the first control signal as an initial signal for the second control signal. A control method described in any one of Appendices B48 to B60.
  • the control method is repeat the second movement process for the first part during a first period from the completion of the first movement process for the first part to the start of the first movement process for the second part until the positional relationship between the processing device and the first part becomes a first positional relationship; repeat the second movement process for the second part during a second period after the first movement process for the second part is started, until the positional relationship between the processing device and the second part becomes a second positional relationship; During the second period, the first control signal generated during the first period is used as an initial signal for the second control signal to generate the second control signal.
  • a control method described in any one of Appendices B48 to B61 including: [Appendix B63]
  • the control method is repeat the second movement process for the first part during a first period from the completion of the first movement process for the first part to the start of the first movement process for the second part until the attitude relationship between the processing device and the first part becomes a first attitude relationship; repeat the second movement process for the second part during a second period after the first movement process for the second part is started, until the attitude relationship between the processing device and the second part becomes a second attitude relationship;
  • the first control signal generated during the first period is used as an initial signal for the second control signal to generate the second control signal.
  • a control method described in any one of Appendices B48 to B62 including: [Appendix B64]
  • the control method includes generating the second control signal that is generated first in the second period by using the first control signal that is generated last in the first period as an initial signal for the second control signal.
  • the target object includes a first target object to be processed by the processing device, and a second target object that is different from the first target object and is to be processed by the processing device,
  • the first movement process includes the first movement process for the first target object and the first movement process for the second target object
  • the control method includes, as at least a part of the second movement process, performing the second movement process on the first target object and the second movement process on the second target object,
  • the first movement process for the first target object includes a process in which the processing device controls the robot to approach the first target object,
  • the control method includes, as at least a part of the second movement process for the first target object, performing a process of generating and outputting a third control signal as the control signal for controlling the robot to move the processing device based on a calculation result of at least one of the position and orientation of the first target object calculated based on the image data generated by the imaging device capturing an image of the first target object after the first movement process for the first target object has been started;
  • the control method includes repeating the second movement process for the first target object until the positional relationship between the processing device and the first target object becomes a third positional relationship between the first movement process for the first target object and the start of the first movement process for the second target object.
  • Appendix B67 The third positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the first target object. Control method described in Appendix B66.
  • the control method includes repeating the second movement process for the first target object until the attitude relationship between the processing device and the first target object becomes a third attitude relationship between the first movement process for the first target object and the start of the first movement process for the second target object.
  • the third attitude relationship includes an attitude relationship in which the processing device assumes a target processing attitude that the processing device should assume when performing processing on the first target object.
  • the control method includes repeating the second movement process for the second target object after the first movement process for the second target object is started, until the positional relationship between the processing device and the second target object becomes a fourth positional relationship.
  • the fourth positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the second target object.
  • the control method includes repeating the second movement process for the second target object after the first movement process for the second target object is started, until the attitude relationship between the processing device and the second target object becomes a fourth attitude relationship.
  • the fourth attitude relationship includes an attitude relationship in which the processing device assumes a target processing attitude that the processing device should assume when performing processing on the second target object.
  • the control method includes generating the fourth control signal based on the third control signal.
  • the control method is repeat the second movement process for the first target object during a third period from the completion of the first movement process for the first target object to the start of the first movement process for the second target object until the positional relationship between the processing device and the first target object becomes a third positional relationship; repeat the second movement process for the second target object during a fourth period after the first movement process for the second target object is started, until the positional relationship between the processing device and the second target object becomes a fourth positional relationship; In the fourth period, the fourth control signal is generated based on the third control signal generated in the third period.
  • the control method is repeat the second movement process for the first target object during a third period from the completion of the first movement process for the first target object to the start of the first movement process for the second target object until the attitude relationship between the processing device and the first target object becomes a third attitude relationship; repeat the second movement process for the second target object during a fourth period after the first movement process for the second target object is started, until the attitude relationship between the processing device and the second target object becomes a fourth attitude relationship; In the fourth period, the fourth control signal is generated based on the third control signal generated in the third period.
  • the control method includes generating the fourth control signal that is generated first in the fourth period based on the third control signal that is generated last in the third period. Control method described in Appendix B75 or B76. [Appendix B78] The control method includes generating the fourth control signal by using the third control signal as an initial signal for the fourth control signal. A control method described in any one of Appendices B65 to B77.
  • the control method is repeat the second movement process for the first target object during a third period from the completion of the first movement process for the first target object to the start of the first movement process for the second target object until the positional relationship between the processing device and the first target object becomes a third positional relationship; repeat the second movement process for the second target object during a fourth period after the first movement process for the second target object is started, until the positional relationship between the processing device and the second target object becomes a fourth positional relationship; During the fourth period, the third control signal generated during the third period is used as an initial signal for the fourth control signal to generate the fourth control signal.
  • the control method is repeat the second movement process for the first target object during a third period from the completion of the first movement process for the first target object to the start of the first movement process for the second target object until the attitude relationship between the processing device and the first target object becomes a third attitude relationship; repeat the second movement process for the second target object during a fourth period after the first movement process for the second target object is started, until the attitude relationship between the processing device and the second target object becomes a fourth attitude relationship; In the fourth period, the third control signal generated in the third period is used as an initial signal for the fourth control signal to generate the fourth control signal.
  • the control method includes generating the fourth control signal that is generated first in the fourth period by using the third control signal that is generated last in the third period as an initial signal for the fourth control signal.
  • the control method includes performing the second movement process using a second control device and performing the first movement process using a first control device different from the second control device,
  • the control device is a second control device
  • the first movement process is performed by a first control device different from the second control device
  • the target object has a first portion that is processed by the processing device, and a second portion that is different from the first portion and is also processed by the processing device
  • the control method is using the first control device, as at least a part of the first movement process, performing the first movement process for the first part and the first movement process for the second part; using the second control device, as at least a part of the second movement process, performing the second movement process for the first part and the second movement process for the second part;
  • Using the first control device, as at least a part of the first movement process for the first part a process of controlling the robot so that the processing device approaches the first part
  • using the second control device as at least a part of the second movement process for the first part, after the first movement process for the
  • the control method includes performing the first movement process for the second part using the second control device instead of using the first control device.
  • the control method includes using the second control device to perform, as at least a part of the first movement process for the second part, a process in which the processing device controls the robot so as to approach the second part from the first part even if the second control device does not receive a signal transmitted from the first control device after the second movement process for the first part. Control method described in Appendix B83.
  • the control method includes repeating the second movement process for the first part using the second control device until the positional relationship between the processing device and the first part becomes a first positional relationship between the completion of the first movement process for the first part and the start of the first movement process for the second part.
  • the first positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the first portion. Control method described in Appendix B85.
  • the control method includes repeating, using the second control device, the second movement process for the first part until the attitude relationship between the processing device and the first part becomes a first attitude relationship between the completion of the first movement process for the first part and the start of the first movement process for the second part.
  • the first attitude relationship includes an attitude relationship in which the processing device assumes a target processing attitude that the processing device should assume when performing processing on the first portion. Control method described in Appendix B87.
  • the control method includes repeating the second movement process for the second part using the second control device after the first movement process for the second part is started, until the positional relationship between the processing device and the second part becomes a second positional relationship.
  • the second positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the second portion.
  • the control method includes repeating, using the second control device, the second movement process for the second part after the first movement process for the second part is started, until the attitude relationship between the processing device and the second part becomes a second attitude relationship.
  • the second attitude relationship includes an attitude relationship in which the processing device is taking a target processing attitude that the processing device should take when performing processing on the second part.
  • the control method includes generating the second control signal based on the first control signal using the second control device.
  • the control method is using the second control device, repeating the second movement process for the first part during a first period from when the first movement process for the first part is completed to when the first movement process for the second part is started, until the positional relationship between the processing device and the first part becomes a first positional relationship; using the second control device, during a second period after the first movement process for the second part is started, repeating the second movement process for the second part until the positional relationship between the processing device and the second part becomes a second positional relationship; In the second period, the second control signal is generated based on the first control signal generated in the first period.
  • the control method is using the second control device, repeating the second movement process for the first part during a first period from the completion of the first movement process for the first part to the start of the first movement process for the second part until the attitude relationship between the processing device and the first part becomes a first attitude relationship; using the second control device, during a second period after the first movement process for the second part is started, repeating the second movement process for the second part until the attitude relationship between the processing device and the second part becomes a second attitude relationship; Using the second control device, during the second period, generate the second control signal based on the first control signal generated during the first period.
  • a control method described in any one of Appendices B82 to B94 includes using the second control device to generate the second control signal that is generated first in the second period based on the first control signal that is generated last in the first period.
  • Control method described in Appendix B94 or B95 Control method described in Appendix B94 or B95.
  • the control method includes generating the second control signal by using the second control device and using the first control signal as an initial signal for the second control signal.
  • a control method described in any one of Appendices B82 to B96 includes using the second control device to generate the second control signal that is generated first in the second period based on the first control signal that is generated last in the first period.
  • the control method is using the second control device, repeating the second movement process for the first part during a first period from when the first movement process for the first part is completed to when the first movement process for the second part is started, until the positional relationship between the processing device and the first part becomes a first positional relationship; using the second control device, during a second period after the first movement process for the second part is started, repeating the second movement process for the second part until the positional relationship between the processing device and the second part becomes a second positional relationship; Using the second control device, in the second period, generate the second control signal by using the first control signal generated in the first period as an initial signal for the second control signal.
  • the control method is using the second control device, repeating the second movement process for the first part during a first period from the completion of the first movement process for the first part to the start of the first movement process for the second part until the attitude relationship between the processing device and the first part becomes a first attitude relationship; using the second control device, during a second period after the first movement process for the second part is started, repeating the second movement process for the second part until the attitude relationship between the processing device and the second part becomes a second attitude relationship; Using the second control device, in the second period, generate the second control signal by using the first control signal generated in the first period as an initial signal for the second control signal.
  • the control method includes using the second control device to generate the second control signal that is generated first in the second period by using the first control signal that is generated last in the first period as an initial signal for the second control signal. Control method described in Appendix B98 or B99.
  • the control method includes performing the second movement process using a second control device and performing the first movement process using a first control device different from the second control device,
  • the target object includes a first target object to be processed by the processing device, and a second target object that is different from the first target object and is to be processed by the processing device,
  • the control method is using the first control device, as at least a part of the first movement process, performing the first movement process for the first target object and the first movement process for the second target object; using the second control device, as at least a part of the second movement process, performing the second movement process for the first target object and the second movement process for the second target object;
  • the processing device controls the robot to approach the first target object;
  • Using the second control device as at least a part of the second movement process for the first target object, after the first movement process for the first target object has been initiated, a process is performed in which a third control signal is generated as the
  • a control method described in any one of Appendices B1 to B102 including: [Appendix B102]
  • the control method includes performing the first movement process on the second target object using the second control device instead of using the first control device.
  • the control method includes using the second control device to perform, as at least a part of the first movement process for the second target object, a process in which the processing device controls the robot to move from the first target object toward the second target object even if the second control device does not receive a signal transmitted from the first control device after the second movement process for the first target object.
  • Control method described in Appendix B102 Control method described in Appendix B102.
  • the control method includes repeating, using the second control device, the second movement process for the first target object until the positional relationship between the processing device and the first target object becomes a third positional relationship between the first movement process for the first target object and the start of the first movement process for the second target object.
  • the third positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the first target object. Control method described in Appendix B104.
  • the control method includes repeating, by using the second control device, the second movement process for the first target object until the attitude relationship between the processing device and the first target object becomes a third attitude relationship, between the completion of the first movement process for the first target object and the start of the first movement process for the second target object.
  • the third attitude relationship includes an attitude relationship in which the processing device assumes a target processing attitude that the processing device should assume when performing processing on the first target object. Control method described in Appendix B106.
  • the control method includes repeating, using the second control device, the second movement process for the second target object after the first movement process for the second target object is started, until the positional relationship between the processing device and the second target object becomes a fourth positional relationship.
  • the fourth positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the second target object. Control method described in Appendix B108.
  • the control method includes repeating, using the second control device, the second movement process for the second target object after the first movement process for the second target object is started, until the attitude relationship between the processing device and the second target object becomes a fourth attitude relationship.
  • the fourth attitude relationship includes an attitude relationship in which the processing device assumes a target processing attitude that the processing device should assume when performing processing on the second target object.
  • the control method includes generating the fourth control signal based on the third control signal using the second control device.
  • the control method is using the second control device, repeating the second movement process for the first target object during a third period from the completion of the first movement process for the first target object to the start of the first movement process for the second target object until the positional relationship between the processing device and the first target object becomes a third positional relationship; using the second control device, during a fourth period after the first movement process for the second target object is started, repeating the second movement process for the second target object until the positional relationship between the processing device and the second target object becomes a fourth positional relationship; Using the second control device, in the fourth period, generate the fourth control signal based on the third control signal generated in the third period.
  • the control method is using the second control device, repeating the second movement process for the first target object during a third period from the completion of the first movement process for the first target object to the start of the first movement process for the second target object until the attitude relationship between the processing device and the first target object becomes a third attitude relationship; using the second control device, during a fourth period after the first movement process for the second target object is started, repeating the second movement process for the second target object until the attitude relationship between the processing device and the second target object becomes a fourth attitude relationship; Using the second control device, in the fourth period, generate the fourth control signal based on the third control signal generated in the third period.
  • the control method includes using the second control device to generate the fourth control signal that is generated first in the fourth period based on the third control signal that is generated last in the third period.
  • the control method includes generating the fourth control signal by using the second control device and using the third control signal as an initial signal for the fourth control signal.
  • the control method is using the second control device, repeating the second movement process for the first target object during a third period from the completion of the first movement process for the first target object to the start of the first movement process for the second target object until the positional relationship between the processing device and the first target object becomes a third positional relationship; using the second control device, during a fourth period after the first movement process for the second target object is started, repeating the second movement process for the second target object until the positional relationship between the processing device and the second target object becomes a fourth positional relationship; Using the second control device, in the fourth period, generate the fourth control signal by using the third control signal generated in the third period as an initial signal for the fourth control signal.
  • the control method is using the second control device, repeating the second movement process for the first target object during a third period from the completion of the first movement process for the first target object to the start of the first movement process for the second target object until the attitude relationship between the processing device and the first target object becomes a third attitude relationship; using the second control device, during a fourth period after the first movement process for the second target object is started, repeating the second movement process for the second target object until the attitude relationship between the processing device and the second target object becomes a fourth attitude relationship; Using the second control device, in the fourth period, generate the fourth control signal by using the third control signal generated in the third period as an initial signal for the fourth control signal.
  • the control method includes using the second control device to generate the fourth control signal that is generated first in the fourth period by using the third control signal that is generated last in the third period as an initial signal for the fourth control signal.
  • the first movement process includes a process of controlling the robot to move the processing device according to movement instruction information.
  • the movement instruction information is set in advance by the user inputting the movement instruction information. Control method described in Appendix B120.
  • the movement instruction information includes at least one of a target movement position indicating a position to which the processing device should move, a target movement distance indicating a distance to which the processing device should move, and a target movement direction indicating a direction to which the processing device should move. Control method described in Appendix B120 or B121.
  • the target movement position is fixed to a pre-set position during the period in which the first movement process is being performed, the target movement distance is fixed to a pre-set distance during the period in which the first movement process is being performed, and the target movement direction is fixed to a pre-set direction during the period in which the first movement process is being performed. Control method described in Appendix B122.
  • the target object is moved by a transport device capable of transporting the target object, At least one of the target movement position, the target movement distance, and the target movement direction is preset based on at least one of the conveying speed of the target member by the conveying device and the conveying direction of the target member by the conveying device. Control method described in Appendix B122 or B123.
  • the target object has a first portion that is processed by the processing device, and a second portion that is different from the first portion and is also processed by the processing device, The processing device processes the first portion, and then processes the second portion, At least one of the target movement position, the target movement distance, and the target movement direction is preset based on the positional relationship between the first part and the second part.
  • the target object includes a first target object to be processed by the processing device, and a second target object that is different from the first target object and is to be processed by the processing device, The processing device performs processing on the first target object, and then performs processing on the second target object, At least one of the target movement position, the target movement distance, and the target movement direction is preset based on the positional relationship between the first target object and the second target object.
  • the first movement process is based on open loop control.
  • the second movement process is a process based on closed-loop control.
  • the closed loop control is at least one of P (ProportionBl) control, PI (ProportionBl-IntegrBl) control, and PID (ProportionBl-IntegrBl-Derivative) control.
  • Appendix B130 The target object is moved by a transport device capable of transporting the target object, The first movement process and the second movement process are performed on the target object being moved by the transport device.
  • the second movement process includes a process of calculating at least one of the position and the orientation of the target object by performing a matching process using the image data generated by the imaging device capturing an image of the target object after the first movement process is started and model data indicating an object model that is a model of the target object.
  • the object model is a model of at least a portion of the edge of the target object. Control method described in Appendix B131.
  • the model data is data that indicates an edge of at least a portion of the target object or the sample in an image obtained by imaging the target object or a sample object having the same shape as the target object in advance.
  • the second movement process includes a process of generating and outputting the control signal for controlling the robot to move the processing device based on at least one of the calculation results of the position and orientation of the target object and at least one of the calculation results of the position and orientation of the processing device, which are calculated based on the image data generated by the imaging device capturing an image of the target object and the processing device after the first movement process is started.
  • the second movement process includes a process of calculating at least one of the position and orientation of the target object and at least one of the position and orientation of the processing device by performing a matching process using the image data generated by the imaging device capturing an image of the target object and the processing device after the first movement process is started, and model data indicating an object model that is a model of the target object and a device model that is a model of the processing device.
  • the object model is a model of at least a portion of the edge of the target object
  • the device model is a model of at least a portion of the edge of the processing device. Control method described in Appendix B135.
  • the model data indicates the object model and the device model aligned with each other
  • the second movement process includes a process of generating the control signal for controlling the robot so that the difference between the positional relationship between the target object and the processing device and the positional relationship between the object model and the device model that are aligned with each other is within a threshold position by moving the processing device, based on at least one of the calculation results of the position and orientation of the target object and at least one of the calculation results of the position and orientation of the processing device. Control method described in Appendix B135 or B136.
  • the model data indicates the object model and the device model aligned with each other
  • the second movement process includes a process of generating the control signal for controlling the robot so that the difference between the posture relationship between the target object and the processing device and the posture relationship between the object model and the device model that are aligned with each other is within a threshold value by moving the processing device, based on at least one of the calculation results of the position and posture of the target object and at least one of the calculation results of the position and posture of the processing device.
  • the model data indicates the object model and the device model aligned with each other
  • the second movement process includes a process of selecting, when a plurality of target objects are detected by the matching process, one target object that satisfies the condition that the positional relationship between the target object and the processing device is closest to the positional relationship between the object model and the device model that are aligned with each other, as the target object to be processed by the processing device.
  • the control method includes performing a preparatory process for performing the second movement process during a period in which the first movement process is being performed and before the second movement process is started.
  • the control method includes using a control device to perform a preparatory process for performing the second movement process before starting the second movement process.
  • the control method includes using the control device to perform a preparatory process for performing the second movement process during a period in which the first movement process is being performed and before starting the second movement process.
  • the preparation process includes a process of acquiring information used to perform the second movement process.
  • the information used to perform the second movement processing includes target object information regarding the target object on which the processing device performs the predetermined processing.
  • the target object information includes information regarding the type of the target object.
  • the information used to perform the second movement processing includes peripheral object information regarding peripheral objects located in the vicinity of the target object on which the processing device performs the predetermined processing.
  • the surrounding object information includes information regarding at least one of the position of the surrounding object, the orientation of the surrounding object, the shape of the surrounding object, and the size of the surrounding object.
  • the second movement process includes a process of calculating at least one of the position and the orientation of the target object by performing a matching process using the image data generated by the imaging device capturing an image of the target object after the first movement process is started and model data indicating an object model that is a model of the target object,
  • the information used to perform the second movement process includes matching information, which is information used in the matching process.
  • the matching information includes at least one of information specifying the model data, information regarding a matching judgment threshold used in the matching process, and information regarding a convergence condition that must be satisfied to terminate the second movement process.
  • the information used to perform the second movement process includes imaging condition information, which is information regarding the imaging conditions of the imaging device.
  • imaging condition information includes at least one of information regarding exposure time and information regarding the intensity of illumination light.
  • the information used to perform the second movement process includes the image data generated by the imaging device capturing an image of the target object after the first movement process is started. A control method described in any one of Appendices B143 to B151.
  • the information used to perform the second movement process includes information regarding the calculation result of at least one of the position and orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object after the first movement process is started.
  • the preparation process includes a process of acquiring the image data generated by the imaging device capturing an image of the target object after the first movement process is started.
  • the preparation process includes a process of generating the image data by capturing an image of the target object using the imaging device after the first movement process is started.
  • the preparation process includes a process of calculating at least one of the position and the orientation of the target object based on the image data generated by the imaging device capturing an image of the target object after the first movement process is started.
  • the preparation process includes a process for adjusting the exposure conditions of the imaging device.
  • the imaging device is provided with an illumination device capable of illuminating the target object with illumination light
  • the preparation process includes a process of causing the lighting device to start illuminating the target object with the illumination light.
  • the second movement process includes, when it becomes impossible to calculate at least one of the position and orientation of the target object based on image data generated by imaging by the imaging device at a first time during the period in which the second movement process is being performed, a process of calculating at least one of the position and orientation of the target object at the first time based on at least one of the position and orientation of the target object calculated based on the image data generated by imaging the target object by the imaging device at a second time prior to the first time.
  • the second movement process includes a process of generating and outputting a control signal for controlling the robot to move the processing device based on information regarding contact between the processing device and the target object when it becomes impossible to calculate at least one of the position and orientation of the target object based on the image data generated by imaging by the imaging device during the period in which the second movement process is being performed.
  • the control method includes inspecting the target object based on the image data.
  • the control method includes inspecting the target object based on a measurement result of the target object by a measuring device capable of measuring the target object.
  • the control method includes inspecting the target object before the processing device processes the target object.
  • the control method includes inspecting the target object after the processing device processes the target object.
  • the control method includes inspecting the results of the processing performed by the processing device on the target object based on the image data or the measurement results of the measuring device.
  • the control method includes inspecting the target object based on the image data using the control device.
  • a control method described in any one of Appendices B1 to B165 includes inspecting the target object using the control device based on a measurement result of the target object by a measuring device capable of measuring the target object.
  • a control method described in any one of Appendices B1 to B166 includes inspecting the target object using the control device before the processing device processes the target object.
  • Control method described in Appendix B166 or B167 includes inspecting the target object using the control device after the processing device performs processing on the target object.
  • the control method includes inspecting the results of processing performed by the processing device on the target object based on the image data or the measurement results of the measuring device using the control device.
  • the target object is moved by a transport device capable of transporting the target object
  • the control method includes performing the second movement process while the transport device is moving the target object
  • the second movement process includes a process of generating a control signal for controlling the robot to move the processing device based on a transport state of the target object when it becomes impossible to calculate at least one of the position and the orientation of the target object based on image data generated by imaging by the imaging device at a first time during the period in which the second movement process is being performed.
  • the target object is moved by a transport device capable of transporting the target object
  • the control method includes performing the second movement process while the transport device is moving the target object
  • the second movement process includes a process of generating a control signal for controlling the robot to move the processing device based on a transport state of the target object when it becomes impossible to calculate at least one of the position and the orientation of the target object based on image data generated by imaging by the imaging device at a first time during the period in which the second movement process is being performed.
  • the transport state of the target object includes at least one of the transport direction of the target object, the transport distance of the target object, and the transport speed of the target object.
  • the transport state of the target object includes the transport direction of the target object
  • the second movement process includes a process of generating a control signal for controlling the robot to move the processing device in a movement direction determined according to the transport direction of the target object.
  • the transport state of the target object includes the transport distance of the target object
  • the second movement process includes a process of generating a control signal for controlling the robot to move the processing device by a movement distance determined according to the transport distance of the target object.
  • the transport state of the target object includes the transport speed of the target object
  • the second movement process includes a process of estimating a transport distance of the target object based on the transport speed of the target object, and a process of generating a control signal for controlling the robot to move the processing device by a travel distance determined according to the estimated result of the transport distance of the target object.
  • the transport state of the target object includes the transport direction of the target object and the transport distance of the target object
  • the second movement process includes a process of generating a control signal for controlling the robot to move the processing device in a movement direction determined according to the transport direction of the target object by a movement distance determined according to the transport distance of the target object.
  • the transport state of the target object includes the transport direction of the target object and the transport speed of the target object
  • the second movement process includes a process of estimating a transport distance of the target object based on the transport speed of the target object, and a process of generating a control signal for controlling the robot to move the processing device by a travel distance determined according to the estimated result of the transport distance of the target object in a travel direction determined according to the transport direction of the target object.
  • the second movement process includes a process of acquiring information about the transport state of the target object from the transport device or a device that controls the transport device, and a process of generating a control signal for controlling the robot to move the processing device based on the acquired information.
  • the second movement process includes a process of estimating a transport state of the target object based on a calculation result of at least one of the position and the orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object during a period prior to the first time, and a process of generating a control signal for controlling the robot to move the processing device based on the estimation result of the transport state of the target object.
  • the control method is performed by a control system, The control system includes the imaging device.
  • the control method is performing the second movement process using a second control device provided in the control system;
  • the first movement process is performed using a first control device that is provided in the control system and is different from the second control device.
  • the imaging device is attached to the robot.
  • the robot is a first robot,
  • the imaging device is attached to a second robot that is different from the first robot.
  • the imaging device is attached to a support member that is different from the robot and can support the imaging device.
  • a control method described in any one of Appendices B1 to B184 A control method described in any one of Appendices B1 to B184.
  • Appendix B186 A control device that performs the control method described in any one of appendices B1 to B185; The robot and A robot system equipped with [Appendix B187] Further comprising the processing device that performs the processing on the target object A robot system as described in Appendix B186.
  • Appendix B188 A computer program that causes a computer to execute a control method described in any one of Appendices B1 to B185.
  • Appendix B189 A recording medium on which the computer program described in Appendix B188 is recorded.

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Automation & Control Theory (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Manipulator (AREA)
PCT/JP2023/017026 2023-05-01 2023-05-01 制御システム、ロボットシステム、制御方法及びコンピュータプログラム Ceased WO2024228231A1 (ja)

Priority Applications (5)

Application Number Priority Date Filing Date Title
CN202380100006.3A CN121464022A (zh) 2023-05-01 2023-05-01 控制系统、机器人系统、控制方法和计算机程序
KR1020257039892A KR20260005965A (ko) 2023-05-01 2023-05-01 제어 시스템, 로봇 시스템, 제어 방법 및 컴퓨터 프로그램
PCT/JP2023/017026 WO2024228231A1 (ja) 2023-05-01 2023-05-01 制御システム、ロボットシステム、制御方法及びコンピュータプログラム
JP2025518059A JPWO2024228231A1 (https=) 2023-05-01 2023-05-01
EP23935746.0A EP4706910A1 (en) 2023-05-01 2023-05-01 Control system, robot system, control method, and computer program

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2023/017026 WO2024228231A1 (ja) 2023-05-01 2023-05-01 制御システム、ロボットシステム、制御方法及びコンピュータプログラム

Publications (1)

Publication Number Publication Date
WO2024228231A1 true WO2024228231A1 (ja) 2024-11-07

Family

ID=93332895

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/017026 Ceased WO2024228231A1 (ja) 2023-05-01 2023-05-01 制御システム、ロボットシステム、制御方法及びコンピュータプログラム

Country Status (5)

Country Link
EP (1) EP4706910A1 (https=)
JP (1) JPWO2024228231A1 (https=)
KR (1) KR20260005965A (https=)
CN (1) CN121464022A (https=)
WO (1) WO2024228231A1 (https=)

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0976185A (ja) * 1995-07-12 1997-03-25 Fanuc Ltd ロボットの移動制御方式
US20130230235A1 (en) 2010-11-19 2013-09-05 Canon Kabushiki Kaisha Information processing apparatus and information processing method
JP2015000454A (ja) * 2013-06-17 2015-01-05 キヤノン株式会社 ロボット装置及びロボット制御方法
JP2015003348A (ja) * 2013-06-19 2015-01-08 セイコーエプソン株式会社 ロボット制御システム、制御装置、ロボット、ロボット制御システムの制御方法及びロボットの制御方法
JP2015174206A (ja) * 2014-03-18 2015-10-05 セイコーエプソン株式会社 ロボット制御装置、ロボットシステム、ロボット、ロボット制御方法及びロボット制御プログラム
JP2016148649A (ja) * 2015-02-05 2016-08-18 キヤノン株式会社 情報処理装置、情報処理装置の制御方法、およびプログラム
JP2018001372A (ja) * 2016-07-06 2018-01-11 島根県 部品供給装置及び方法
JP2020157461A (ja) * 2019-03-28 2020-10-01 セイコーエプソン株式会社 ロボットシステムおよびロボットシステムの制御方法
JP2022073012A (ja) * 2020-10-30 2022-05-17 セイコーエプソン株式会社 ロボット制御方法およびロボットシステム
JP2023010327A (ja) * 2021-07-09 2023-01-20 株式会社不二越 ロボットビジョンシステム

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0976185A (ja) * 1995-07-12 1997-03-25 Fanuc Ltd ロボットの移動制御方式
US20130230235A1 (en) 2010-11-19 2013-09-05 Canon Kabushiki Kaisha Information processing apparatus and information processing method
JP2015000454A (ja) * 2013-06-17 2015-01-05 キヤノン株式会社 ロボット装置及びロボット制御方法
JP2015003348A (ja) * 2013-06-19 2015-01-08 セイコーエプソン株式会社 ロボット制御システム、制御装置、ロボット、ロボット制御システムの制御方法及びロボットの制御方法
JP2015174206A (ja) * 2014-03-18 2015-10-05 セイコーエプソン株式会社 ロボット制御装置、ロボットシステム、ロボット、ロボット制御方法及びロボット制御プログラム
JP2016148649A (ja) * 2015-02-05 2016-08-18 キヤノン株式会社 情報処理装置、情報処理装置の制御方法、およびプログラム
JP2018001372A (ja) * 2016-07-06 2018-01-11 島根県 部品供給装置及び方法
JP2020157461A (ja) * 2019-03-28 2020-10-01 セイコーエプソン株式会社 ロボットシステムおよびロボットシステムの制御方法
JP2022073012A (ja) * 2020-10-30 2022-05-17 セイコーエプソン株式会社 ロボット制御方法およびロボットシステム
JP2023010327A (ja) * 2021-07-09 2023-01-20 株式会社不二越 ロボットビジョンシステム

Also Published As

Publication number Publication date
JPWO2024228231A1 (https=) 2024-11-07
CN121464022A (zh) 2026-02-03
EP4706910A1 (en) 2026-03-11
KR20260005965A (ko) 2026-01-12

Similar Documents

Publication Publication Date Title
US20230286167A1 (en) Robot choreographer
US10140010B2 (en) Moving an object by drag operation on a touch panel
JP6657469B2 (ja) ロボットシステムの自動較正方法
CN107787497B (zh) 用于在基于用户的空间坐标系中检测手势的方法和装置
KR20230006507A (ko) 계단 트랙커를 위한 인식 및 피팅
EP3354418B1 (en) Robot control method and device
WO2021238222A1 (zh) 机器人避障方法、机器人避障装置、机器人以及存储介质
CN110053039A (zh) 一种机器人行走中重力补偿的方法、装置及机器人
EP4263152A1 (en) Supervised autonomous grasping
CN107457783B (zh) 基于pd控制器的六自由度机械臂自适应智能检测方法
CN113001542B (zh) 一种机器人轨迹规划方法、装置、可读存储介质及机器人
WO2024228231A1 (ja) 制御システム、ロボットシステム、制御方法及びコンピュータプログラム
Zhang et al. Deep learning-based robot vision: High-end tools for smart manufacturing
JP7079047B1 (ja) 作業システム、作業方法
Ryll et al. Towards an intelligent fixturing system with rapid reconfiguration and part positioning
CN106141810A (zh) 机器人操作下筒状工件内腔加工壁厚的保证方法
EP3786633B1 (en) Display control system, inspection control system, display control method, and storage medium
Ghiyasi et al. Layer-by-layer pick and place collaboration between human and robot using optimization
CN114815596B (zh) 一种步态规划方法、装置、计算机可读存储介质及机器人
CN115338862B (zh) 一种基于部分可观测马尔科夫的机械手移动路径规划方法
EP4516465A1 (en) Control device, control system, robot system, control method, and computer program
CN105499676B (zh) 机壳亮边的走刀路径的获取方法及仿真加工方法
Hurnen Overcoming the Limitations of Standard LIDAR in Autonomous Robots using Ultrasonic Sensors
CN119820576B (zh) 一种四足单臂机器人协同运动控制方法及系统
CN116652928A (zh) 直角墙砖块的砌筑方法、砌砖机器人、系统及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23935746

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2025518059

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2025518059

Country of ref document: JP

ENP Entry into the national phase

Ref document number: 1020257039892

Country of ref document: KR

Free format text: ST27 STATUS EVENT CODE: A-0-1-A10-A15-NAP-PA0105 (AS PROVIDED BY THE NATIONAL OFFICE)

WWE Wipo information: entry into national phase

Ref document number: 2023935746

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2023935746

Country of ref document: EP

Effective date: 20251201

ENP Entry into the national phase

Ref document number: 2023935746

Country of ref document: EP

Effective date: 20251201

ENP Entry into the national phase

Ref document number: 2023935746

Country of ref document: EP

Effective date: 20251201

ENP Entry into the national phase

Ref document number: 2023935746

Country of ref document: EP

Effective date: 20251201

ENP Entry into the national phase

Ref document number: 2023935746

Country of ref document: EP

Effective date: 20251201

ENP Entry into the national phase

Ref document number: 2023935746

Country of ref document: EP

Effective date: 20251201

ENP Entry into the national phase

Ref document number: 2023935746

Country of ref document: EP

Effective date: 20251201

ENP Entry into the national phase

Ref document number: 2023935746

Country of ref document: EP

Effective date: 20251201

ENP Entry into the national phase

Ref document number: 2023935746

Country of ref document: EP

Effective date: 20251201

WWP Wipo information: published in national office

Ref document number: 2023935746

Country of ref document: EP