WO2023209974A1 - 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム - Google Patents

制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム Download PDF

Info

Publication number
WO2023209974A1
WO2023209974A1 PCT/JP2022/019364 JP2022019364W WO2023209974A1 WO 2023209974 A1 WO2023209974 A1 WO 2023209974A1 JP 2022019364 W JP2022019364 W JP 2022019364W WO 2023209974 A1 WO2023209974 A1 WO 2023209974A1
Authority
WO
WIPO (PCT)
Prior art keywords
imaging
image data
similarity
object obj
coordinate system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/019364
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
大輔 小柳
祥也 横田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nikon Corp
Original Assignee
Nikon Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nikon Corp filed Critical Nikon Corp
Priority to JP2024517781A priority Critical patent/JPWO2023209974A1/ja
Priority to CN202280096481.3A priority patent/CN119300961A/zh
Priority to EP22940248.2A priority patent/EP4516465A4/en
Priority to PCT/JP2022/019364 priority patent/WO2023209974A1/ja
Priority to KR1020247036541A priority patent/KR20250003671A/ko
Priority to US18/859,185 priority patent/US20250326125A1/en
Publication of WO2023209974A1 publication Critical patent/WO2023209974A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1694—Program controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • B25J13/088—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices with position, velocity or acceleration sensors
    • B25J13/089—Determining the position of the robot with reference to its environment
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
    • B25J19/02—Sensing devices
    • B25J19/021—Optical sensing devices
    • B25J19/023—Optical sensing devices including video camera means
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1656—Program controls characterised by programming, planning systems for manipulators
    • B25J9/1664—Program controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1679—Program controls characterised by the tasks executed
    • B25J9/1692—Calibration of manipulator
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/251—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving models
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681—Motion detection
    • H04N23/6812—Motion detection based on additional sensors, e.g. acceleration sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/37—Measurements
    • G05B2219/37555—Camera detects orientation, position workpiece, points of workpiece
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/39—Robotics, robotics to robotics hand
    • G05B2219/39057—Hand eye calibration, eye, camera on hand, end effector
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40564—Recognize shape, contour of object, extract position and orientation
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40577—Multisensor object recognition
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40584—Camera, non-contact sensor mounted on wrist, indep from gripper
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40613—Camera, laser scanner on end effector, hand eye manipulator, local
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30164—Workpiece; Machine component

Definitions

  • the present invention relates to the technical field of, for example, a control device, a control system, a robot system, a control method, and a computer program that can generate a control signal for controlling a robot arm.
  • Patent Document 1 An example of a control device that calculates at least one of the position and orientation of an object that is a target of processing performed by the robot and controls the robot based on at least one of the calculated position and orientation is described in Patent Document 1. There is. Such a control device is required to accurately calculate at least one of the position and orientation of an object.
  • a robot arm is provided with a processing device that processes an object, a first imaging device that outputs first image data, and a second imaging device that outputs second image data.
  • a control device that generates a control signal for controlling the control device, wherein the control signal controls the robot arm to move the processing device and the first and second imaging devices, and the control device includes a calculation unit that generates the control signal, and an output unit that outputs the control signal generated by the calculation unit, and the calculation unit generates the control signal generated by the first imaging device imaging the object.
  • first information indicating the position and orientation of the object using the first image data, and generating information from the second image data generated by the second imaging device imaging the object; and determining the position of the object using three-dimensional position data indicating the three-dimensional positions of each of the plurality of points on the object and three-dimensional model data of the object at the position and orientation determined based on the first information.
  • a control device that generates second information indicating a posture and generates the control signal based on the second information.
  • the robot arm is provided with a processing device that processes an object, a first imaging device that outputs first image data, and a second imaging device that outputs second image data.
  • a control device that generates a control signal for controlling the control device, wherein the control signal controls the robot arm to move the processing device and the first and second imaging devices, and the control device includes a calculation unit that generates the control signal, and an output unit that outputs the control signal generated by the calculation unit, and the calculation unit outputs the control signal to the first imaging device at mutually different first and second times. is generated from two first image data, each generated by imaging the object, and the second image data generated by the second imaging device imaging the object at a third time, and the second image data is generated by imaging the object at a third time.
  • the amount of change in the position and orientation of the object between the first and second times is calculated based on three-dimensional position data indicating the three-dimensional position of each of the plurality of points on the object, and the control is performed based on the amount of change.
  • a controller is provided that generates a signal.
  • a processing device that performs processing on at least one of a first object and a second object in which at least one of a position and an orientation differs from the first object; and a processing device that outputs first image data.
  • a control device that generates a control signal for controlling a robot arm that is provided with a first imaging device and a second imaging device that outputs second image data, the control device generating a control signal that controls the processing device, The robot arm is controlled to move the first and second imaging devices, and the control device includes a calculation unit that generates the control signal, and an output that outputs the control signal generated by the calculation unit.
  • the calculation unit is configured to calculate the first or second object based on the first image data generated by the first imaging device imaging the first and second objects.
  • the processing device determines whether or not to select the object as the target object to be processed, and if the first or second object is selected as the target object as a result of the determination, the second imaging device and three-dimensional position data generated from the second image data generated by imaging the second object and indicating the three-dimensional positions of each of the plurality of points on the object, and the first image data.
  • a control system including the control device provided by any one of the first to third aspects, the first imaging device, and the second imaging device. be done.
  • control device provided by any one of the first to third aspects, the first imaging device, the second imaging device, and the robot arm.
  • a robot system is provided.
  • the robot arm is provided with a processing device that processes an object, a first imaging device that outputs the first image data, and a second imaging device that outputs the second image data.
  • a control method that generates a control signal for controlling the object, the first image data indicating the position and orientation of the object using the first image data generated by the first imaging device imaging the object. 1 information, and three-dimensional position data generated from the second image data generated by imaging the object by the second imaging device and indicating three-dimensional positions of each of a plurality of points on the object.
  • a control method is provided in which the robot arm is controlled to move.
  • the robot arm is provided with a processing device that processes an object, a first imaging device that outputs first image data, and a second imaging device that outputs second image data.
  • a control method for generating a control signal for controlling the object comprising: two pieces of first image data each generated by the first imaging device capturing the object at first and second times different from each other; Based on three-dimensional position data generated from the second image data generated by the second imaging device imaging the object at a third time and indicating three-dimensional positions of each of the plurality of points on the object, Processing the object with the processing device, including calculating an amount of change in the position and orientation of the object between the first and second times, and generating the control signal based on the amount of change. Therefore, a control method is provided in which the robot arm is controlled to move the processing device and the first and second imaging devices based on the generated control signal.
  • a processing device that performs processing on at least one of a first object and a second object in which at least one of a position and an orientation differs from the first object; and a processing device that outputs first image data.
  • a controlled method is provided.
  • a computer program that causes a computer to execute the control method provided by any one of the sixth to eighth aspects.
  • FIG. 1 is a block diagram showing the configuration of the robot system of this embodiment.
  • FIG. 2 is a side view showing the external appearance of the robot of this embodiment.
  • FIG. 3 is a block diagram showing the configuration of the control device of this embodiment.
  • FIG. 4 is a flowchart showing the flow of robot control processing.
  • FIGS. 5(a) to 5(d) shows a point in time during a period during which a holding process is being performed to hold a workpiece placed on a placing device moving on a support surface.
  • FIG. 3 is a side view showing the positional relationship between the robot and the workpiece in FIG.
  • FIGS. 5(a) to 5(d) shows a point in time during a period during which a holding process is being performed to hold a workpiece placed on a placing device moving on a support surface.
  • FIG. 3 is a side view showing the positional relationship between the robot and the workpiece in FIG.
  • FIGS. 5(a) to 5(d) shows a point in time
  • FIGS. 7(a) to 7(b) shows a point in time during a period during which a holding process is being performed to hold a workpiece placed on a placing device that is stationary on a support surface.
  • 7(c) to 7(e) are side views showing the positional relationship between the robot and the workpiece in FIG.
  • FIG. 3 is a side view showing the positional relationship between the robot and the workpiece at a certain point in time during processing.
  • FIG. 8(a) to 8(e) shows a holding process for sequentially holding multiple workpieces placed on a mounting device one by one, and a holding process for sequentially holding multiple workpieces one by one.
  • FIG. 3 is a side view showing the positional relationship between the robot and the workpiece at a certain point in time during a period in which a release process for placing the workpiece in the apparatus is being performed.
  • FIGS. 9(a) to 9(e) shows a holding process for sequentially holding multiple workpieces placed on a mounting device one by one, and a holding process for sequentially holding multiple workpieces one by one on a mounting device.
  • FIG. 3 is a side view showing the positional relationship between the robot and the workpiece at a certain point in time during a period in which a release process for placing the workpiece in the apparatus is being performed.
  • FIG. 7 is a block diagram showing logical processing blocks for calculating the position and orientation of an object by performing matching processing and tracking processing.
  • FIG. 11 schematically shows the 2D matching process.
  • FIG. 12 is a timing chart showing the relationship between the timing at which the imaging device generates image data and the timing at which the 2D matching section performs 2D matching processing.
  • FIG. 13 schematically shows the 3D matching process.
  • FIG. 14 shows the initial position and initial orientation of the three-dimensional model.
  • FIG. 15 is a timing chart showing the relationship between the timing at which the imaging device generates image data and the timing at which the 3D matching section performs 3D matching processing.
  • FIG. 16 schematically shows 3D matching target data that is part of the 3D position data.
  • FIG. 17 schematically shows the tracking process.
  • FIG. 18 is a timing chart showing the relationship between the timing at which the imaging device generates image data and the timing at which the tracking unit performs tracking processing.
  • FIG. 19 is a timing chart showing the relationship between the timing at which the 2D matching section performs 2D matching processing, the timing at which the 3D matching section performs 3D matching processing, and the timing at which the tracking section performs tracking processing.
  • FIG. 20 is a block diagram showing the configuration of a control device in the first modification.
  • FIG. 20 is a block diagram showing the configuration of a control device in the first modification.
  • FIG. 21 is a flowchart showing the flow of robot control processing in the first modification.
  • FIG. 22 is a plan view showing two objects detected by 2D matching processing.
  • FIG. 23(a) shows the imaging device in rotational movement
  • FIG. 23(b) shows the imaging device in translational movement.
  • FIG. 24(a) shows a robot arm moving an end effector toward a first object
  • FIG. 24(b) shows a robot arm moving an end effector toward a second object.
  • FIG. 25 is a perspective view showing an imaging device that images an object from a first position and an imaging device that images an object from a second position different from the first position.
  • FIG. 26 is a block diagram showing the configuration of a control device in a second modification.
  • FIG. 27 schematically shows the plane removal process.
  • FIG. 28 is a block diagram showing the configuration of the imaging unit.
  • FIG. 29 schematically shows high dynamic range synthesis processing.
  • FIG. 30 schematically shows the noise removal process.
  • FIG. 31 is a
  • FIG. 1 is a block diagram showing the overall configuration of the robot system SYS.
  • the robot system SYS includes a robot 1, an imaging unit 2, a control device 3, and an end effector 13.
  • the robot 1 is a device that can perform predetermined processing on the object OBJ.
  • An example of the robot 1 is shown in FIG.
  • FIG. 2 is a side view showing the external appearance of the robot 1.
  • the robot 1 includes, for example, a base 11, a robot arm 12, and a robot control device 14.
  • the base 11 is a member that serves as the foundation of the robot 1.
  • the base 11 is placed on a support surface S such as a floor surface.
  • the base 11 may be fixed to the support surface S.
  • the base 11 may be movable with respect to the support surface S.
  • the base 11 may be self-propelled on the support surface S.
  • the base 11 may be installed in an automatic guided vehicle (AGV).
  • AGV automatic guided vehicle
  • FIG. 2 shows an example in which the base 11 is fixed to the support surface S.
  • the robot arm 12 is attached to the base 11.
  • the robot arm 12 is a device in which a plurality of links 121 are connected via joints 122.
  • the joint 122 has a built-in actuator.
  • the link 121 may be rotatable around an axis defined by the joint 122 by an actuator built into the joint 122.
  • at least one link 121 may be expandable and contractible along the direction in which the link 121 extends.
  • a device including a device in which a plurality of links 121 are connected via joints 122 and the base 11 may be referred to as a robot arm 12.
  • An end effector 13 is attached to the robot arm 12. That is, the end effector 13 is attached to the robot 1.
  • the end effector 13 is attached to the tip of the robot arm 12.
  • End effector 13 is movable by movement of robot arm 12. That is, the robot arm 12 moves the end effector 13. That is, the robot 1 moves the end effector 13.
  • the end effector 13 is a device that performs a predetermined process (in other words, a predetermined operation) on the object OBJ.
  • the end effector 13 may be called a processing device because it performs predetermined processing on the object OBJ.
  • the end effector 13 may perform a holding process (in other words, a holding operation) for holding the object OBJ, as an example of the predetermined process.
  • the end effector 13 may be considered to be performing a holding process on the object OBJ that it should hold.
  • the end effector 13 capable of performing a holding process may be referred to as a holding device.
  • the end effector 13 may perform a release process (in other words, a release operation) to release (that is, release) the held object OBJ.
  • the end effector 13 may be considered to be performing a release process on the object OBJ held by the end effector 13.
  • the holding device may perform a release process on the object OBJ held by the holding device.
  • the end effector 13 capable of performing a release process may be referred to as a release device.
  • the release process includes a release process of releasing (that is, releasing) the first object OBJ held by the end effector 13 so that the first object OBJ is placed on the second object OBJ.
  • the end effector 13 may be considered to be performing a release process on the first object OBJ held by the end effector 13.
  • the end effector 13 may be considered to be performing release processing on the first object OBJ that the end effector 13 should release.
  • the end effector 13 may be considered to be performing a release process on the second object OBJ on which the first object OBJ to be released by the end effector 13 is placed.
  • a hand gripper is an example of the end effector 13 that can perform holding processing and releasing processing.
  • the hand gripper is an end effector 13 that can hold the object OBJ by physically pinching the object OBJ using a plurality of (for example, two, three, or four) finger members or claw members. be.
  • Another example of the end effector 13 that can perform the holding process is at least one vacuum gripper that can hold the object OBJ by vacuum suctioning the object OBJ.
  • FIG. 2 shows an example in which the end effector 13 is a hand gripper.
  • the robot 1 may perform placement processing (in other words, placement operation) to place the object OBJ at a desired position using the end effector 13 that is capable of performing holding processing and release processing. For example, the robot 1 uses the end effector 13 to hold a first object OBJ, and then transfers the first object OBJ held by the end effector 13 to a second object OBJ different from the first object OBJ. You may also perform placement processing to place it at a desired position.
  • placement processing in other words, placement operation
  • the robot 1 uses an end effector 13 capable of performing a holding process and a release process to perform a fitting process (in other words, a fitting process) for fitting the first object OBJ into a second object OBJ that is different from the first object OBJ.
  • a fitting operation may be performed.
  • the robot 1 uses the end effector 13 to hold a first object OBJ, and then transfers the first object OBJ held by the end effector 13 to a second object OBJ different from the first object OBJ.
  • An inset process may be performed to fit it in.
  • the object OBJ on which the end effector 13 performs predetermined processing may include a workpiece W, as shown in FIG. 2.
  • the work W may include, for example, parts or members used to manufacture a desired product.
  • the work W may include, for example, parts or members that are processed to manufacture a desired product.
  • the work W may include, for example, parts or members that are transported to manufacture a desired product.
  • the object OBJ on which the end effector 13 performs predetermined processing may include a mounting device T on which a workpiece W is mounted, as shown in FIG.
  • An example of the mounting device T is a pallet.
  • the mounting device T may be arranged on the support surface S.
  • the mounting device T may be fixed to the support surface S.
  • the mounting device T may be movable with respect to the support surface S.
  • the mounting device T may be able to move on the support surface S by itself.
  • FIG. 2 shows an example in which the mounting device T can move on the support surface S by itself.
  • the mounting device T may be called an automatic guided vehicle (AGV).
  • AGV automatic guided vehicle
  • the movable mounting device T may be able to function as a moving device that moves the workpiece W.
  • a belt conveyor may be used as the mounting device T.
  • the object OBJ does not need to include the mounting device T.
  • the self-propelled device capable of self-propelled on the support surface S and the pallet may be separate bodies.
  • the pallet may be installed on a self-propelled device.
  • a device including both a self-propelled device and a pallet may be referred to as a mounting device T.
  • the object OBJ may not include at least one of the self-propelled device and the pallet.
  • a self-propelled device may also be referred to as an automated guided vehicle.
  • the mounting device T may be able to fly on the support surface S.
  • the mounting device T may be referred to as an unmanned aerial vehicle.
  • the flight device capable of flying over the support surface S and the pallet may be separate bodies.
  • the pallet may be installed on the flight device.
  • a device including a flying device and a pallet may be referred to as a mounting device T.
  • a flying device may also be referred to as an unmanned aerial vehicle.
  • the workpiece W does not need to be placed on the placement device T.
  • the workpiece W may be placed on the support surface S.
  • the above-mentioned holding process includes processing for holding the workpiece W placed on a stationary or moving mounting device T.
  • the above-described holding process may include a process of holding the workpiece W placed on the support surface S.
  • the above-mentioned release process is performed in order to place the workpiece W held by the end effector 13 at a desired position on the stationary or moving mounting device T. It may also include processing to release.
  • the release process described above may include a process of releasing the workpiece W held by the end effector 13 in order to place the workpiece W held by the end effector 13 at a desired position on the support surface S. .
  • the above-described release process is performed in order to fit the first workpiece W held by the end effector 13 into the second workpiece W placed on the stationary or moving mounting device T.
  • the process may include a process of releasing the first workpiece W held by the end effector 13.
  • the above-described release process is performed in order to fit the first workpiece W held by the end effector 13 into the second workpiece W placed on the support surface S.
  • the process may include a process of releasing the work W of .
  • the robot control device 14 controls the operation of the robot 1.
  • the robot control device 14 may control the operation of the robot arm 12.
  • the robot controller 14 may control the operation of the robot arm 12 such that a desired link 121 rotates around an axis defined by a desired joint 122.
  • the robot control device 14 may control the operation of the robot arm 12 so that the end effector 13 attached to the robot arm 12 is positioned at a desired position.
  • the robot control device 14 may control the operation of the robot arm 12 so that the end effector 13 attached to the robot arm 12 moves to a desired position.
  • the robot control device 14 may also control the operation of the end effector 13 attached to the robot 1.
  • the robot control device 14 may control the operation of the end effector 13 so that the end effector 13 holds the object OBJ at a desired timing. That is, the robot control device 14 may control the operation of the end effector 13 so that the end effector 13 performs the holding process at a desired timing.
  • the robot control device 14 may control the operation of the end effector 13 so that the end effector 13 releases the object OBJ held at a desired timing. That is, the robot control device 14 may control the operation of the end effector 13 so that the end effector 13 performs the release process at a desired timing.
  • the robot control device 14 may control the timing at which the hand gripper opens and closes.
  • the robot control device 14 may control the timing of turning on and off the vacuum device of the vacuum gripper.
  • FIG. 2 shows an example in which the robot 1 is a robot arm 12 (that is, a vertically articulated robot).
  • the robot 1 may be a robot different from the vertically articulated robot.
  • the robot 1 may be a SCARA robot (that is, a horizontal articulated robot).
  • the robot 1 may be a parallel link robot.
  • the robot 1 may be a dual-arm robot including two robot arms 12.
  • the robot 1 may be a Cartesian coordinate robot.
  • the robot 1 may be a cylindrical coordinate robot.
  • the robot 1 may also be referred to as a mobile device.
  • the mobile device may include at least one of an automatic guided vehicle and an unmanned aerial vehicle.
  • the robot 1 may be installed in at least one of an automatic guided vehicle and an unmanned aircraft.
  • the imaging unit 2 images the object OBJ.
  • the imaging unit 2 includes an imaging device 21, an imaging device 22, and a projection device 23. Note that the imaging unit 2 may be referred to as an imaging section.
  • the imaging device 21 is a camera that can image the object OBJ.
  • the imaging device 21 generates image data IMG_2D by imaging the object OBJ.
  • Image data IMG_2D generated by the imaging device 21 is output from the imaging device 21 to the control device 3.
  • the imaging device 21 is a monocular camera.
  • the imaging device 21 is capable of imaging the object OBJ using a monocular camera (in other words, an imaging device).
  • the imaging device 21 is not limited to a monocular camera.
  • the imaging device 21 may be a stereo camera that can image the object OBJ using two monocular cameras, or may include three or more monocular cameras.
  • the imaging device 21 may be at least one of a light field camera, a plenoptic camera, and a multispectral camera.
  • the imaging device 22 is a camera that can image the object OBJ.
  • the imaging device 22 is a stereo camera.
  • the imaging device 22 is a stereo camera that can image the object OBJ using two monocular cameras (in other words, two imaging elements).
  • the imaging device 22 generates image data IMG_3D by imaging the object OBJ.
  • the imaging device 22 since the imaging device 22 is a stereo camera, the imaging device 22 generates image data IMG_3D including two pieces of image data respectively generated by two monocular cameras.
  • Image data IMG_3D generated by the imaging device 22 is output from the imaging device 22 to the control device 3.
  • the imaging device 22 is not limited to a stereo camera.
  • the imaging device 22 may be a monocular camera or may include three or more monocular cameras.
  • the imaging device 22 may be at least one of a light field camera, a plenoptic camera, and a multispectral camera.
  • the projection device 23 is a device that can irradiate projection light onto the object OBJ.
  • the projection device 23 is a device that can project a desired projection pattern onto the object OBJ by irradiating the object OBJ with projection light.
  • the desired projection pattern may include, for example, a random pattern.
  • the random pattern may be a projection pattern having a different pattern for each unit irradiation area.
  • the random pattern may include a random dot pattern.
  • the desired projection pattern may include, for example, a one-dimensional or two-dimensional grid pattern.
  • the desired projection pattern may include other projection patterns.
  • the imaging device 22 images the object OBJ onto which the projection pattern from the projection device 23 is projected.
  • the object OBJ on which the projection pattern is projected is reflected in the image indicated by the image data IMG_3D.
  • the imaging device 21 does not need to image the object OBJ onto which the projection pattern is projected.
  • the imaging device 21 may image an object OBJ on which no projection pattern is projected.
  • the object OBJ on which the projection pattern is projected does not need to be reflected in the image indicated by the image data IMG_2D.
  • the image indicated by the image data IMG_2D may include an object OBJ on which no projection pattern is projected.
  • the projection light for projecting a desired projection pattern onto the object OBJ may be referred to as pattern light or structured light.
  • the projection light may include pattern light or structured light.
  • the projection device 23 may consider that by irradiating the object OBJ with the projection light, the projection device 23 illuminates the object OBJ with the projection light. In this case, the projection device 23 may be able to function as a lighting device that illuminates the object OBJ.
  • the projection light may be referred to as illumination light.
  • the projection light does not need to be light that can project a desired projection pattern onto the object OBJ.
  • the imaging unit 2 like the end effector 13, is provided on the robot arm 12. That is, the imaging devices 21 and 22 and the projection device 23 are attached to the robot arm 12.
  • the imaging devices 21 and 22 and the projection device 23 may be attached to the tip of the robot arm 12 similarly to the end effector 13.
  • the imaging devices 21 and 22 and the projection device 23 are movable by the movement of the robot arm 12. That is, the robot arm 12 moves the imaging devices 21 and 22 and the projection device 23.
  • the imaging unit 2 does not need to be provided on the robot arm 12.
  • the imaging unit 2 may be provided at any position where it can irradiate the object OBJ with projection light and image the object OBJ.
  • at least one of the imaging device 21 , the imaging device 22 , and the projection device 23 is provided on the robot arm 12
  • at least one other of the imaging device 21 , the imaging device 22 , and the projection device 23 is provided on the robot arm 12 . They may be provided at different locations.
  • at least one of the imaging device 21 and the imaging device 22 is provided at a location different from the robot arm 12, at least one of the imaging device 21 and the imaging device 22 is installed on a pillar or the like arranged so as to be able to image the object OBJ.
  • the projection device 23 may be installed in a structure. Further, when the projection device 23 is installed at a different location from the robot arm 12, the projection device 23 may be installed on a structure such as a pillar arranged so as to be able to irradiate the object OBJ with projection light. .
  • the imaging devices 21 and 22 may image the object OBJ in synchronization with each other.
  • the imaging devices 21 and 22 may simultaneously image the object OBJ.
  • the imaging devices 21 and 22 image the object OBJ so that the 2D imaging time when the imaging device 21 images the object OBJ and the 3D imaging time when the imaging device 22 images the object OBJ are the same time.
  • the imaging devices 21 and 22 have a 2D imaging time at which the imaging device 21 images the object OBJ to generate image data IMG_2D, and a 3D imaging time at which the imaging device 22 images the object OBJ to generate the image data IMG_3D.
  • the object OBJ may be imaged so that the times are the same.
  • the imaging devices 21 and 22 may image the object OBJ under the control of the control device 3.
  • the time (in other words, timing) at which each of the imaging devices 21 and 22 images the object OBJ may be controlled by the control device 3.
  • the control device 3 may control the imaging devices 21 and 22 so that the imaging devices 21 and 22 image the object OBJ in synchronization with each other.
  • the control device 3 may control the imaging devices 21 and 22 so that the imaging devices 21 and 22 simultaneously capture images of the object OBJ. That is, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same time.
  • the state that "the 2D imaging time and the 3D imaging time are the same time” may include the state that "the 2D imaging time and the 3D imaging time are literally the same time”.
  • the state where "2D imaging time and 3D imaging time are the same” means "2D imaging time and 3D imaging time are not exactly the same time, but the time difference between 2D imaging time and 3D imaging time is It may also include a state in which the 2D imaging time and the 3D imaging time can be considered to be substantially the same time because the deviation is smaller than the allowable upper limit.
  • the allowable upper limit value may be an allowable upper limit value based on a control error of the robot arm 12.
  • an error occurs in the calculation result of at least one of the position and orientation of the object OBJ, which will be described later, due to a time difference between the 2D imaging time and the 3D imaging time (in other words, at least one of the calculated position and orientation of the object OBJ is There are cases where the accuracy of one side decreases).
  • an error in the calculation of at least one of the position and orientation of the object OBJ may cause a control error in the robot arm 12.
  • a control error of the robot arm 12 becomes a movement error of the end effector 13, and the end effector 13 may not be able to appropriately perform a predetermined process on the object OBJ.
  • the allowable upper limit value may be considered to be equivalent to the allowable upper limit value of the movement error of the end effector 13 by the robot arm 12. For example, even if there is a time difference between the 2D imaging time and the 3D imaging time due to a synchronization error in the imaging processing of the imaging devices 21 and 22, the 2D imaging time and the 3D imaging time are substantially the same time. It may be considered that Note that the synchronization error in the imaging processing of the imaging devices 21 and 22 may be a synchronization control error in the imaging processing of the imaging devices 21 and 22 by the control device 3.
  • the imaging devices 21 and 22 do not need to simultaneously image the object OBJ.
  • the imaging devices 21 and 22 are configured to image the object OBJ so that the 2D imaging time when the imaging device 21 images the object OBJ and the 3D imaging time when the imaging device 22 images the object OBJ are different times. good.
  • the state where "2D imaging time and 3D imaging time are different times" means “2D imaging time and 3D imaging time are different because the time difference between 2D imaging time and 3D imaging time is larger than the allowable upper limit value.” may include a state in which the two cannot be considered to be at substantially the same time.
  • the imaging devices 21 and 22 when the imaging devices 21 and 22 image the object OBJ during a period in which each of the imaging devices 21 and 22 and the object OBJ are relatively displaced, the imaging devices 21 and 22 The object OBJ may be imaged so that the imaging time and the 3D imaging time are the same time. That is, when the imaging devices 21 and 22 image the object OBJ during a period in which each of the imaging devices 21 and 22 and the object OBJ are relatively displaced, the control device 3 controls the 2D imaging time and the 3D imaging time. The imaging devices 21 and 22 may be controlled so that the times are the same. The reason for this will be explained again when explaining the effects of the robot system SYS.
  • each of the imaging devices 21 and 22 and the object OBJ are relatively displaced means a state in which the relative positional relationship between each of the imaging devices 21 and 22 and the object OBJ is changing. You can leave it there.
  • the state in which each of the imaging devices 21 and 22 and the object OBJ are relatively displaced may mean the state in which each of the imaging devices 21 and 22 and the object OBJ are moving relative to each other.
  • the imaging devices 21 and 22 capture the 2D imaging time. It is not necessary to image the object OBJ so that the 3D imaging time and the 3D imaging time are the same time. That is, the control device 3 does not have to control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same time.
  • the imaging devices 21 and 22 may image the object OBJ such that the 2D imaging time and the 3D imaging time are different times. That is, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are different times.
  • each of the imaging devices 21 and 22 and the object OBJ are not relatively displaced includes the state where the relative positional relationship between each of the imaging devices 21 and 22 and the object OBJ is not changed. You can stay there.
  • the state in which each of the imaging devices 21 and 22 and the object OBJ are not relatively displaced may include the state in which each of the imaging devices 21 and 22 and the object OBJ are not moved relative to each other.
  • the state in which each of the imaging devices 21 and 22 and the object OBJ are not relatively displaced may include a state in which each of the imaging devices 21 and 22 and the object OBJ are stationary.
  • the imaging devices 21 and 22 when the imaging devices 21 and 22 image the object OBJ during a period in which each of the imaging devices 21 and 22 and the object OBJ are not relatively displaced, the imaging devices 21 and 22 change the 2D imaging time and the object OBJ.
  • the object OBJ may be imaged so that the 3D imaging time is the same time. That is, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same time.
  • the control device 3 performs robot control processing.
  • the robot control process includes a process of generating a robot control signal for controlling the robot 1.
  • the control device 3 generates a robot control signal based on the image data IMG_2D and IMG_3D output from the imaging unit 2.
  • the control device 3 calculates at least one of the position and orientation of the object OBJ in the global coordinate system of the robot system SYS based on the image data IMG_2D and IMG_3D, and calculates the calculated position and orientation of the object OBJ in the global coordinate system of the robot system SYS.
  • a robot control signal is generated based on at least one of the following.
  • the global coordinate system is a coordinate system that serves as a reference for the robot system SYS.
  • the global coordinate system may be a coordinate system that serves as a reference for the robot 1. Note that when the global coordinate system is a coordinate system that serves as a reference for the robot 1, the global coordinate system may be referred to as a robot coordinate system.
  • control device 3 may perform end effector processing.
  • the end effector control process may include a process of generating an end effector control signal for controlling the end effector 13. Specifically, the control device 3 may generate the end effector control signal based on at least one of the calculated position and orientation of the object OBJ.
  • the end effector process may or may not be included in the robot control process. That is, the end effector control signal generated by the control device 3 may or may not be included in the robot control signal.
  • robot control processing may mean processing for generating at least one of a robot control signal and an end effector control signal.
  • the robot control signal may mean at least one of a signal for controlling the robot 1 and a signal for controlling the end effector 13.
  • robot control signal may also be simply referred to as a control signal.
  • control device 3 uses image data IMG_2D and IMG_3D in a coordinate system different from the global coordinate system of the robot system SYS (for example, in a 2D imaging coordinate system, a 3D imaging coordinate system, or a common coordinate system, which will be described later).
  • the robot control signal may be generated based on at least one of the calculated position and orientation of the object OBJ.
  • control device 3 and the imaging unit 2 are used to control the robot 1. Therefore, the system including the control device 3 and the imaging unit 2 may be referred to as a robot control system or a control system.
  • the robot control signal generated by the control device 3 is output to the robot control device 14 of the robot 1.
  • the robot control device 14 controls the operation of the robot 1 based on the robot control signal generated by the control device 3. Therefore, the robot control signal may include a signal for controlling the operation of the robot 1.
  • the robot control signal may include a signal for controlling the robot arm 12. That is, the robot arm 12 may be controlled by the robot control signal.
  • the robot control device 14 may control the operation of the robot arm 12 based on the robot control signal.
  • the robot control device 14 may control the operation of the robot arm 12 by controlling the drive of an actuator built into the joint 122 based on the robot control signal.
  • the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 13 is positioned at a desired position.
  • the robot control signals may include signals for controlling the robot arm 12 so that the end effector 13 moves to a desired position.
  • the robot control signal may include a signal for controlling the robot arm 12 so that the positional relationship between the end effector 13 and the object OBJ becomes a desired positional relationship.
  • the robot control device 14 may control the robot arm 12 based on the robot control signal so that the end effector 13 is located at a desired position.
  • the robot control device 14 may control the robot arm 12 so that the end effector 13 moves to a desired position based on the robot control signal.
  • the robot control device 14 may control the robot arm 12 based on the robot control signal so that the positional relationship between the end effector 13 and the object OBJ becomes a desired positional relationship.
  • the robot control signal causes the end effector 13 to move toward a holding position where the end effector 13 can hold the object OBJ ( In other words, it may include a signal for controlling the robot arm 12 to move closer. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 13 is located at the holding position.
  • the robot control device 14 may control the robot arm 12 so that the end effector 13 moves toward (that is, approaches) the holding position based on the robot control signal. That is, the robot control signal may control the robot arm 12 so that the end effector 13 is located in the holding position.
  • the robot control signal when performing a holding process to release the object OBJ held by the end effector 13, causes the end effector 13 to release the object OBJ held by the end effector 13. It may also include signals for controlling the robot arm 12 to move toward (ie, approach) the desired release position. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 13 is located at the release position. In this case, the robot control device 14 may control the robot arm 12 so that the end effector 13 moves toward (that is, approaches) the release position based on the robot control signal. That is, the robot control signal may control the robot arm 12 so that the end effector 13 is located at the release position.
  • the robot control signal may include a signal for controlling the end effector 13 (that is, an end effector control signal) in addition to or instead of a signal for controlling the robot arm 12. That is, the end effector 13 may be controlled by the robot control signal.
  • the robot control device 14 may control the operation of the end effector 13 based on the robot control signal.
  • the robot control device 14 may control the operation of the end effector 13 by controlling the drive of an actuator that moves a hand gripper that constitutes the end effector 13 based on the robot control signal.
  • the robot control device 14 may control the operation of the end effector 13 by controlling the drive of a vacuum device of a vacuum gripper that constitutes the end effector 13 based on the robot control signal.
  • the robot control signal controls the end effector 13 so that the end effector 13 located at the above-mentioned holding position holds the object OBJ. It may also include a signal to do so. In this case, the robot control device 14 may control the end effector 13 based on the robot control signal so that the end effector 13 located at the holding position holds the object OBJ.
  • the robot control signal The end effector 13 may include a signal for controlling the end effector 13 to release the end effector 13 .
  • the robot control device 14 may control the end effector 13 to release the object OBJ held by the end effector 13 located at the release position based on the robot control signal.
  • the robot control signal may include a signal that can be used as is by the robot control device 14 to control the operation of the robot 1.
  • the robot control signal may include a signal that can be used as is as a robot drive signal used by the robot control device 14 to control the operation of the robot 1.
  • the robot control device 14 may control the operation of the robot 1 using the robot control signal as it is.
  • the control device 3 generates a drive signal for an actuator built in the joint 122 of the robot arm 12 as a robot control signal, and the robot control device 14 uses the robot control signal generated by the control device 3 as it is to control the robot.
  • An actuator built into the joint 122 of the arm 12 may be controlled.
  • the robot control signals may include signals that can be directly used by the robot controller 14 to control the operation of the end effector 13.
  • the robot control signal may include a signal that can be used as is as an end effector drive signal used by the robot control device 14 to control the operation of the end effector 13.
  • the control device 3 generates a drive signal (end effector drive signal) for an actuator that moves the hand gripper constituting the end effector 13 as a robot control signal, and the robot control device 14
  • the actuator of the end effector 13 may be controlled using the robot control signal as is.
  • control device 3 generates a drive signal (end effector drive signal) for driving the vacuum device of the vacuum gripper that constitutes the end effector 13 as a robot control signal
  • robot control device 14 generates a
  • the vacuum device of the end effector 13 may be controlled using the robot control signal as it is.
  • the robot control signal is a signal that can be used as is by the robot controller 14 to control the operation of the robot 1, and a signal that can be used as is by the robot controller 14 to control the operation of the end effector 13. If the robot 1 includes a possible signal, the robot 1 does not need to be equipped with the robot control device 14.
  • the robot control signal may include a signal that can be used by the robot control device 14 to generate a robot drive signal for controlling the operation of the robot 1.
  • the robot control device 14 generates a robot drive signal for controlling the motion of the robot 1 based on the robot control signal, and controls the motion of the robot 1 based on the generated robot drive signal.
  • the robot control device 14 generates a drive signal for the actuator built in the joint 122 of the robot arm 12 based on the robot control signal, and based on the generated signal, the drive signal for the actuator built in the joint 122 of the robot arm 12 is generated. Actuators may also be controlled.
  • the robot control signal may include a signal that can be used by the robot controller 14 to generate an end effector drive signal for controlling the operation of the end effector 13.
  • the robot control device 14 generates an end effector drive signal for controlling the operation of the end effector 13 based on the end effector control signal, and operates the end effector 13 based on the generated end effector drive signal. may be controlled.
  • the robot control device 14 may generate a drive signal for the actuator or vacuum device of the end effector 13 based on the robot control signal, and control the actuator or vacuum device of the end effector 13 based on the generated signal. good.
  • the signal that can be used by the robot control device 14 to generate the robot drive signal may be a signal that represents the position of the object OBJ in the global coordinate system.
  • the robot control device 14 controls the robot 1 (end effector A drive signal is generated for the actuator built in the joint 122 of the robot arm 12 so that the positional relationship between the robot arm 13) and the object OBJ becomes a desired positional relationship, and the robot arm 12 is operated based on the generated drive signal. may be controlled.
  • the signal that can be used by the robot control device 14 to generate the robot drive signal may be a signal that represents the desired positional relationship between the robot 1 and the object OBJ in the global coordinate system.
  • the robot control device 14 controls the robot based on the robot control signal so that the positional relationship between the robot 1 (end effector 13) and the object OBJ becomes the desired positional relationship expressed by the robot control signal.
  • a drive signal for an actuator built into the joint 122 of the arm 12 may be generated, and the operation of the robot arm 12 may be controlled based on the generated drive signal.
  • the signals that can be used by robot controller 14 to generate robot drive signals may include signals representing the desired position of end effector 13 in the global coordinate system.
  • the robot control device 14 controls the robot 1 (end effector 13) and the object OBJ based on the robot control signal so that the end effector 13 is located at the desired position indicated by the robot control signal.
  • a drive signal is generated for the actuator built in the joint 122 of the robot arm 12 so that the positional relationship between the good.
  • the robot control device 14 causes the end effector 13 to move to a holding position where the end effector 13 can hold the object OBJ located at the position indicated by the robot control signal, based on the robot control signal.
  • a drive signal for an actuator built into the joint 122 of the robot arm 12 may be generated, and the operation of the robot arm 12 may be controlled based on the generated drive signal.
  • the robot control device 14 causes the end effector 13 holding the object OBJ located at the position indicated by the robot control signal to move away from the position where the object OBJ was placed.
  • a drive signal for an actuator built in the joint 122 of the robot arm 12 may be generated, and the operation of the robot arm 12 may be controlled based on the generated drive signal.
  • the signals that can be used by the robot control device 14 to generate the robot drive signal may include, for example, a signal representing the desired position of the tip of the robot arm 12 in the global coordinate system, or a signal that represents the desired position of the tip of the robot arm 12 in the global coordinate system. It may also include a signal representing the desired position of the imaging unit 2 in the system.
  • the reference coordinate system for the robot control signal may be a coordinate system other than the global coordinate system (for example, within a 2D imaging coordinate system, within a 3D imaging coordinate system, or within a common coordinate system, which will be described later).
  • FIG. 3 is a block diagram showing the configuration of the control device 3. As shown in FIG.
  • the control device 3 includes a calculation device 31, a storage device 32, and a communication device 33. Furthermore, the control device 3 may include an input device 34 and an output device 35. However, the control device 3 does not need to include at least one of the input device 34 and the output device 35.
  • the arithmetic device 31, the storage device 32, the communication device 33, the input device 34, and the output device 35 may be connected via a data bus 36.
  • the arithmetic unit 31 is, for example, one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array). Contains at least one.
  • Arithmetic device 31 reads a computer program.
  • the arithmetic device 31 may read a computer program stored in the storage device 32.
  • the arithmetic device 31 may read a computer program stored in a computer-readable, non-temporary recording medium using a recording medium reading device (not shown) included in the control device 3.
  • the arithmetic device 31 may obtain (that is, download) a computer program from a device (not shown) located outside the control device 3 via the communication device 33 (or other communication device). (or you can load it).
  • the arithmetic device 31 executes the loaded computer program.
  • a logical functional block is realized for executing the processing that the control device 3 should perform (for example, the robot control processing described above). That is, the arithmetic device 31 can function as a controller for realizing a logical functional block for executing the processing that the control device 3 should perform.
  • a calculation model that can be constructed by machine learning may be implemented in the calculation device 31 by the calculation device executing a computer program.
  • An example of a calculation model that can be constructed by machine learning is a calculation model that includes a neural network (so-called artificial intelligence (AI)).
  • learning the computational model may include learning parameters (eg, at least one of weights and biases) of the neural network.
  • the calculation device 31 may use the calculation model to execute robot control processing. That is, the operation of executing the robot control process may include the operation of executing the robot control process using a calculation model.
  • the calculation device 31 may be equipped with a calculation model that has been constructed by offline machine learning using teacher data. Further, the calculation model installed in the calculation device 31 may be updated by online machine learning on the calculation device 31.
  • the calculation device 31 may use a calculation model installed in a device external to the calculation device 31 (that is, a device provided outside the control device 3) in addition to or in place of the calculation model installed in the calculation device 31.
  • the robot control process may be performed using the .
  • recording media for recording computer programs executed by the arithmetic unit 31 include CD-ROM, CD-R, CD-RW, flexible disk, MO, DVD-ROM, DVD-RAM, DVD-R, DVD+R, and DVD.
  • optical disks such as RW, DVD+RW and Blu-ray (registered trademark)
  • magnetic media such as magnetic tape, magneto-optical disks, semiconductor memories such as USB memory, and any other arbitrary medium capable of storing programs is used. It's okay to be hit.
  • the recording medium may include a device capable of recording a computer program (for example, a general-purpose device or a dedicated device in which a computer program is implemented in an executable state in the form of at least one of software and firmware).
  • each process or function included in the computer program may be realized by a logical processing block that is realized in the arithmetic device 31 when the arithmetic device 31 (that is, a computer) executes the computer program, or It may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) included in the arithmetic unit 31, or a logical processing block. and some of the hardware It may also be realized in a mixed format with partial hardware modules that realize the elements.
  • FPGA Field Programmable Gate Array
  • ASIC Application Specific Integrated Circuit
  • FIG. 3 shows an example of logical functional blocks implemented within the arithmetic unit 31 to execute robot control processing.
  • a three-dimensional position data generation section 311, a position/orientation calculation section 312, and a signal generation section 313 are implemented within the arithmetic device 31. Note that the processing performed by each of the three-dimensional position data generation section 311, position/orientation calculation section 312, and signal generation section 313 will be described in detail later with reference to FIG. 4, etc., so a description thereof will be omitted here.
  • the storage device 32 can store desired data.
  • the storage device 32 may temporarily store a computer program executed by the arithmetic device 31.
  • the storage device 32 may temporarily store data that is temporarily used by the arithmetic device 31 when the arithmetic device 31 is executing a computer program.
  • the storage device 32 may store data that the control device 3 stores for a long period of time.
  • the storage device 32 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. good. That is, the storage device 32 may include a non-temporary recording medium.
  • the communication device 33 is capable of communicating with each of the robot 1 and the imaging unit 2 via a communication network (not shown). Alternatively, the communication device 33 can communicate with another device different from the robot 1 and the imaging unit 2, in addition to or instead of at least one of the robot 1 and the imaging unit 2, via a communication network not shown. Good too. In the present embodiment, the communication device 33 may receive (that is, acquire) the image data IMG_2D and IMG_3D from the imaging unit 2. Furthermore, the communication device 33 may transmit (that is, output) a robot control signal to the robot 1. Note that the communication device 33 that outputs the robot control signal to the robot 1 may be referred to as an output section.
  • the input device 34 is a device that accepts input of information to the control device 3 from outside the control device 3.
  • the input device 34 may include an operating device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by a user of the control device 3.
  • the input device 34 may include a recording medium reading device capable of reading information recorded as data on a recording medium that can be externally attached to the control device 3.
  • information can be input as data to the control device 3 from a device external to the control device 3 via the communication device 33.
  • the communication device 33 may function as an input device that receives information input to the control device 3 from outside the control device 3 .
  • the output device 35 is a device that outputs information to the outside of the control device 3.
  • the output device 35 may output the information as an image. That is, the output device 35 may include a display device (so-called display) capable of displaying an image.
  • the output device 35 may output the information as audio. That is, the output device 35 may include an audio device (so-called speaker) that can output audio.
  • the output device 35 may output information on paper. That is, the output device 35 may include a printing device (so-called printer) that can print desired information on paper.
  • the output device 35 may output information as data to a recording medium that can be externally attached to the control device 3.
  • information can be output as data from the control device 3 to a device external to the control device 3 via the communication device 33.
  • the communication device 33 may function as an output device that outputs information to the outside of the control device 3.
  • FIG. 4 is a flowchart showing the flow of robot control processing.
  • the control device 3 uses the communication device 33 to acquire image data IMG_2D from the imaging device 21 (step S1). Specifically, the imaging device 21 images the object OBJ at a predetermined 2D imaging rate. For example, the imaging device 21 may image the object OBJ at a 2D imaging rate that images the object OBJ several tens to hundreds of times (as an example, 500 times) per second. As a result, the imaging device 21 generates image data IMG_2D at a cycle according to the predetermined 2D imaging rate. For example, the imaging device 21 may generate several tens to hundreds of pieces of image data IMG_2D (500 pieces as an example) per second. The control device 3 acquires the image data IMG_2D each time the imaging device 21 generates the image data IMG_2D. That is, the control device 3 may acquire several dozen to several hundred (for example, 500) pieces of image data IMG_2D per second.
  • the imaging device 21 may image other objects different from the object OBJ. For example, if both the object OBJ and the other object are included in the imaging range (field of view) of the imaging device 21, the imaging device 21 may image both the object OBJ and the other object. . As a result, the imaging device 21 may generate image data IMG_2D representing an image in which both the object OBJ and another object are reflected.
  • the control device 3 further uses the communication device 33 to obtain image data IMG_3D from the imaging device 22 (step S1). Specifically, the imaging device 22 images the object OBJ at a predetermined 3D imaging rate.
  • the 3D imaging rate is the same as the 2D imaging rate. However, the 3D imaging rate may be different from the 2D imaging rate.
  • the imaging device 22 may image the object OBJ at a 3D imaging rate that images the object OBJ several tens to hundreds of times (as an example, 500 times) per second. As a result, the imaging device 22 generates image data IMG_3D at a cycle according to the predetermined 3D imaging rate.
  • the imaging device 22 may generate several tens to hundreds (500 pieces of image data, for example) of image data IMG_3D per second.
  • the control device 3 acquires the image data IMG_3D each time the imaging device 22 generates the image data IMG_3D. That is, the control device 3 may acquire several dozen to several hundred (500 pieces of image data as an example) image data IMG_3D per second.
  • the imaging device 22 may image other objects different from the object OBJ. For example, if both the object OBJ and the other object are included in the imaging range (field of view) of the imaging device 22, the imaging device 22 may image both the object OBJ and the other object. . As a result, the imaging device 22 may generate image data IMG_3D representing an image in which both the object OBJ and another object are reflected.
  • the three-dimensional position data generation unit 311 Each time the control device 3 acquires image data IMG_3D, the three-dimensional position data generation unit 311 generates three-dimensional position data WSD based on the acquired image data IMG_3D (step S2). Further, the three-dimensional position data generation unit 311 outputs the generated three-dimensional position data WSD to the position and orientation calculation unit 312.
  • the three-dimensional position data WSD is data indicating the three-dimensional position of the object OBJ reflected in the image data IMG_3D.
  • the three-dimensional position data WSD may be data indicating the three-dimensional position of at least a portion of the surface of the object OBJ.
  • the three-dimensional position data WSD is data indicating the three-dimensional position of each of a plurality of points on the object OBJ.
  • the three-dimensional position data WSD may be data indicating the respective three-dimensional positions of multiple points on the surface of the object OBJ.
  • the three-dimensional position data WSD may be data indicating the respective three-dimensional positions of a plurality of points corresponding to a plurality of parts on the surface of the object OBJ.
  • the object OBJ on which the projection pattern is projected is reflected in the image indicated by the image data IMG_3D.
  • the projection pattern reflected in the image indicated by the image data IMG_3D reflects the three-dimensional shape of at least a portion of the surface of the object OBJ on which the projection pattern is projected.
  • the shape of the projection pattern reflected in the image indicated by the image data IMG_3D reflects the three-dimensional shape of at least a portion of the surface of the object OBJ on which the projection pattern is projected.
  • the three-dimensional position data generation unit 311 can calculate the three-dimensional shape of at least a portion of the surface of the object OBJ based on the projection pattern reflected in the image indicated by the image data IMG_3D.
  • the three-dimensional shape of at least a portion of the surface of object OBJ substantially indicates the three-dimensional position of each of a plurality of points on object OBJ. This is because each of the plurality of points of object OBJ is included in the surface of object OBJ. Therefore, the process of calculating the three-dimensional shape of at least a portion of the surface of the object OBJ is considered to be substantially equivalent to the process of calculating the three-dimensional position of each of the plurality of points on the object OBJ. It's okay. Therefore, the three-dimensional position data generation unit 311 can generate the three-dimensional position data WSD based on the image data IMG_3D.
  • the three-dimensional position data WSD indicating the three-dimensional position of each of the plurality of points of the object OBJ is substantially equal to the three-dimensional position data WSD of the plurality of points of the object OBJ.
  • the three-dimensional position data generation unit 311 associates each part (for example, each pixel) of the images indicated by each of the two image data included in the image data IMG_3D. Parallax may also be calculated. Specifically, in this association, the three-dimensional position data generation unit 311 generates each part of the projection pattern reflected in the image indicated by each of the two image data (that is, the projection pattern reflected in each image). The parallax may be calculated by associating each part of the image. The three-dimensional position data generation unit 311 may calculate the three-dimensional position of each of the plurality of points of the object OBJ using a well-known method based on the principle of triangulation using the calculated parallax.
  • three-dimensional position data WSD indicating the three-dimensional positions of each of the plurality of points on the object OBJ is generated.
  • the accuracy of parallax calculation becomes higher. Therefore, the accuracy of the generated three-dimensional position data WSD (that is, the calculation accuracy of the three-dimensional positions of each of the plurality of points on the object OBJ) becomes higher.
  • the three-dimensional position data WSD may be any data as long as it can indicate the three-dimensional position of each of the plurality of points on the object OBJ. That is, the three-dimensional position data WSD may be any data that directly or indirectly indicates the three-dimensional position of each of the plurality of points on the object OBJ.
  • the three-dimensional position data WSD may include coordinate information indicating the three-dimensional position of each of a plurality of points on the object OBJ.
  • the three-dimensional position data WSD may include information indicating the three-dimensional shape of at least a portion of the object OBJ.
  • the three-dimensional position data generation unit 311 also sequentially generates the plurality of three-dimensional position data WSD from the plurality of image data IMG_3D.
  • the three-dimensional position data WSD generated from the image data IMG_3D at one time may indicate the three-dimensional position of each of the plurality of first points of the object OBJ.
  • the three-dimensional position data WSD generated from the image data IMG_3D at another time different from the one time may indicate the three-dimensional position of each of the plurality of second points of the object OBJ.
  • the plurality of first points of the object OBJ may be the same as the plurality of second points of the object OBJ.
  • at least one of the plurality of first points of the object OBJ may be different from at least one of the plurality of second points of the object OBJ.
  • Depth image data is image data in which depth information is associated with each pixel of a depth image indicated by the depth image data in addition to brightness information.
  • the depth information is information indicating the distance (that is, the depth) between each part of the object OBJ reflected in each pixel and the imaging device 22.
  • the depth image data may be image data in which the brightness information of each pixel indicates the depth of each part of the object OBJ (that is, the distance between each part of the object OBJ and the imaging device 22).
  • the three-dimensional position data generation unit 311 determines the distance between each part of the object OBJ reflected in the image indicated by image data IMG_3D and the imaging device 22 based on the projection pattern reflected in the image indicated by image data IMG_3D.
  • a depth image may be generated by calculating a distance and associating each pixel of the image indicated by the image data IMG_3D with the calculated distance as depth information.
  • the point cloud data is data indicating a set in a three-dimensional space of points corresponding to each part of the object OBJ reflected in the image indicated by the image data IMG_3D.
  • the three-dimensional position data generation unit 311 may generate point cloud data based on the depth image data and camera parameters of the imaging device 22.
  • the three-dimensional position data WSD is generated based on the image data IMG_3D, it may be data indicating the three-dimensional position of the object OBJ in a 3D imaging coordinate system, which will be described later.
  • the image data IMG_3D is An image in which both object OBJ and another object are reflected is shown.
  • the three-dimensional position data WSD may be data indicating not only the three-dimensional position of the object OBJ reflected in the image data IMG_3D but also the three-dimensional position of other objects reflected in the image data IMG_3D. good.
  • the three-dimensional position data WSD includes data indicating the three-dimensional position of the object OBJ
  • the three-dimensional position data WSD indicates the three-dimensional position of each of the plurality of points on the object OBJ. It may be considered as data. This is because even if data indicating the three-dimensional position of another object is included in the three-dimensional position data WSD, data indicating the three-dimensional position of object OBJ is included in the three-dimensional position data WSD. This is because there is no difference.
  • the position and orientation calculation unit 312 calculates at least one of the position and orientation of the object OBJ based on the image data IMG_2D acquired in step S1 and the three-dimensional position data WSD generated in step S2 (step S3 ). As a result, the position and orientation calculation unit 312 generates position and orientation information POI0 indicating at least one of the position and orientation of the object OBJ (step S3).
  • the position/orientation calculation unit 312 calculates at least one of the position and orientation of the object OBJ in the global coordinate system. That is, the position/orientation calculation unit 312 generates position/orientation information POI0 indicating at least one of the position and orientation of the object OBJ in the global coordinate system.
  • the global coordinate system is a coordinate system that serves as a reference for the robot system SYS. Specifically, the global coordinate system is a coordinate system used to control the robot 1.
  • the robot control device 14 may control the robot arm 12 so that the end effector 13 is located at a desired position within the global coordinate system.
  • the global coordinate system is a coordinate system defined by an X-axis (GL), a Y-axis (GL), and a Z-axis (GL) that are orthogonal to each other.
  • the X-axis (GL) may be an axis along a horizontal plane.
  • the Y axis (GL) may be an axis along a horizontal plane.
  • the Z-axis (GL) may be an axis perpendicular to the horizontal plane.
  • the Z-axis (GL) may be an axis extending along the direction of gravity. Note that the X-axis, Y-axis, and Z-axis shown in FIG. 2 may be the X-axis (GL), Y-axis (GL), and Z-axis (GL), respectively.
  • the origin of the global coordinate system does not have to be the origin of the X-axis (GL), Y-axis (GL), and Z-axis (GL) shown in FIG. 2.
  • the origin of the global coordinate system may be set at an arbitrary position on the base 11 in FIG. 2.
  • the origin of the global coordinate system may be set at any position on the support surface S.
  • the origin of the global coordinate system may be set at an arbitrary position of the contact surface of the base 11 with respect to the support surface S (for example, the center or center of gravity of the contact surface).
  • the position and orientation calculation unit 312 calculates the position of the object OBJ in the global coordinate system by determining the position Tx (GL) of the object OBJ in the X-axis direction (GL) parallel to the X-axis (GL) and the Y-axis parallel to the Y-axis (GL). At least one of the position Ty (GL) of the object OBJ in the axial direction (GL) and the position Tz (GL) of the object OBJ in the Z-axis direction (GL) parallel to the Z-axis (GL) may be calculated. .
  • the position and orientation calculation unit 312 calculates the rotation amount Rx (GL) of the object OBJ around the X-axis (GL) and the rotation amount Ry (GL) of the object OBJ around the Y-axis (GL) as the orientation of the object OBJ in the global coordinate system. and the amount of rotation Rz(GL) of the object OBJ around the Z-axis (GL). This is because the rotation amount Rx (GL) of the object OBJ around the X-axis (GL), the rotation amount Ry (GL) of the object OBJ around the Y-axis (GL), and the rotation amount Rz of the object OBJ around the Z-axis (GL).
  • GL represents a parameter representing the attitude of the object OBJ around the X-axis (GL), a parameter representing the attitude of the object OBJ around the Y-axis (GL), and an attitude of the object OBJ around the Z-axis (GL), respectively. This is because it is equivalent to the parameter.
  • the rotation amount Rz (GL) of OBJ is determined by the attitude Rx (GL) of the object OBJ around the X-axis (GL), the attitude Ry (GL) of the object OBJ around the Y-axis (GL), and the attitude Ry (GL) of the object OBJ around the Z-axis (GL), respectively.
  • the attitude of the object OBJ is called Rz(GL).
  • the posture Rx (GL) of the object OBJ around the X-axis (GL), the posture Ry (GL) of the object OBJ around the Y-axis (GL), and the posture Rz (GL) of the object OBJ around the Z-axis (GL) are , respectively, indicate the position of the object OBJ in the rotational direction around the X-axis (GL), the position of the object OBJ in the rotational direction around the Y-axis (GL), and the position of the object OBJ in the rotational direction around the Z-axis (GL), respectively. It may be assumed that there is.
  • the posture Rx (GL) of the object OBJ around the X-axis (GL), the posture Ry (GL) of the object OBJ around the Y-axis (GL), and the posture Rz (GL) of the object OBJ around the Z-axis (GL) are , may be regarded as parameters representing the position of the object OBJ.
  • the position and orientation calculation unit 312 calculates the position Tx (GL), the position Ty (GL), and the position Tz (GL) as at least one of the position and orientation of the object OBJ in the global coordinate system. , posture Rx (GL), posture Ry (GL), and posture Rz (GL).
  • the process of calculating at least one of the position and orientation of the object OBJ in the global coordinate system in step S3 in FIG. 4 (that is, the process of generating position and orientation information POI0) will be described in detail later with reference to FIG. 10 etc. Therefore, the explanation here will be omitted.
  • the signal generation unit 313 generates a robot control signal using the position and orientation information POI0 generated in step S3 (step S4).
  • the signal generation unit 313 may generate a robot control signal so that the robot 1 (end effector 13) can perform predetermined processing on the object OBJ.
  • the signal generation unit 313 may generate a robot control signal so that the positional relationship between the robot 1 (end effector 13) and the object OBJ is a desired positional relationship.
  • the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the positional relationship between the robot 1 (end effector 13) and the object OBJ becomes a desired positional relationship.
  • the signal generation unit 313 generates a robot control signal so that the end effector 13 performs a predetermined process on the object OBJ when the positional relationship between the robot 1 and the object OBJ becomes a desired positional relationship.
  • the signal generation unit 313 performs robot control to control the operation of the end effector 13 so as to perform a predetermined process on the object OBJ when the positional relationship between the robot 1 and the object OBJ reaches a desired positional relationship.
  • a signal may also be generated.
  • the robot control signal for controlling the operation of the end effector 13 may be referred to as an end effector control signal.
  • the signal generation unit 313 may generate a robot control signal under feedback control using the position and orientation information POI0 generated in step S3.
  • the signal generation unit 313 may generate a robot control signal under P (Proportional) control using the position and orientation information POI0.
  • the signal generation unit 313 may generate a robot control signal under PI (Proportional-Integral) control using the position and orientation information POI0.
  • the signal generation unit 313 may generate a robot control signal under PID (Proportional-Integral-Differential) control using the position and orientation information POI0.
  • FIGS. 5(a) to 5(d) shows a mounting device T#1 (that is, an example of an object OBJ) such as an AGV (Automatic Guided Vehicle) moving on a support surface S.
  • FIG. 3 is a side view showing the positional relationship between the robot 1 and the workpiece W at a certain point in time during a period in which a holding process for holding the placed workpiece W (that is, another example of object OBJ) is being performed; .
  • the control device 3 generates position and orientation information POI0 indicating at least one of the position and orientation of the workpiece W to be held, and uses the generated position and orientation information POI0 to generate a robot control signal. Good too.
  • FIG. 1 that is, an example of an object OBJ
  • AGV Automatic Guided Vehicle
  • the signal generation unit 313 generates a robot control signal for controlling the operation of the robot arm 12 so that the end effector 13 moves toward the space directly above the moving workpiece W. may be generated. As shown in FIG. 5(b), the signal generation unit 313 enables the end effector 13 located directly above the moving workpiece W to hold the workpiece W while continuing to be located directly above the workpiece W. A robot control signal may be generated to control the operation of the robot arm 12 so as to approach the workpiece W until the distance is reached. As shown in FIG. 5C, the signal generation unit 313 causes the end effector 13 located at a position where it can hold the moving workpiece W to hold the workpiece W while following the moving workpiece W.
  • Robot control signals may be generated to control the operation of the robot arm 12 and end effector 13.
  • the signal generation unit 313 controls the robot arm 12 and the end effector so that the end effector 13 holding the workpiece W is separated from the mounting device T#1 that moves while holding the workpiece W.
  • a robot control signal may be generated to control the operation of 13.
  • the robot 1 uses the end effector 13 to hold the workpiece W placed on the mounting device T#1, the robot 1 places the held workpiece W on a different mounting device than that on the mounting device T#1.
  • a release process may be performed to release the held workpiece W so that it is placed on the device T#2. That is, the robot 1 may perform a placement process (or a fitting process) for placing the workpiece W on the mounting device T#2 by sequentially performing the holding process and the release process.
  • the control device 3 generates position and posture information POI0 indicating at least one of the position and posture of the mounting device T#2 that is the target of the release process, and uses the generated position and posture information POI0 to send robot control signals. may be generated. For example, each of FIGS.
  • FIG. 3 is a side view showing the positional relationship between the robot 1 and the workpiece W at a certain point in time.
  • the signal generation unit 313 causes the end effector 13 holding the work W to move into a space directly above the mounting device T#2 that moves while holding the work W.
  • a robot control signal may be generated to control the operation of the robot arm 12 to move the robot arm 12 to the robot arm 12 . As shown in FIG.
  • the signal generation unit 313 detects that the end effector 13, which is located directly above the moving mounting device T#2 and holds the workpiece W, is directly above the mounting device T#2. In order to control the operation of the robot arm 12 so that it approaches the mounting device T#2 while continuing to be located at and holding the workpiece W until the workpiece W can be placed on the mounting device T#2. robot control signals may be generated. As shown in FIG. 6(c), the signal generation unit 313 causes the end effector 13 located at a position where the workpiece W can be placed on the mounting device T#2 to follow the moving mounting device T#2.
  • the signal generation unit 313 causes the robot arm 12 to move so that the end effector 13, which has finished placing the workpiece W on the mounting device T#2, moves away from the mounting device T#2.
  • a robot control signal may be generated to control the robot.
  • the control device 3 controls the position and position of the mounting device T#2, which is the first target of the release process.
  • position and orientation information POI0 that indicates at least one of the orientations
  • position and orientation information POI0 that represents at least one of the position and orientation of the workpiece W that is the second target of the release process may be further generated.
  • the control device 3 Before the end effector 13 holds the workpiece W, the control device 3 generates position/orientation information POI0 indicating at least one of the position and orientation of the workpiece W not yet held by the end effector 13; After the end effector 13 holds the workpiece W, position and orientation information POI0 indicating at least one of the position and orientation of the workpiece W held by the end effector 13 may be generated.
  • the control device 3 uses the end effector in addition to the position and orientation information POI0 regarding the mounting device T#2.
  • the robot control signal may be generated using position and orientation information POI0 indicating at least one of the position and orientation of the workpiece W held by the robot controller 13. That is, the control device 3 uses the position and orientation information POI0 indicating at least one of the position and orientation of the workpiece W held by the end effector 13 during at least part of the period during which the end effector 13 holds the workpiece W.
  • the robot control signal may be generated using the robot control signal.
  • the signal generation unit 313 generates a robot control signal for controlling the robot arm 12 to move the workpiece W held by the end effector 13 to a desired position (for example, a position where the workpiece W should be released). You may. In this case, the robot 1 can appropriately move the workpiece W held by the end effector 13 to a desired position, compared to the case where the position and orientation information POI0 regarding the workpiece W is not used. This is because the position of the workpiece W held by the end effector 13 becomes known information to the control device 3. Further, for example, the signal generation unit 313 may generate a robot control signal for controlling the end effector 13 so as to change the posture of the workpiece W held by the end effector 13 to a desired posture.
  • a desired position for example, a position where the workpiece W should be released. You may.
  • the robot 1 can appropriately move the workpiece W held by the end effector 13 to a desired position, compared to the case where the position and orientation information POI0 regarding the workpiece W is not used. This is
  • the robot 1 can place or fit the workpiece W in the desired orientation onto the mounting device T using the end effector 13. . This is because at least one of the position and orientation of the work W held by the end effector 13 becomes information known to the control device 3. If the attitude of the workpiece W is not the desired attitude, there is a possibility that the workpiece W will collide with the mounting device T. However, in this embodiment, since the robot 1 can be controlled so that the posture of the work W becomes a desired posture, there is no or low possibility that the work W will collide with the mounting device T.
  • control device 3 controls at least one of the position and orientation of the workpiece W held by the end effector 13.
  • a robot control signal may be generated using position/orientation information POI0 indicating .
  • FIG. 3 is a side view showing the positional relationship between the robot 1 and the workpiece W at a certain point in time during a period of time.
  • the signal generation unit 313 moves the robot arm 12 so that the end effector 13 approaches the workpiece W until it can hold the stationary workpiece W.
  • Robot control signals may be generated to control motion.
  • FIG. 7(b) the signal generation unit 313 controls the robot arm 12 and the end effector so that the end effector 13 located at a position where it can hold the stationary workpiece W holds the workpiece W.
  • a robot control signal may be generated to control the operation of 13.
  • FIGS. 7(c) to 7(e) shows the period during which the release process for placing the workpiece W on the mounting device T#2 which is stationary on the support surface S is being performed.
  • FIG. 3 is a side view showing the positional relationship between the robot 1 and the workpiece W at a certain point in time. In this case, as shown in FIG.
  • the signal generation unit 313 causes the end effector 13 holding the workpiece W to place the workpiece on the stationary mounting device T#2 while holding the workpiece W.
  • a robot control signal may be generated to control the operation of the robot arm 12 so as to approach the mounting device T#2 until the robot arm 12 can be placed.
  • the signal generation unit 313 generates a message that the end effector 13, which is located at a position where the workpiece W can be placed on the mounting device T#2, is placed in the stationary mounting device T#2.
  • a robot control signal may be generated to control the operations of the robot arm 12 and the end effector 13 so as to place the work W on (that is, release the held work W).
  • the signal generation unit 313 causes the robot arm 12 to move so that the end effector 13, which has finished placing the workpiece W on the mounting device T#2, moves away from the mounting device T#2.
  • a robot control signal may be generated to control the robot.
  • a plurality of works W may be placed on the mounting device T#1.
  • a plurality of workpieces W may be placed on the placement device T#1 so that the plurality of workpieces W are lined up on the placement device T#1 with a certain standard.
  • a plurality of workpieces W may be placed on the placement device T#1 so that the plurality of workpieces W are randomly stacked on the placement device T#1.
  • the robot 1 may perform a holding process to selectively hold a desired one of the plurality of works W placed on the mounting device T#1.
  • the robot 1 may perform a holding process to sequentially hold the plurality of works W placed on the mounting device T#1 one by one.
  • the robot 1 may perform a release process to place a plurality of workpieces W on the mounting device T#2.
  • the robot 1 sequentially transfers the plurality of workpieces W placed on the mounting device T#1 one by one to the mounting device T#2 (or further A placement process may also be performed to place the item on a placement device (a placement device).
  • the robot 1 performs a release process to sequentially place the plurality of workpieces W on the mounting device T#2 one by one so that the plurality of workpieces W are lined up on the mounting device T#2 with a certain standard. It's okay.
  • the robot 1 may perform a release process to sequentially place the plurality of works W on the mounting device T#2 one by one so that the plurality of works W are randomly stacked on the mounting device T#2.
  • FIGS. 8(a) to 8(e) and 9(a) to 9(e) shows two workpieces W#1 and W placed on mounting device T#1.
  • FIG. 2 is a side view showing the positional relationship between the robot 1 and the workpiece W at a certain point in time.
  • the signal generation unit 313 determines whether the end effector 13 is capable of handling one of the workpieces W#1 and W#2 (in the example shown in FIG.
  • the workpiece W# 2 A robot control signal may be generated to control the operation of the robot arm 12 so as to approach the workpiece W#2 until it can hold the workpiece W#2.
  • the signal generation unit 313 causes the robot arm 12 and the end effector 13 located at a position where it can hold the workpiece W#2 to A robot control signal for controlling the operation of the end effector 13 may be generated.
  • the signal generation unit 313 causes the end effector 13 holding the workpiece W#2 to place the workpiece W#2 on the mounting device T#2 while holding the workpiece W#2.
  • a robot control signal may be generated to control the operation of the robot arm 12 so as to approach the mounting device T#2 until it is possible to place the robot arm T#2.
  • the signal generation unit 313 causes the end effector 13 located at a position where the workpiece W#2 can be placed on the mounting device T#2 to
  • a robot control signal may be generated to control the operations of the robot arm 12 and the end effector 13 so as to place the work W#2 on (that is, release the held work W#2).
  • the signal generation unit 313 causes the robot to move the end effector 13, which has finished placing the workpiece W#2 onto the mounting device T#2, away from the mounting device T#2.
  • Robot control signals may be generated to control movement of arm 12.
  • the signal generation unit 313 causes the end effector 13 to transmit the remaining one of the workpieces W#1 and W#2 (in the example shown in FIG. 9(a), the workpiece W#1 ) may be generated to control the operation of the robot arm 12 so as to approach the workpiece W#1 until it can hold the workpiece W#1.
  • the signal generation unit 313 causes the robot arm 12 and the end effector 13 located at a position where it can hold the work W#1 to A robot control signal for controlling the operation of the end effector 13 may be generated. Thereafter, as shown in FIG.
  • the signal generation unit 313 causes the end effector 13 holding the workpiece W#1 to transfer the workpiece W#2 to the mounting device T#2 while holding the workpiece W#1.
  • a robot control signal may be generated to control the operation of the robot arm 12 so as to approach the placement device T#2 until it is possible to place #1.
  • the signal generation unit 313 causes the end effector 13 located at a position where the workpiece W#1 can be placed on the mounting device T#2 to A robot control signal may be generated for controlling the operations of the robot arm 12 and the end effector 13 so as to place the work W#1 on (that is, release the held work W#1).
  • the signal generation unit 313 causes the robot to move the end effector 13, which has finished placing the workpiece W#1 onto the mounting device T#2, away from the mounting device T#2.
  • Robot control signals may be generated to control movement of arm 12.
  • the robot 1 may perform a holding process to sequentially hold the plurality of works W placed on the mounting device T#1 moving on the support surface S one by one.
  • the robot 1 may perform a holding process to sequentially hold the plurality of works W placed on the placing device T#1 that is stationary on the support surface S, one by one.
  • the robot 1 may perform a release process in which the plurality of works W are placed one by one on the mounting device T#2 that moves on the support surface S.
  • the robot 1 may perform a release process in which the plurality of works W are placed one by one on the mounting device T#2 that is stationary on the support surface S.
  • the signal generation unit 313 uses the communication device 33 to output the robot control signal generated in step S4 to the robot 1 (in particular, the robot control device 14).
  • the robot control device 14 controls at least one of the operation of the robot 1 (for example, the operation of the robot arm 12) and the operation of the end effector 13 based on the robot control signal.
  • control device 3 repeats a series of processes from step S1 to step S4 until it is determined to end the robot control process (step S5).
  • control device 3 continues to acquire image data IMG_2D and IMG_3D from the imaging devices 21 and 22, respectively, even during a period when the operation of at least one of the robot arm 12 and the end effector 13 is controlled based on the robot control signal. .
  • each of the imaging devices 21 and 22 is configured to control the object OBJ and the imaging devices 21 and 22
  • the object OBJ may be imaged while moving relative to the object OBJ.
  • each of the imaging devices 21 and 22 may image the object OBJ while the imaging devices 21 and 22 are moving while the object OBJ is stationary.
  • each of the imaging devices 21 and 22 may image the object OBJ while the imaging devices 21 and 22 are stationary while the object OBJ is moving.
  • each of the imaging devices 21 and 22 may image the object OBJ while the object OBJ is moving and the imaging devices 21 and 22 are moving.
  • control device 3 controls the robot shown in FIG. Processing may continue.
  • the control device 3 indicates at least one of the position and orientation of the object OBJ based on the newly acquired image data IMG_2D and IMG_3D even during the period when the operation of the robot 1 is being controlled based on the robot control signal.
  • the position and orientation information POI0 can be newly generated (that is, updated).
  • each of the imaging devices 21 and 22 may image the object OBJ while the object OBJ is stationary and the imaging devices 21 and 22 are stationary.
  • the control device 3 may perform the robot control process shown in FIG. 4 while the imaging devices 21 and 22 and the object OBJ are stationary.
  • the position/orientation calculation unit 312 does not need to calculate at least one of the position and orientation of the object OBJ in the global coordinate system in step S3.
  • the position/orientation calculation unit 312 does not need to generate the position/orientation information POI0 indicating at least one of the position and orientation of the object OBJ in the global coordinate system.
  • the position and orientation calculation unit 312 calculates the object OBJ in a coordinate system different from the global coordinate system (for example, within a 2D imaging coordinate system, a 3D imaging coordinate system, or a common coordinate system, which will be described later). At least one of the position and orientation may be calculated.
  • the position and orientation calculation unit 312 calculates the position and orientation of the object OBJ in a coordinate system different from the global coordinate system (for example, within a 2D imaging coordinate system, a 3D imaging coordinate system, or a common coordinate system, which will be described later).
  • Position and orientation information POI0 indicating at least one may be generated.
  • the signal generation unit 313 generates a robot control signal using position and orientation information POI0 indicating at least one of the position and orientation of the object OBJ in a coordinate system different from the global coordinate system calculated in step S3. may be generated.
  • step S3 of FIG. Details of the process for calculating at least one of the postures (that is, the process for generating the position and posture information POI0) will be described.
  • the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the object OBJ by performing the same operation as the processing for calculating both the position and orientation of the object OBJ.
  • the position/orientation calculation unit 312 performs matching processing using image data IMG_2D and three-dimensional position data WSD, and tracking processing using image data IMG_2D and three-dimensional position data WSD, so that object OBJ Calculate the position and orientation of.
  • FIG. 10 is a block diagram showing logical processing blocks for calculating the position and orientation of object OBJ by performing matching processing and tracking processing.
  • the position and orientation calculation section 312 includes a 2D matching section 3121, a 3D matching section 3122, a tracking section 3123, and a coordinate transformation section 3125.
  • the 2D matching unit 3121 performs 2D matching processing, which is an example of matching processing.
  • the 3D matching unit 3122 performs 3D matching processing, which is an example of matching processing.
  • the tracking unit 3123 performs tracking processing.
  • the coordinate conversion unit 3125 performs a position and orientation calculation process that calculates the position and orientation of the object OBJ in the global coordinate system based on the results of the matching process and the tracking process. Therefore, below, 2D matching processing, 3D matching processing, tracking processing, and position calculation processing will be explained in order.
  • the 2D matching unit 3121 calculates the position and orientation of object OBJ in the 2D imaging coordinate system by performing 2D matching processing. That is, the 2D matching unit 3121 generates position and orientation information POI1 indicating the position and orientation of the object OBJ in the 2D imaging coordinate system by performing 2D matching processing.
  • the 2D imaging coordinate system is a coordinate system of the imaging device 21.
  • the 2D imaging coordinate system is a coordinate system based on the imaging device 21.
  • the position and orientation information POI1 may be considered to indicate the position and orientation of the object OBJ with respect to the imaging device 21.
  • the position and orientation information POI1 may be considered to indicate the position and orientation of the object OBJ as seen from the imaging device 21.
  • the 2D imaging coordinate system is a coordinate system defined by an X-axis (2D), a Y-axis (2D), and a Z-axis (2D) that are orthogonal to each other. At least one of the X-axis (2D), Y-axis (2D), and Z-axis (2D) is the optical axis AX21 (FIG. 2 (see). Note that the optical axis AX21 may be considered to be the optical axis of the imaging device 21. In the following description, an example will be described in which the Z-axis (2D) is an axis along the optical axis of the optical system included in the imaging device 21.
  • the 2D matching unit 3121 performs matching processing using image data IMG_2D as 2D matching processing.
  • the matching process using image data IMG_2D is matching process using image data IMG_2D and two-dimensional model data IMG_2M.
  • Two-dimensional model data IMG_2M is data indicating a two-dimensional model WM2 of object OBJ.
  • the two-dimensional model data IMG_2M is data indicating a two-dimensional model WM2 having a two-dimensional shape that serves as a reference for the object OBJ.
  • the two-dimensional model data IMG_2M is image data that represents a two-dimensional image that serves as a reference for the object OBJ. More specifically, the two-dimensional model data IMG_2M is image data indicating a two-dimensional image including the two-dimensional model WM2 of the object OBJ.
  • the two-dimensional model data IMG_2M is image data indicating a two-dimensional image including a two-dimensional model WM2 having a two-dimensional shape serving as a reference for the object OBJ.
  • the two-dimensional model data IMG_2M is, for example, a plurality of objects each generated by virtually projecting the three-dimensional model WM3 of the object OBJ from a plurality of different directions onto a virtual plane orthogonal to a plurality of different directions. It may be two-dimensional image data showing the two-dimensional model WM2 of OBJ.
  • the three-dimensional model WM3 of the object OBJ is a three-dimensional model having a three-dimensional shape that serves as a reference for the object OBJ.
  • An example of the three-dimensional model WM3 of the object OBJ is a CAD (Computer Aided Design) model of the object OBJ.
  • the three-dimensional model WM3 of the object OBJ is a three-dimensional model having the same shape as the three-dimensional shape of the object OBJ obtained by measuring the actual object OBJ in advance.
  • the three-dimensional model WM3 is generated in advance by the three-dimensional position data generation unit 311 based on image data IMG_3D generated by capturing an image of the object OBJ on which the projection pattern from the projection device 23 is projected using the imaging device 22. You may.
  • the three-dimensional model WM3 may be generated in advance using a known three-dimensional shape measuring device different from the robot system SYS.
  • the actual object OBJ whose shape is measured in advance may be a reference object or a non-defective object OBJ.
  • the two-dimensional model data IMG_2M may be image data indicating a two-dimensional image generated by capturing an image of the actual object OBJ in advance.
  • the two-dimensional model data IMG_2M may be image data indicating a plurality of two-dimensional images each generated by capturing images of the actual object OBJ from a plurality of different photographing directions.
  • image data IMG_2D generated by imaging the object OBJ by the imaging device 21 may be used as the two-dimensional model data IMG_2M.
  • image data generated by imaging the object OBJ by an arbitrary imaging device different from the robot system SYS may be used as the two-dimensional model data IMG_2M.
  • the object OBJ reflected in the two-dimensional image indicated by the image data IMG_2D as the two-dimensional model data IMG_2M may be referred to as a two-dimensional model WM2 of the object OBJ.
  • the actual object OBJ to be imaged in advance may be a reference or a non-defective object OBJ.
  • the 2D matching unit 3121 may perform matching processing on the image indicated by the image data IMG_2D using the two-dimensional image indicated by the two-dimensional model data IMG_2M as a template image. Specifically, the 2D matching unit 3121 may perform object detection processing to detect the object OBJ indicated by the template image within the image indicated by the image data IMG_2D as the matching processing. In other words, the 2D matching unit 3121 performs object detection to detect the object OBJ in the image indicated by the image data IMG_2D by detecting a similar image part similar to the template image in the image indicated by the image data IMG_2D as a matching process. Processing may be performed.
  • the 2D matching process itself may be the same as the existing matching process.
  • the 2D matching unit 3121 may perform 2D matching processing using a well-known method such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed-Upped Robust Feature).
  • SIFT Scale-Invariant Feature Transform
  • SURF Speed-Upped Robust Feature
  • the 2D matching unit 3121 determines whether a feature point (for example, at least one of a feature point or an edge) in the entire two-dimensional model WM2 of the object OBJ that is reflected in the template image is reflected in the image indicated by the image data IMG_2D. Changing one template image to another template image with a different projection direction or imaging direction so as to approach (typically, coincide with) characteristic points in the entire object OBJ, and objects reflected in the template image. At least one of parallel translation, enlargement, reduction, and/or rotation of the two-dimensional model WM2 of OBJ may be performed.
  • the 2D matching unit 3121 determines that the characteristic points in the entire two-dimensional model WM2 of the object OBJ reflected in the template image approach the characteristic points in the entire object OBJ reflected in the image indicated by the image data IMG_2D (typical Specifically, the coordinate system of the two-dimensional model data IMG_2M (for example, the coordinate system of the CAD model) and the 2D imaging coordinate system of the imaging device 21 that imaged the object OBJ (generated the image data IMG_2D) You may change the positional relationship. As a result, the 2D matching unit 3121 can specify the positional relationship between the coordinate system of the two-dimensional model data IMG_2M and the 2D imaging coordinate system.
  • the 2D matching unit 3121 calculates the 2D imaging coordinate system from the position and orientation of the object OBJ in the coordinate system of the 2D model data IMG_2M based on the positional relationship between the coordinate system of the 2D model data IMG_2M and the 2D imaging coordinate system.
  • the position and orientation of the object OBJ may also be calculated.
  • the matching similarity which is the similarity between the template image (that is, the two-dimensional model WM2 of the object OBJ) and the image indicated by the image data IMG_2D (particularly the image portion in which the object OBJ of the template image is combined), is calculated. Good too.
  • the 2D matching unit 3121 changes one template image to another template image with a different projection direction or imaging direction, and creates a two-dimensional model of the object OBJ reflected in the template image so that the matching similarity is maximized.
  • At least one of parallel translation, enlargement, reduction, and/or rotation of WM2 may be performed.
  • the image portion of the image indicated by the image data IMG_2D to which the two-dimensional model WM2 of the object OBJ of the template image is matched is within the image indicated by the image data IMG_2D. It is detected as a similar image portion that is similar to the template image. In other words, the object OBJ indicated by the similar image portion is detected.
  • the matching similarity may be considered to be equivalent to the correlation that indicates the correlation between the template image and the image indicated by the image data IMG_2D.
  • the degree of correlation can also be said to be an index indicating the correlation between the two-dimensional model WM2 of the object OBJ reflected in the template image and the object OBJ reflected in the image indicated by the image data IMG_2D.
  • the matching similarity may also be referred to as a matching score.
  • the 2D matching unit 3121 matches the characteristic points in a part of the two-dimensional model WM2 of the object OBJ reflected in the template image to the characteristic points in a part of the object OBJ reflected in the image indicated by the image data IMG_2D.
  • the positional relationship between the coordinate system of the two-dimensional model data IMG_2M (for example, the coordinate system of the CAD model) and the 2D imaging coordinate system of the imaging device 21 that imaged the object OBJ is determined so that they approach (typically, match). May be changed. Even in this case, the 2D matching unit 3121 may calculate the matching similarity.
  • the object OBJ is There may be a situation where the object OBJ is so large that a part of the object OBJ is out of the imaging range (field of view) of the imaging device 21 when the object OBJ approaches .
  • the imaging range field of view
  • the object OBJ is large, the shorter the distance between the object OBJ and the imaging device 21 (that is, the closer the object OBJ is to the imaging device 21), the more the imaging of the entire object OBJ becomes.
  • the proportion occupied by a part of the object OBJ located within the imaging range (field of view) of the device 21 is reduced.
  • the ratio of the part of the object OBJ that is actually reflected in the image indicated by the image data IMG_2D to the entire object OBJ decreases.
  • the state where "2D matching processing fails" refers to the state in which the object OBJ indicated by the template image is not included in the image indicated by the image data IMG_2D, even though the object OBJ is reflected in the image indicated by the image data IMG_2D.
  • the 2D matching unit 3121 performs the 2D matching process using a part of the two-dimensional model WM2 that corresponds to a part of the object OBJ included in the imaging range (field of view) of the imaging device 21. You may go. As a result, even if the object is large, the 2D matching unit 3121 can detect the object OBJ indicated by the template image within the image indicated by the image data IMG_2D.
  • the part of the two-dimensional model WM2 of the object OBJ that is reflected in each template image with a different projection direction or imaging direction is subjected to the 2D matching processing. It may be specified (in other words, selected) in advance as part of the two-dimensional model WM2 used for. For example, if the object OBJ is so large that when the object OBJ approaches the imaging device 21, a part of the object OBJ is out of the imaging range (field of view) of the imaging device 21, the object OBJ is included in the imaging range (field of view) of the imaging device 21.
  • a part of the two-dimensional model WM2 corresponding to a part of the object OBJ assumed to be may be specified in advance as a part of the two-dimensional model WM2 used for the 2D matching process.
  • a part of the two-dimensional model WM2 used in the 2D matching process may be specified (in other words, selected) while the 2D matching process is being performed. That is, one range of the two-dimensional model WM2 used for the 2D matching process may be changed while the 2D matching process is being performed. For example, consider that as the distance between the object OBJ and the imaging device 21 becomes shorter, the proportion of the part of the object OBJ located within the imaging range (field of view) of the imaging device 21 to the entire object OBJ decreases. Then, a part of the range of the two-dimensional model WM2 used for the 2D matching process may be changed depending on the distance between the object OBJ and the imaging device 21.
  • the range of the part of the two-dimensional model WM2 used for the 2D matching process becomes smaller (in other words, narrower) as the distance between the object OBJ and the imaging device 21 becomes shorter.
  • a part of the two-dimensional model WM2 may be specified.
  • a single portion of the two-dimensional model WM2 may be specified as a part of the two-dimensional model WM2 used in the 2D matching process.
  • multiple parts of the two-dimensional model WM2 may be specified as a part of the two-dimensional model WM2 used in the 2D matching process.
  • a plurality of parts of the two-dimensional model WM2, each of which can be used as a characteristic part of the two-dimensional model WM2 may be designated as part of the two-dimensional model WM2 used in the 2D matching process.
  • a part of the two-dimensional model WM2 used in the 2D matching process may be specified by the user of the robot system SYS.
  • the user may specify a part of the two-dimensional model WM2 displayed on a display (not shown).
  • the user estimates a part of the object OBJ that is assumed to be included in the imaging range (field of view) of the imaging device 21, and specifies a part of the two-dimensional model WM2 that corresponds to the estimated part of the object OBJ. It's okay.
  • the control device 3 may automatically designate a part of the two-dimensional model WM2 used in the 2D matching process.
  • the control device 3 estimates a part of the object OBJ that is assumed to be included in the imaging range (field of view) of the imaging device 21, and selects a part of the two-dimensional model WM2 corresponding to the estimated part of the object OBJ. May be specified.
  • the control device 3 controls, for example, the direction of relative movement between the object OBJ and the imaging device 21 (or the direction in which the robot 1 approaches the object OBJ), the distance between the object OBJ and the imaging device 21 (or the robot Object OBJ that is assumed to be included in the imaging range (field of view) of the imaging device 21 based on information regarding at least one of the distance between 1 and the object OBJ) and the imaging range (field of view) of the imaging device 21. You may estimate a part of it.
  • the 2D matching unit 3121 performs a process to bring the characteristic points of the two-dimensional model WM2 of the object OBJ reflected in the template image closer to the characteristic points of the object OBJ reflected in the image indicated by the image data IMG_2D.
  • the process may be performed sequentially for a plurality of objects OBJ that are reflected in the image indicated by the data IMG_2D.
  • the 2D matching unit 3121 may select one object OBJ with the maximum matching similarity among the plurality of objects OBJ reflected in the image indicated by the image data IMG_2D as a target for subsequent processing. .
  • the 2D matching unit 3121 selects one object OBJ whose matching similarity exceeds the matching determination threshold among the plurality of objects OBJ reflected in the image indicated by the image data IMG_2D as a target for subsequent processing. Good too.
  • the matching determination threshold value is determined between a state in which the object OBJ detected by the 2D matching process is the same as the object OBJ on which the end effector 13 should perform predetermined processing, and a 2D
  • the object OBJ detected by the matching process may be set to an appropriate value that allows the end effector 13 to perform a predetermined process to appropriately distinguish a different state from the object OBJ based on the matching similarity. .
  • the matching judgment threshold is determined based on the condition that the object OBJ detected by the 2D matching process is the same as the object OBJ reflected in the template image, and the condition that the object OBJ detected by the 2D matching process is the same as the object OBJ reflected in the template image.
  • the matching similarity may be set to an appropriate value that can appropriately distinguish between the object OBJ and a different state based on the matching similarity.
  • the method for calculating the position and orientation of the object OBJ is not limited to the matching process using the image data IMG_2D described above.
  • the 2D matching unit 3121 may calculate the position and orientation of the object OBJ using another well-known method of calculating the position and orientation of the object OBJ using the image data IMG_2D.
  • the method of calculating the position and orientation of the object OBJ may be a well-known method of calculating the position and orientation of the object OBJ based on the image data IMG_2D without using the two-dimensional model data IMG_2M, or the method of calculating the position and orientation of the object OBJ based on the image data IMG_2D without using the two-dimensional model data IMG_2M.
  • a well-known method may be used to calculate the position and orientation of the object OBJ using data different from IMG_2M and image data IMG_2D.
  • the method of calculating the position and orientation of the object OBJ may be a method of calculating the position and orientation of the object OBJ based on the image data IMG_2D using machine learning or deep learning.
  • a prediction model is constructed in advance to output the position and orientation of the object OBJ when the image data IMG_2D is input, and by inputting the image data IMG_2D to this prediction model, the position and orientation of the object OBJ is and the posture may be calculated.
  • This prediction model may be stored in the 2D matching section 3121. Note that the 2D matching unit 3121 may read out this prediction model stored in the storage device 32.
  • the 2D matching unit 3121 determines, as the position of the object OBJ in the 2D imaging coordinate system, a position Tx (2D) of the object OBJ in the X-axis direction (2D) parallel to the X-axis (2D), and a Y parallel to the Y-axis (2D). At least one of the position Ty (2D) of the object OBJ in the axial direction (2D) and the position Tz (2D) of the object OBJ in the Z-axis direction (2D) parallel to the Z-axis (2D) may be calculated. .
  • the 2D matching unit 3121 determines the rotation amount Rx (2D) of the object OBJ around the X-axis (2D) and the rotation amount Ry (2D) of the object OBJ around the Y-axis (2D) as the posture of the object OBJ in the 2D imaging coordinate system. and the amount of rotation Rz (2D) of the object OBJ around the Z-axis (2D).
  • the rotation amount Rx (2D) of the object OBJ around the X axis (2D), the Y axis (2D) The rotation amount Ry (2D) of the surrounding object OBJ and the rotation amount Rz (2D) of the object OBJ around the Z axis (2D) are calculated as the posture Rx (2D) of the object OBJ around the X axis (2D) and the Y axis, respectively.
  • (2D) is referred to as the posture Ry (2D) of the surrounding object OBJ and the posture Rz (2D) of the object OBJ around the Z axis (2D).
  • the posture Rx (2D) of the object OBJ around the X-axis (2D), the posture Ry (2D) of the object OBJ around the Y-axis (2D), and the posture Rz (2D) of the object OBJ around the Z-axis (2D) are , respectively, indicate the position of the object OBJ in the rotational direction around the X-axis (2D), the position of the object OBJ in the rotational direction around the Y-axis (2D), and the position of the object OBJ in the rotational direction around the Z-axis (2D), respectively. It may be assumed that there is.
  • the posture Rx (2D) of the object OBJ around the X-axis (2D), the posture Ry (2D) of the object OBJ around the Y-axis (2D), and the posture Rz (2D) of the object OBJ around the Z-axis (2D) are , may be regarded as parameters representing the position of the object OBJ.
  • the 2D matching unit 3121 Each time the 2D matching unit 3121 generates the position and orientation information POI1, it outputs the generated position and orientation information POI1 to the 3D matching unit 3122.
  • the 2D matching unit 3121 selects all of the positions and orientations of 6DOF (Degree Of Freedom) (that is, position Tx (2D), position Ty (2D), position Tz (2D), orientation Rx (2D), and orientation Ry). (2D) and orientation Rz(2D)), and output position/orientation information POI1 indicating the calculated position and orientation of 6DOF to the 3D matching unit 3122.
  • 6DOF Degree Of Freedom
  • the 2D matching unit 3121 calculates the positions and orientations of the 6DOFs, and calculates some of the calculated positions and orientations of the 6DOFs (i.e., position Tx (2D), position Ty (2D), position Tz ( 2D), orientation Rx (2D), orientation Ry (2D), and part of orientation Rz (2D)) may be output to the 3D matching unit 3122.
  • the 2D matching unit 3121 may calculate a part of the position and orientation of the 6DOF, and output position and orientation information POI1 indicating a part of the calculated position and orientation of the 6DOF to the 3D matching unit 3122. good.
  • the 2D matching unit 3121 sets the position and orientation of the object OBJ as position Tx (2D), position Ty (2D), position Tz (2D), An example of calculating posture Rx (2D), posture Ry (2D), and posture Rx (2D) will be described. That is, in the following description, for convenience of explanation, as shown in FIG. An example in which position and orientation information POI1 indicating (2D) and orientation Rx (2D) is generated and output to the 3D matching unit 3122 will be described.
  • the 2D matching unit 3121 may calculate the position and orientation of the object OBJ at a predetermined 2D matching cycle. In other words, the 2D matching unit 3121 may calculate the position and orientation of the object OBJ every time the time corresponding to the 2D matching period elapses. In other words, it may be assumed that a time corresponding to the 2D matching period is required from when the 2D matching unit 3121 starts the 2D matching process until it completes the 2D matching process. In this case, the 2D matching unit 3121 may output position and orientation information POI1 indicating the position and orientation of the object OBJ to the 3D matching unit 3122 at a predetermined 2D matching cycle.
  • the 2D matching period may be longer than the 2D imaging period corresponding to the 2D imaging rate at which the imaging device 21 images the object OBJ.
  • the 2D matching period may be a period in which the 2D matching unit 3121 can perform the 2D matching process from ten to several dozen times (as an example, from 15 to 30 times) per second.
  • the imaging device 21 generates new image data IMG_2D from when the 2D matching unit 3121 starts the 2D matching process until it completes the 2D matching process.
  • the 2D matching unit 3121 will not be able to match the newly generated image data IMG_2D until the started 2D matching process is completed.
  • the 2D matching unit 3121 may start a new 2D matching process using the latest image data IMG_2D at that time. Note that even if the 2D matching process that has been started is not completed, the 2D matching unit 3121 uses the newly generated image data IMG_2D by the imaging device 21 when the imaging device 21 generates new image data IMG_2D. You may start a new 2D matching process using . Note that the timing at which the image data IMG_2D shown in FIG. 12 is generated may be considered to be the timing at which the imaging device 21 images the object OBJ in order to generate the image data IMG_2D.
  • the 2D matching cycle does not have to be longer than the 2D imaging cycle.
  • the 2D matching period may be shorter than the 2D imaging period.
  • the 2D matching period may be the same as the 2D imaging period.
  • the 3D matching unit 3122 calculates the position and orientation of object OBJ in the 3D imaging coordinate system by performing 3D matching processing. That is, the 3D matching unit 3122 generates position and orientation information POI2 indicating the position and orientation of the object OBJ in the 3D imaging coordinate system by performing 3D matching processing.
  • the 3D imaging coordinate system is a coordinate system of the imaging device 22.
  • the 3D imaging coordinate system is a coordinate system based on the imaging device 22.
  • the position and orientation information POI2 may be considered to indicate the position and orientation of the object OBJ with respect to the imaging device 22.
  • the position and orientation information POI2 may be considered to indicate the position and orientation of the object OBJ as seen from the imaging device 22.
  • the 3D imaging coordinate system is a coordinate system defined by an X-axis (3D), a Y-axis (3D), and a Z-axis (3D) that are orthogonal to each other. At least one of the X-axis (3D), Y-axis (3D), and Z-axis (3D) is the optical axis AX22 (FIG. 2 (see). Note that the optical axis AX22 may be considered to be the optical axis of the imaging device 22.
  • the optical axis AX22 is the optical axis of the optical system included in either one of the two monocular cameras. Good too.
  • the optical axis AX22 may be the optical axis of either one of the two monocular cameras.
  • the Z-axis (3D) is an axis along the optical axis of the optical system included in the imaging device 22.
  • the 3D matching unit 3122 performs matching processing using the three-dimensional position data WSD as at least part of the 3D matching processing. That is, the 3D matching unit 3122 performs 3D matching processing including matching processing using the three-dimensional position data WSD.
  • the matching process using the three-dimensional position data WSD is a matching process using the three-dimensional position data WSD and the three-dimensional model data WMD.
  • the three-dimensional model data WMD is data indicating the three-dimensional model WM3 of the object OBJ.
  • the three-dimensional model data WMD is data indicating a three-dimensional model WM3 having a three-dimensional shape that serves as a reference for the object OBJ.
  • the three-dimensional model WM3 may be a CAD model of the object OBJ, which is an example of the three-dimensional model WM3 of the object OBJ.
  • the three-dimensional model WM3 may be a three-dimensional model having a shape similar to the three-dimensional shape of the object OBJ obtained by measuring the three-dimensional shape of the actual object OBJ in advance.
  • the three-dimensional model data WMD is generated by the three-dimensional position data generation unit 311 based on the image data IMG_3D generated by imaging the object OBJ on which the projection pattern from the projection device 23 is projected using the imaging device 22. may be generated in advance.
  • the three-dimensional model data WMD may be generated in advance by shape measurement using a known three-dimensional shape measuring device different from the robot system SYS.
  • the three-dimensional model data WMD may be depth image data indicating the three-dimensional model WM3 of the object OBJ.
  • the three-dimensional model data WMD may be point group data indicating the three-dimensional model WM3 of the object OBJ.
  • the actual object OBJ that is imaged or measured in advance for generating the three-dimensional model data WMD may be a reference or a non-defective object OBJ.
  • the 3D matching unit 3122 may perform matching processing on the three-dimensional position data WSD using the three-dimensional model WM3 indicated by the three-dimensional model data WMD as a template model.
  • the matching process itself using the three-dimensional position data WSD may be the same as the existing matching process.
  • the 3D matching unit 3122 uses RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), and ICP (Iterative Closest Point). using a well-known method including at least one of t) and DSO (Direct Sparse Odometry).
  • RANSAC Random Sample Consensus
  • SIFT Scale-Invariant Feature Transform
  • ICP Iterative Closest Point
  • the 3D matching unit 3122 determines that a feature location in the entire three-dimensional model WM3 indicated by the three-dimensional model data WMD approaches a feature location in the entire object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD (typically
  • the three-dimensional model WM3 indicated by the three-dimensional model data WMD may be translated, enlarged, reduced, and/or rotated within the 3D imaging coordinate system so that the three-dimensional model data WMD matches the three-dimensional model data WMD.
  • the 3D matching unit 3122 determines that the feature location in the entire object OBJ whose 3D position is indicated by the 3D position data WSD approaches the feature location in the entire 3D model WM3 indicated by the 3D model data WMD (typically The positional relationship between the coordinate system of the three-dimensional model data WMD (for example, the coordinate system of the CAD model) and the 3D imaging coordinate system may be changed so that they match. As a result, the 3D matching unit 3122 can specify the positional relationship between the coordinate system of the three-dimensional model data WMD and the 3D imaging coordinate system.
  • the 3D matching unit 3122 performs 3D imaging from the position and orientation of the 3D model WM3 in the coordinate system of the 3D model data WMD, based on the positional relationship between the coordinate system of the 3D model data WMD and the 3D imaging coordinate system.
  • the position and orientation of the object OBJ in the coordinate system may be calculated.
  • the coordinate system of the three-dimensional model data WMD may be referred to as the coordinate system of the three-dimensional model WM3 indicated by the three-dimensional model data WMD.
  • the 3D matching unit 3122 determines that a characteristic point in a part of the three-dimensional model WM3 indicated by the three-dimensional model data WMD approaches a characteristic point in a part of the object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD (typical In other words, the positional relationship between the coordinate system of the three-dimensional model data WMD (for example, the coordinate system of the CAD model) and the 3D imaging coordinate system of the imaging device 22 that imaged the object OBJ is changed so that they match Good too.
  • the object OBJ is connected to the imaging device 22.
  • the object OBJ is so large that a part of the object OBJ is out of the imaging range (field of view) of the imaging device 22 when the object OBJ approaches .
  • the imaging range field of view
  • the proportion occupied by a portion of the object OBJ located within the imaging range (field of view) of the device 22 is reduced.
  • the ratio of the part of the object OBJ that is actually reflected in the image indicated by the image data IMG_3D to the entire object OBJ decreases. Therefore, the proportion of the part of the object OBJ in which the three-dimensional position data WSD generated from the image data IMG_3D indicates the three-dimensional position occupies with respect to the entire object OBJ. If 3D matching processing is performed using the entire three-dimensional model WM3 under such circumstances, there is a high possibility that the 3D matching processing will fail.
  • the state where "3D matching processing fails” refers to the state where the 3D matching process fails even though the 3D position data WSD includes the 3D position data of the object OBJ. It may also mean a state in which the indicated object OBJ cannot be detected. Specifically, the smaller the proportion of the part of the object OBJ that is actually reflected in the image indicated by the image data IMG_3D (in other words, the more the three-dimensional position data WSD generated from the image data IMG_3D indicates the three-dimensional position The smaller the proportion occupied by a part of the object OBJ), the higher the possibility that the 3D matching process will fail.
  • the 3D matching unit 3122 selects a part of the object OBJ included in the imaging range (field of view) of the imaging device 22 (a part of the object OBJ whose three-dimensional position data WSD indicates the three-dimensional position).
  • 3D matching processing may be performed using a part of the three-dimensional model WM3 corresponding to .
  • a part of the 3D model WM3 of the object OBJ is specified in advance as a part of the 3D model WM3 used for the 3D matching process ( In other words, it may be selected (selected). For example, if the object OBJ is so large that when the object OBJ approaches the imaging device 22, a part of the object OBJ is out of the imaging range (field of view) of the imaging device 22, the object OBJ is included in the imaging range (field of view) of the imaging device 22.
  • a part of the three-dimensional model WM3 corresponding to a part of the object OBJ assumed to be may be specified in advance as a part of the three-dimensional model WM3 used in the 3D matching process.
  • a part of the three-dimensional model WM3 used in the 3D matching process may be specified (in other words, selected) while the 3D matching process is being performed. That is, one range of the three-dimensional model WM3 used for the 3D matching process may be changed while the 3D matching process is being performed. For example, consider that as the distance between the object OBJ and the imaging device 22 becomes shorter, the proportion of the part of the object OBJ located within the imaging range (field of view) of the imaging device 22 to the entire object OBJ decreases. Then, a part of the range of the three-dimensional model WM3 used for the 3D matching process may be changed depending on the distance between the object OBJ and the imaging device 22.
  • the range of the part of the three-dimensional model WM3 used for the 3D matching process becomes smaller (in other words, narrower) as the distance between the object OBJ and the imaging device 22 becomes shorter.
  • a part of the three-dimensional model WM3 may be specified.
  • a single portion of the three-dimensional model WM3 may be specified as a part of the three-dimensional model WM3 used in the 3D matching process.
  • multiple parts of the three-dimensional model WM3 may be specified as parts of the three-dimensional model WM3 used in the 3D matching process.
  • a plurality of parts of the three-dimensional model WM3, each of which can be used as a characteristic part of the three-dimensional model WM3, may be designated as part of the three-dimensional model WM3 used in the 3D matching process.
  • a part of the three-dimensional model WM3 used in the 3D matching process may be specified by the user of the robot system SYS.
  • the user may specify a part of the three-dimensional model WM3 displayed on a display (not shown).
  • the user estimates a part of the object OBJ that is assumed to be included in the imaging range (field of view) of the imaging device 22, and specifies a part of the three-dimensional model WM3 that corresponds to the estimated part of the object OBJ. You can.
  • the control device 3 may automatically designate a part of the three-dimensional model WM3 used in the 3D matching process.
  • control device 3 controls a part of the object OBJ that is assumed to be included in the imaging range (field of view) of the imaging device 22 (the three-dimensional position data WSD that is assumed to be generated from the image data IMG_3D A portion of the three-dimensional model WM3 corresponding to the estimated portion of the object OBJ may be specified.
  • control device 3 controls, for example, the direction of relative movement between the object OBJ and the imaging device 22 (or the direction in which the robot 1 approaches the object OBJ), the distance between the object OBJ and the imaging device 22 (or the robot 1 and the object OBJ) and the imaging range (field of view) of the imaging device 22, the object OBJ is assumed to be included in the imaging range (field of view) of the imaging device 22. may be partially estimated.
  • a process of bringing the feature parts of the three-dimensional model WM3 indicated by the three-dimensional model data WMD closer to the feature parts of the object OBJ indicated by the three-dimensional position data WSD that is, matching processing using the three-dimensional position data WSD
  • the 3D matching unit 3122 determines at least one of the initial position and initial orientation of the three-dimensional model WM3. That is, before starting the matching process using the three-dimensional position data WSD, the 3D matching unit 3122 determines at least one of the initial position and initial orientation of the three-dimensional model WM3 as at least part of the 3D matching process. Perform posture determination processing.
  • the 3D matching process performed by the 3D matching unit 3122 includes position and orientation determination processing that determines at least one of the initial position and initial orientation of the three-dimensional model WM3, and matching using the three-dimensional position data WSD. It may also include processing.
  • the 3D matching process includes a matching process using the three-dimensional position data WSD, it may not include the position and orientation determination process.
  • the 3D matching unit 3122 may perform position and orientation determination processing as processing separate from the 3D matching processing.
  • a processing block or device different from the 3D matching unit 3122 may perform the position and orientation determination processing.
  • the 3D matching process converts the characteristic points of the three-dimensional model WM3 indicated by the three-dimensional model data WMD to the characteristic points of the object OBJ indicated by the three-dimensional position data WSD in the 3D imaging coordinate system. (that is, matching processing using three-dimensional position data WSD). Therefore, the 3D matching unit 3122 may determine the initial position and initial orientation of the three-dimensional model WM3 within the 3D imaging coordinate system. Thereafter, the 3D matching unit 3122 arranges the three-dimensional model WM3 at the determined initial position and in the determined initial posture within the 3D imaging coordinate system.
  • the 3D matching unit 3122 starts a process of bringing the feature points of the three-dimensional model WM3 placed in the initial position and initial posture closer to the feature points of the object OBJ indicated by the three-dimensional position data WSD. That is, the 3D matching unit 3122 starts matching processing using the three-dimensional model WM3 and the three-dimensional position data WSD at the initial position and initial orientation.
  • the 3D matching process is performed by matching the 3D model so that the feature location of the 3D model WM3 indicated by the 3D model data WMD approaches the feature location of the object OBJ whose 3D position data WSD indicates the 3D position. It can also be said that it includes processing for changing the positional relationship between the coordinate system of the data WMD (for example, the coordinate system of the CAD model) and the 3D imaging coordinate system.
  • the operation of determining the initial position and initial orientation of the three-dimensional model WM3 within the 3D imaging coordinate system is essentially the initial positional relationship between the coordinate system of the three-dimensional model data WMD and the 3D imaging coordinate system. It may be considered to be equivalent to the operation of determining the state.
  • the 3D matching unit 3122 adjusts the coordinate system of the 3D model data WMD and the 3D imaging coordinate system so that the positional relationship between the coordinate system of the 3D model data WMD and the 3D imaging coordinate system is in the determined initial state. may be aligned. Thereafter, the 3D matching unit 3122 may start a process of bringing the feature points of the three-dimensional model WM3 closer to the feature points of the object OBJ indicated by the three-dimensional position data WSD.
  • the position and orientation calculation unit 312 calculates both the position and orientation of the object OBJ in step S3 of FIG. Both the initial position and initial attitude of the three-dimensional model WM3 within the imaging coordinate system are determined.
  • the 3D matching unit 3122 calculates both the initial position and initial orientation of the three-dimensional model WM3 within the 3D imaging coordinate system. may be determined.
  • the 3D matching unit 3122 may use the position and orientation information POI1 output from the 2D matching unit 3121 to the 3D matching unit 3122. That is, the 3D matching unit 3122 may determine the initial position and initial orientation of the three-dimensional model WM3 based on the position and orientation information POI1.
  • the 3D matching unit 3122 converts the position and orientation information POI1 indicating the position and orientation of the object OBJ in the 2D imaging coordinate system into the position and orientation of the object OBJ in the 3D imaging coordinate system. and the position/orientation information POI1_conv indicating the orientation.
  • the position and orientation information POI1_conv may indicate the position Tx (2D to 3D) of the object OBJ in the X-axis direction (3D) parallel to the X-axis (3D) of the 3D imaging coordinate system, which is calculated by conversion.
  • the position and orientation information POI1_conv may indicate the position Ty (2D to 3D) of the object OBJ in the Y-axis direction (3D) parallel to the Y-axis (3D) of the 3D imaging coordinate system, which is calculated by conversion.
  • the position and orientation information POI1_conv may indicate the position Tz (2D to 3D) of the object OBJ in the Z-axis direction (3D) parallel to the Z-axis (3D) of the 3D imaging coordinate system, which is calculated by conversion.
  • the position and orientation information POI1_conv may indicate the amount of rotation Rx (2D to 3D) of the object OBJ around the X-axis (3D) of the 3D imaging coordinate system, which is calculated by conversion.
  • the position and orientation information POI1_conv may indicate the amount of rotation Ry (2D to 3D) of the object OBJ around the Y axis (3D) of the 3D imaging coordinate system, which is calculated by conversion.
  • the position and orientation information POI1_conv may indicate the amount of rotation Rz (2D to 3D) of the object OBJ around the Z axis (3D) of the 3D imaging coordinate system, which is calculated by conversion.
  • the amount of rotation Rz (2Dto3D) of the object OBJ around the axis (3D) is determined by the attitude Rx (2Dto3D) of the object OBJ in the rotational direction around the X-axis (3D) and the object OBJ in the rotational direction around the Y-axis (3D), respectively.
  • the posture Ry (2Dto3D) of the object OBJ in the rotation direction around the Z axis (3D) is referred to as the posture Rz (2Dto3D).
  • the posture Rx (2Dto3D) of the object OBJ around the X axis (2Dto3D), the posture Ry (2Dto3D) of the object OBJ around the Y axis (2Dto3D), and the posture Rz (2Dto3D) of the object OBJ around the Z axis (2Dto3D) are , respectively indicate the position of the object OBJ in the rotational direction around the X-axis (2Dto3D), the position of the object OBJ in the rotational direction around the Y-axis (2Dto3D), and the position of the object OBJ in the rotational direction around the Z-axis (2Dto3D). It may be assumed that there is.
  • the posture Rx (2Dto3D) of the object OBJ around the X-axis (2Dto3D), the posture Ry (2Dto3D) of the object OBJ around the Y-axis (2Dto3D), and the posture Rz (2Dto3D) of the object OBJ around the Z-axis (2Dto3D) are , may be regarded as parameters representing the position of the object OBJ.
  • the 3D matching unit 3122 may use coordinate system information indicating the positional relationship between the 2D imaging coordinate system and the 3D imaging coordinate system.
  • the coordinate system information may be stored in the storage device 32 in advance, for example.
  • the coordinate system information is information for converting a position in either the 2D imaging coordinate system or the 3D imaging coordinate system to a position in the other coordinate system of the 2D imaging coordinate system or the 3D imaging coordinate system. It may also include a transformation matrix (typically a rigid transformation matrix). The transformation matrix can be calculated using a mathematical method from external parameters indicating the positional relationship between the imaging device 21 and the imaging device 22.
  • the transformation matrix can be calculated by a mathematical method that solves a PnP (Perspective n Point) problem based on external parameters indicating the positional relationship between the imaging device 21 and the imaging device 22.
  • the control device 3 converts the position and orientation information POI1 indicating the position and orientation of the object OBJ in the 2D imaging coordinate system to the position and orientation of the object OBJ in the 3D imaging coordinate system based on the coordinate system information (in particular, the transformation matrix). It can be converted into position and orientation information POI1_conv indicating the position and orientation.
  • the 3D matching unit 3122 may set the position indicated by the position and orientation information POI1_conv as the initial position of the three-dimensional model WM3. Furthermore, the 3D matching unit 3122 may set the orientation indicated by the position and orientation information POI1_conv as the initial orientation of the three-dimensional model WM3. For example, the 3D matching unit 3122 may set the position Tx (2D to 3D) indicated by the position and orientation information POI1_conv as the initial position of the three-dimensional model WM3 in the X-axis direction (3D).
  • the 3D matching unit 3122 may set the position Ty (2D to 3D) indicated by the position and orientation information POI1_conv as the initial position of the three-dimensional model WM3 in the Y-axis direction (3D).
  • the 3D matching unit 3122 may set the position Tz (2D to 3D) indicated by the position and orientation information POI1_conv as the initial position of the three-dimensional model WM3 in the Z-axis direction (3D).
  • the 3D matching unit 3122 may set the orientation Rx (2D to 3D) indicated by the position and orientation information POI1_conv as the initial orientation of the three-dimensional model WM3 around the X-axis (3D).
  • the 3D matching unit 3122 may set the posture Ry (2D to 3D) indicated by the position and posture information POI1_conv as the initial posture of the three-dimensional model WM3 around the Y-axis (3D).
  • the 3D matching unit 3122 may set the orientation Rz (2D to 3D) indicated by the position and orientation information POI1_conv as the initial orientation of the three-dimensional model WM3 around the Z-axis (3D).
  • the 3D matching unit 3122 places the three-dimensional model WM3 at the determined initial position and in the determined initial posture within the 3D imaging coordinate system.
  • the 3D matching unit 3122 may arrange the three-dimensional model WM3 so that the position of the three-dimensional model WM3 in the X-axis direction (3D) is the position Tx (2D to 3D) within the 3D imaging coordinate system.
  • the 3D matching unit 3122 may arrange the three-dimensional model WM3 so that the position of the three-dimensional model WM3 in the Y-axis direction (3D) is the position Ty (2D to 3D) within the 3D imaging coordinate system.
  • the 3D matching unit 3122 may arrange the three-dimensional model WM3 so that the position of the three-dimensional model WM3 in the Z-axis direction (3D) is the position Tz (2D to 3D) within the 3D imaging coordinate system.
  • the 3D matching unit 3122 may arrange the three-dimensional model WM3 so that the posture of the three-dimensional model WM3 around the X-axis (3D) becomes the posture Rx (2D to 3D) in the 3D imaging coordinate system.
  • the 3D matching unit 3122 may arrange the three-dimensional model WM3 so that the posture of the three-dimensional model WM3 around the Y-axis (3D) becomes the posture Ry (2D to 3D) in the 3D imaging coordinate system.
  • the 3D matching unit 3122 may arrange the three-dimensional model WM3 so that the posture of the three-dimensional model WM3 around the Z-axis (3D) becomes the posture Rz (2D to 3D) in the 3D imaging coordinate system.
  • the 3D matching unit 3122 may arrange the three-dimensional model WM3 at the determined initial position and in the determined initial posture within the 3D imaging coordinate system using the procedure described below. Specifically, the 3D matching unit 3122 may first place (set) the three-dimensional model WM3 within the coordinate system of the three-dimensional model WM3 (within the coordinate system of the three-dimensional model data WSD). In this case, the 3D matching unit 3122 may arrange the three-dimensional model WM3 within the coordinate system of the three-dimensional model WM3 so that the center of gravity of the three-dimensional model WM3 is located at the origin within the coordinate system of the three-dimensional model WM3. good.
  • the 3D matching unit 3122 aligns the coordinate system of the three-dimensional model WM3 and the 3D imaging coordinate system. For example, the 3D matching unit 3122 aligns the coordinate system of the 3D model WM3 and the 3D imaging coordinate system so that the origin of the coordinate system of the 3D model WM3 matches the origin of the 3D imaging coordinate system. Good too. For example, the 3D matching unit 3122 matches the coordinate system of the 3D model WM3 and the 3D imaging coordinate system so that the X axis of the coordinate system of the 3D model WM3 matches the X axis (3D) of the 3D imaging coordinate system. Positioning may also be performed.
  • the 3D matching unit 3122 matches the coordinate system of the three-dimensional model WM3 and the 3D imaging coordinate system so that the Y-axis of the coordinate system of the three-dimensional model WM3 matches the Y-axis (3D) of the 3D imaging coordinate system. Positioning may also be performed. For example, the 3D matching unit 3122 adjusts the coordinate system of the 3D model WM3 and the 3D imaging coordinate system so that the Z axis of the coordinate system of the 3D model WM3 matches the Z axis (3D) of the 3D imaging coordinate system. Positioning may also be performed.
  • the 3D matching unit 3122 positions the coordinate system of the three-dimensional model WM3 and the 3D imaging coordinate system so that the scaling of the coordinates in the coordinate system of the three-dimensional model WM3 matches the scaling of the coordinates in the 3D imaging coordinate system. You may also perform matching. Thereafter, the 3D matching unit 3122 may arrange the three-dimensional model WM3 at the determined initial position and in the determined initial attitude within the coordinate system of the three-dimensional model WM3 in which the three-dimensional model WM3 is placed.
  • positioning the three-dimensional model WM3 within the coordinate system of the three-dimensional model WM3 means that the 3D imaging coordinate system This is equivalent to arranging the three-dimensional model WM3 within.
  • the 3D matching unit 3122 can arrange the three-dimensional model WM3 at the determined initial position and in the determined initial attitude within the 3D imaging coordinate system.
  • the 3D matching unit 3122 aligns the coordinate system of the three-dimensional model WM3 and the coordinate system of the three-dimensional model WM3.
  • a transformation matrix typically a rigid transformation matrix
  • the three-dimensional model WM3 may be placed at the determined initial position and in the determined initial posture.
  • the 3D matching unit 3122 places the 3D model WM3 in the coordinate system of the 3D model WM3 based on the transformation matrix.
  • the position and orientation of the three-dimensional model WM3 (that is, the position and orientation within the coordinate system of the three-dimensional model WM3) may be converted into the position and orientation of the three-dimensional model WM3 within the 3D imaging coordinate system.
  • the 3D matching unit 3122 may arrange the three-dimensional model WM3 at the determined initial position and in the determined initial posture within the 3D imaging coordinate system.
  • the 3D matching unit 3122 determines the initial position and initial orientation of the three-dimensional model WM3 within the 2D imaging coordinate system. and an initial attitude may be determined. In this case, the 3D matching unit 3122 converts the position and orientation indicated by the position and orientation information POI1 to the initial positions of the three-dimensional model WM3 within the 2D imaging coordinate system, without converting the position and orientation information POI1 to position and orientation information POI1_conv. It may be set to the position and initial attitude.
  • the 3D matching unit 3122 aligns the coordinate system of the three-dimensional model WM3 with the 2D imaging coordinate system. You may do so. Specifically, the 3D matching unit 3122 may align the coordinate system of the three-dimensional model WM3 and the 2D imaging coordinate system after arranging the three-dimensional model WM3 within the coordinate system of the three-dimensional model WM3. . For example, the 3D matching unit 3122 aligns the coordinate system of the 3D model WM3 and the 2D imaging coordinate system so that the origin of the coordinate system of the 3D model WM3 matches the origin of the 2D imaging coordinate system. Good too.
  • the 3D matching unit 3122 matches the coordinate system of the 3D model WM3 and the 2D imaging coordinate system so that the X axis of the coordinate system of the 3D model WM3 matches the X axis (2D) of the 2D imaging coordinate system. Positioning may also be performed.
  • the 3D matching unit 3122 matches the coordinate system of the 3D model WM3 and the 2D imaging coordinate system so that the Y axis of the coordinate system of the 3D model WM3 matches the Y axis (2D) of the 2D imaging coordinate system. Positioning may also be performed.
  • the 3D matching unit 3122 adjusts the coordinate system of the 3D model WM3 and the 3D imaging coordinate system so that the Z axis of the coordinate system of the 3D model WM3 matches the Z axis (2D) of the 2D imaging coordinate system. Positioning may also be performed.
  • the 3D matching unit 3122 positions the coordinate system of the three-dimensional model WM3 and the 2D imaging coordinate system so that the scaling of the coordinates in the coordinate system of the three-dimensional model WM3 matches the scaling of the coordinates in the 2D imaging coordinate system. You may also perform matching.
  • the 3D matching unit 3122 may arrange the three-dimensional model WM3 at the determined initial position and in the determined initial attitude within the coordinate system of the three-dimensional model WM3 in which the three-dimensional model WM3 is placed.
  • the 3D matching unit 3122 can arrange the three-dimensional model WM3 at the determined initial position and in the determined initial attitude within the 2D imaging coordinate system.
  • the 3D matching unit 3122 converts the position in either the 2D imaging coordinate system or the 3D imaging coordinate system to the position in the other coordinate system of the 2D imaging coordinate system or the 3D imaging coordinate system.
  • the position and orientation of the three-dimensional model WM3 in the 2D imaging coordinate system are changed using a transformation matrix (typically, a rigid transformation matrix) for You can also convert it into a posture.
  • the position and orientation of the three-dimensional model WM3 within the 3D imaging coordinate system obtained by the transformation reach the initial position and initial orientation of the three-dimensional model WM3 within the 3D imaging coordinate system, respectively.
  • the 3D matching unit 3122 can arrange the three-dimensional model WM3 at the determined initial position and in the determined initial attitude within the 3D imaging coordinate system.
  • the 3D matching unit 3122 starts a process of bringing the feature points of the three-dimensional model WM3 closer to the feature points of the object OBJ indicated by the three-dimensional position data WSD. As a result, the 3D matching unit 3122 can calculate the position and orientation of the object OBJ. In other words, the 3D matching unit 3122 can generate position and orientation information POI2.
  • the method for calculating the position and orientation of the object OBJ is not limited to the above-described matching process using the three-dimensional position data WSD.
  • the 3D matching unit 3122 may calculate the position and orientation of the object OBJ using another well-known method of calculating the position and orientation of the object OBJ using the three-dimensional position data WSD.
  • the method of calculating the position and orientation of the object OBJ may be a well-known method of calculating the position and orientation of the object OBJ based on the three-dimensional position data WSD without using the three-dimensional model data WMD, or the method of calculating the position and orientation of the object OBJ based on the three-dimensional position data WSD without using the three-dimensional model data WMD.
  • a well-known method may be used to calculate the position of the object OBJ using data different from the data WMD and the three-dimensional position data WSD.
  • the method of calculating the position and orientation of the object OBJ may be a method of calculating the position and orientation of the object OBJ based on the three-dimensional position data WSD by machine learning or deep learning.
  • a predictive model is constructed in advance to output the position and orientation of the object OBJ when the three-dimensional position data WSD is input, and the three-dimensional position data WSD is input to this predictive model.
  • the position and orientation of object OBJ may also be calculated.
  • This prediction model may be stored in the 3D matching section 3122. Note that the 3D matching unit 3122 may read out this prediction model stored in the storage device 32.
  • the 3D matching unit 3122 performs 3D matching processing using the above-mentioned point cloud data as the three-dimensional position data WSD. As a result, the accuracy of calculating the position and orientation of the object OBJ becomes higher than when depth image data is used as the three-dimensional position data WSD.
  • the 3D matching unit 3122 uses point cloud data (for example, point cloud data indicating a CAD model) indicating a three-dimensional shape serving as a reference of the object OBJ as the three-dimensional model data WMD.
  • the 3D matching unit 3122 may perform 3D matching processing using the depth image data described above as the three-dimensional position data WSD.
  • the 3D matching unit 3122 may use, as the three-dimensional model data WMD, depth image data indicating a three-dimensional shape serving as a reference of the object OBJ (for example, depth image data indicating a CAD model).
  • the 3D matching unit 3122 determines, as the position of the object OBJ in the 3D imaging coordinate system, a position Tx (3D) of the object OBJ in the X-axis direction (3D) parallel to the X-axis (3D), and a Y parallel to the Y-axis (3D). At least one of the position Ty (3D) of the object OBJ in the axial direction (3D) and the position Tz (3D) of the object OBJ in the Z-axis direction (3D) parallel to the Z-axis (3D) may be calculated. .
  • the 3D matching unit 3122 determines the rotation amount Rx (3D) of the object OBJ around the X-axis (3D) and the rotation amount Ry (3D) of the object OBJ around the Y-axis (3D) as the posture of the object OBJ in the 3D imaging coordinate system. and the amount of rotation Rz (3D) of the object OBJ around the Z-axis (3D).
  • the rotation amounts Rx (3D) of the object OBJ around the X-axis (3D), Y-axis (3D) The rotation amount Ry (3D) of the surrounding object OBJ and the rotation amount Rz (3D) of the object OBJ around the Z axis (3D) are calculated as the posture Rx (3D) of the object OBJ around the X axis (3D) and the Y axis, respectively.
  • (3D) is referred to as the posture Ry (3D) of the surrounding object OBJ and the posture Rz (3D) of the object OBJ around the Z axis (3D).
  • the posture Rx (3D) of the object OBJ around the X-axis (3D), the posture Ry (3D) of the object OBJ around the Y-axis (3D), and the posture Rz (3D) of the object OBJ around the Z-axis (3D) are , respectively, indicate the position of the object OBJ in the rotational direction around the X-axis (3D), the position of the object OBJ in the rotational direction around the Y-axis (3D), and the position of the object OBJ in the rotational direction around the Z-axis (3D), respectively. It may be assumed that there is.
  • the posture Rx (3D) of the object OBJ around the X-axis (3D), the posture Ry (3D) of the object OBJ around the Y-axis (3D), and the posture Rz (3D) of the object OBJ around the Z-axis (3D) are , may be regarded as parameters representing the position of the object OBJ.
  • the 3D matching unit 3122 Each time the 3D matching unit 3122 generates the position and orientation information POI2, it outputs the generated position and orientation information POI2 to the coordinate conversion unit 3125.
  • the 3D matching unit 3122 stores all of the positions and orientations of 6DOF (that is, position Tx (3D), position Ty (3D), position Tz (3D), orientation Rx (3D), orientation Ry (3D), and orientation Rz(3D)) and outputs position/orientation information POI1 indicating the calculated position and orientation of 6DOF to the coordinate conversion unit 3125.
  • the 3D matching unit 3122 may calculate the positions and orientations of the 6 DOFs, and output position and orientation information POI2 indicating a part of the calculated positions and orientations of the 6 DOFs to the coordinate conversion unit 3125. . As another example, the 3D matching unit 3122 may calculate a part of the position and orientation of the 6DOF, and output position and orientation information POI2 indicating a part of the calculated position and orientation of the 6DOF to the coordinate conversion unit 3125. good.
  • the 3D matching unit 3122 may calculate the position and orientation of the object OBJ at a predetermined 3D matching cycle. In other words, the 3D matching unit 3122 may calculate the position and orientation of the object OBJ each time the time corresponding to the 3D matching period elapses. In other words, it may be assumed that a time period corresponding to the 3D matching period is required from when the 3D matching unit 3122 starts the 3D matching process until it completes the 3D matching process. In this case, the 3D matching unit 3122 may output position and orientation information POI2 indicating the position and orientation of the object OBJ to the coordinate conversion unit 3125 at a predetermined 3D matching cycle.
  • the 3D matching period may be longer than the 3D imaging period corresponding to the 3D imaging rate at which the imaging device 22 images the object OBJ.
  • the 3D matching period may be a period in which the 3D matching unit 3122 can perform the 3D matching process from several times to several dozen times (as an example, from 1 time to 30 times) per second.
  • the imaging device 22 generates new image data IMG_3D from when the 3D matching unit 3122 starts the 3D matching process until it completes the 3D matching process.
  • the 3D matching unit 3122 will not be able to process the newly generated image data IMG_3D until the started 3D matching process is completed.
  • the 3D matching unit 3122 may start a new 3D matching process using new image data IMG_3D after completing the already started 3D matching process. Note that even if the 3D matching process that has already been started has not been completed, the 3D matching unit 3122 uses the newly generated image by the imaging device 22 when the imaging device 22 generates new image data IMG_3D. A new 3D matching process using data IMG_3D may be started. Note that the timing at which the image data IMG_3D shown in FIG. 15 is generated may be considered to be the timing at which the imaging device 22 images the object OBJ in order to generate the image data IMG_3D. However, the 3D matching cycle does not need to be longer than the 3D imaging cycle. For example, the 3D matching period may be shorter than the 3D imaging period. For example, the 3D matching period may be the same as the 3D imaging period.
  • the 3D matching unit 3122 performs 3D matching processing using the result of the 2D matching processing performed by the 2D matching unit 3121 (that is, the position and orientation information POI1). Therefore, when the imaging devices 21 and 22 generate image data IMG_2D and IMG_3D, respectively, by capturing an image of the object OBJ at a certain time ta, the control device 3 generates a 2D image using the image data IMG_2D generated at the time ta.
  • the matching process and the 3D matching process using the three-dimensional position data WSD generated from the image data IMG_3D generated at time ta may be performed at different timings.
  • the 2D matching unit 3121 starts 2D matching processing using the image data IMG_2D generated at time ta.
  • the 3D matching unit 3122 uses the three-dimensional position data WSD generated from the image data IMG_3D generated at time ta until the 2D matching process using image data IMG_2D generated at time ta is completed. There is no need to start the 3D matching process. This is because, as described above, the 3D matching unit 3122 determines the initial position and initial orientation of the three-dimensional model WM3 using the results of the 2D matching process.
  • the 3D matching unit 3122 calculates the result of the 2D matching process (that is, the object OBJ in the 2D imaging coordinate system at time ta).
  • the 3D matching process may be started using the position and orientation information POI1) indicating the position and orientation and the three-dimensional position data WSD generated from the image data IMG_3D generated at time ta.
  • the 3D matching unit 3122 may generate position and orientation information POI2 indicating the position and orientation of the object OBJ in the 3D imaging coordinate system at time ta.
  • the 3D matching unit 3122 uses the position and orientation information POI2 that indicates the position and orientation of the object OBJ in the 3D imaging coordinate system at the time ta, and the imaging devices 21 and 22 to image the object OBJ at the time ta. It may be generated after a period corresponding to the sum of the 2D matching period and the 3D matching period has elapsed.
  • the 2D matching unit 3121 performs 2D matching processing using image data IMG_2D generated at one time, and the three-dimensional position generated from image data IMG_3D generated at another time different from one time.
  • the 3D matching unit 3122 may perform 3D matching processing using the data WSD. For example, when the 2D matching unit 3121 performs the 2D matching process using the image data IMG_2D generated at time ta, the 3D matching unit 3122 performs the 2D matching process using the image data IMG_2D generated at time ta. 3D matching processing may be performed using the three-dimensional position data WSD generated from IMG_3D.
  • the 3D matching unit 3122 determines that the remaining time until the 2D matching processing ends is less than a predetermined time.
  • the 3D matching process may be performed using the three-dimensional position data WSD generated from the image data IMG_3D generated at the time.
  • the 3D matching period may be the same as the 2D matching period.
  • the length of the 3D matching period may be the same as the length of the 2D matching period.
  • the time from when the 3D matching section 3122 starts to the completion of the 3D matching process may be the same as the time from when the 2D matching section 3121 starts to the completion of the 2D matching process.
  • the 3D matching period does not have to be the same as the 2D matching period.
  • the length of the 3D matching period may be longer than the length of the 2D matching period.
  • the length of the 3D matching period may be shorter than the length of the 2D matching period.
  • the data size of the three-dimensional position data WSD and the three-dimensional model data WMD used in the 3D matching process is usually larger than the data size of the image data IMG_2D and the two-dimensional model data IMG_2M used in the 2D matching process. Therefore, the 3D matching cycle, which is the time required for the 3D matching unit 3122 to complete the 3D matching process, is longer than the 2D matching cycle, which is the time required for the 2D matching unit 3121 to complete the 2D matching process. There is a possibility that it will happen. Furthermore, in this embodiment, the 3D matching process includes a position/orientation determination process for determining the initial position and initial orientation of the 3D model WM3 in addition to the matching process using the 3D position data WSD.
  • the matching period can be longer than the 2D matching period. Therefore, the 3D matching unit 3122 may perform processing to shorten the 3D matching period so that the 3D matching period becomes the same as the 2D matching period or so that the 3D matching period becomes shorter than the 2D matching period. .
  • the 3D matching unit 3122 uses the results of the 2D matching process (for example, the above-mentioned position Tx (2D), position Ty (2D), position Tz (2D), attitude Rx (2D), attitude Ry (2D), and Based on the position and orientation information POI1) indicating Rz (2D), a part of the data portion of the three-dimensional position data WSD may be selected as the 3D matching target data for performing the 3D matching process.
  • the 3D matching unit 3122 may estimate the area where the object OBJ is estimated to exist within the 3D imaging coordinate system based on the position and orientation information POI1.
  • the 3D matching unit 3122 converts position and orientation information POI1 indicating the position and orientation of the object OBJ in the 2D imaging coordinate system to position and orientation information POI1_conv indicating the position and orientation of the object OBJ in the 3D imaging coordinate system. You can. Thereafter, the 3D matching unit 3122 may estimate an area where the object OBJ is estimated to exist within the 3D imaging coordinate system based on the position and orientation information POI1_conv. For example, the 3D matching unit 3122 may estimate an area of a predetermined size that includes the position indicated by the position and orientation information POI1_conv as an area in which the object OBJ is estimated to exist within the 3D imaging coordinate system.
  • the 3D matching unit 3122 may estimate a region of a predetermined shape that includes the position indicated by the position and orientation information POI1_conv as the region in which the object OBJ is estimated to exist within the 3D imaging coordinate system. Thereafter, the 3D matching unit 3122 may select a part of the data portion of the three-dimensional position data WSD that corresponds to a region where the object OBJ is estimated to exist, as the 3D matching target data. Thereafter, the 3D matching unit 3122 may perform 3D matching processing on the 3D matching target data that is part of the 3D position data WSD.
  • the three-dimensional position data WSD may include not only a data part related to the object OBJ but also a data part related to another object different from the object OBJ. This is because the imaging range (field of view) of the imaging device 22 that generates the image data IMG_3D used to generate the three-dimensional position data WSD includes not only the object OBJ but also other objects different from the object OBJ.
  • the three-dimensional position data WSD is generated from the image data IMG_3D generated by the imaging device 22 capturing an image of the work W placed on the support surface S
  • the three-dimensional position data WSD is There is a possibility that it includes not only a data part related to the support surface S but also a data part related to the support surface S.
  • the process of selecting the 3D matching target data described above is equivalent to the process of selecting the data portion related to the workpiece W.
  • the 3D matching target data may typically be considered to be equivalent to data obtained by removing a data portion related to another object different from the object OBJ from the three-dimensional position data WSD.
  • the 3D matching target data may be considered to be typically equivalent to data obtained by selectively extracting a data portion related to the object OBJ from the three-dimensional position data WSD.
  • the operation of selecting the 3D matching target data that is part of the 3D position data WSD is equivalent to the operation of not selecting (or deleting) the data part other than the 3D matching target data of the 3D position data WSD. It may be assumed that Therefore, based on the result of the 2D matching process, the 3D matching unit 3122 selects a part of the data portion of the three-dimensional position data WSD that corresponds to an area where it is estimated that the object OBJ does not exist (for example, as described above). 3D matching target data, which is a part of the three-dimensional position data WSD, may be generated by deleting the data portion related to an object different from the object OBJ.
  • the 3D matching unit 3122 may select 3D matching target data based on a user's instruction specifying a part of the data portion of the three-dimensional position data WSD.
  • the control device 3 uses an output device 35 including a display device to provide a GUI that can be operated by the user in order to designate a part of the data portion of the three-dimensional position data WSD as 3D matching target data. (Graphical User Interface) may be displayed.
  • the control device 3 uses the output device 35 including a display device to display a plurality of points indicated by point cloud data that is an example of the three-dimensional position data WSD, and displays a plurality of points corresponding to the object OBJ among the plurality of points.
  • a GUI that can be operated by the user may be displayed to specify a point.
  • the control device 3 uses the output device 35 including a display device to display a depth image indicated by depth image data that is an example of the three-dimensional position data WSD, and selects pixels corresponding to the object OBJ in the depth image.
  • a GUI that can be operated by the user may be displayed for specification.
  • the 3D matching unit 3122 may perform 3D matching processing on 3D matching target data, which is a part of the 3D position data WSD specified by the user. As a result, the 3D matching cycle becomes shorter than when the 3D matching process is performed on the entire three-dimensional position data WSD.
  • Tracking processing performed by the tracking unit 3123 performs tracking processing such that the imaging device 21 images the object OBJ at a first time t1 and a second time t2 after the first time t1. Tracking processing is performed using the two image data IMG_2D#t1 and IMG_2D#t2 generated by this. As shown in FIG. 17, the tracking process using the two image data IMG_2D#t1 and IMG_2D#t2 is performed by tracking at least one feature point (for example, a feature point and an edge) of the object OBJ reflected in the image data IMG_2D#t1.
  • the tracking process using the two image data IMG_2D#t1 and IMG_2D#t2 is performed by tracking at least one feature point (for example, a feature point and an edge) of the object OBJ reflected in the image data IMG_2D#t1.
  • the tracking unit 3123 calculates the amount of change in the position of at least one feature point between the first time t1 and the second time t2 by performing tracking processing. Specifically, the tracking unit 3123 generates position information PI4 indicating the position of at least one feature of the object OBJ at the first time t1, based on the image data IMG_2D#t1 at the first time t1. For example, the tracking unit 3123 may generate position information PI4 indicating the position of the feature in at least one of the X-axis direction (2D) and the Y-axis direction (2D) in the 2D imaging coordinate system.
  • the tracking unit 3123 generates position information PI5 indicating the position of at least one feature of the object OBJ at the second time t2, based on the image data IMG_2D#t2 at the second time t2. For example, the tracking unit 3123 may generate position information PI5 indicating the position of the feature in at least one of the X-axis direction (2D) and the Y-axis direction (2D) in the 2D imaging coordinate system.
  • each of the position information PI4 and PI5 indicates the position of a characteristic location in each of the X-axis direction (2D) and the Y-axis direction (2D).
  • the tracking unit 3123 calculates the amount of change in at least one of the position and orientation of the object OBJ between the first time t1 and the second time t2 based on the position information PI4 and PI5. As a result, the tracking unit 3123 generates change amount information VI indicating the amount of change in at least one of the position and orientation of the object OBJ between the first time t1 and the second time t2.
  • the position and orientation calculation unit 312 calculates both the position and orientation of the object OBJ.
  • the tracking unit 3123 may generate change amount information VI indicating the amount of change in both the position and orientation of the object OBJ.
  • change amount information VI indicating the amount of change in both the position and orientation of the object OBJ.
  • the tracking unit 3123 may calculate the amount of change in the position of the object OBJ in the 2D imaging coordinate system.
  • the tracking unit 3123 calculates the amount of change ⁇ Tx (2D) in the position Tx (2D) of the object OBJ in the X-axis direction (2D) and the object in the Y-axis direction (2D) as the amount of change in the position of the object OBJ in the 2D imaging coordinate system.
  • At least one of the amount of change ⁇ Ty (2D) in the position Ty (2D) of OBJ and the amount of change ⁇ Tz (2D) in the position Tz (2D) of the object OBJ in the Z-axis direction (2D) may be calculated.
  • the tracking unit 3123 calculates the change amount ⁇ Rx (2D) in the posture Rx (2D) of the object OBJ around the X-axis (2D) and the object around the Y-axis (2D) as the amount of change in the posture of the object OBJ in the 2D imaging coordinate system. At least one of the amount of change ⁇ Ry(2D) in the posture Ry(2D) of OBJ and the amount ⁇ Rz(2D) of change in the posture Rz(2D) of the object OBJ around the Z-axis (2D) may be calculated.
  • the tracking unit 3123 may calculate the amount of change in the position of the object OBJ in the 3D imaging coordinate system.
  • the tracking unit 3123 calculates the amount of change ⁇ Tx (3D) in the position Tx (3D) of the object OBJ in the X-axis direction (3D) and the object in the Y-axis direction (3D) as the amount of change in the position of the object OBJ in the 3D imaging coordinate system.
  • At least one of the amount of change ⁇ Ty (3D) in the position Ty (3D) of OBJ and the amount of change ⁇ Tz (3D) in the position Tz (3D) of the object OBJ in the Z-axis direction (3D) may be calculated.
  • the tracking unit 3123 calculates the change amount ⁇ Rx (3D) in the posture Rx (3D) of the object OBJ around the X-axis (3D) and the object around the Y-axis (3D) as the amount of change in the posture of the object OBJ in the 3D imaging coordinate system. At least one of the amount of change ⁇ Ry (3D) in the posture Ry (3D) of OBJ and the amount ⁇ Rz (3D) of change in the posture Rz (3D) of the object OBJ around the Z-axis (3D) may be calculated.
  • the tracking unit 3123 may calculate the amount of change in the position of the object OBJ in the global coordinate system.
  • the tracking unit 3123 calculates, as the amount of change in the position of the object OBJ in the global imaging coordinate system, the amount of change ⁇ Tx (GL) in the position Tx (GL) of the object OBJ in the X-axis direction (GL), and the amount of change ⁇ Tx (GL) in the position Tx (GL) of the object OBJ in the Y-axis direction (GL).
  • At least one of the amount of change ⁇ Ty (GL) in the position Ty (GL) of OBJ and the amount of change ⁇ Tz (GL) in the position Tz (GL) of object OBJ in the Z-axis direction (GL) may be calculated.
  • the tracking unit 3123 calculates the amount of change ⁇ Rx (GL) in the posture Rx (GL) of the object OBJ around the X-axis (GL) and the amount of change ⁇ Rx (GL) in the posture of the object OBJ around the Y-axis (GL) as the amount of change in the posture of the object OBJ in the global imaging coordinate system.
  • At least one of the amount of change ⁇ Ry(GL) in the posture Ry(GL) of OBJ and the amount ⁇ Rz(GL) of change in the posture Rz(GL) of the object OBJ around the Z-axis (GL) may be calculated.
  • the amount of change in the position of the object OBJ calculated by the tracking unit 3123 may be considered to be equivalent to the amount of change in the relative position of the object OBJ and the imaging device 21.
  • the amount of change in the posture of the object OBJ calculated by the tracking unit 3123 may be considered to be equivalent to the amount of change in the relative posture of the object OBJ and the imaging device 21.
  • the tracking unit 3123 generates the change amount information VI based on the three-dimensional position data WSD in addition to the two image data IMG_2D#t1 and IMG_2D#t2. Specifically, in addition to the two image data IMG_2D#t1 and IMG_2D#t2 generated by the imaging device 22 capturing images of the object OBJ at the first time t1 and the second time t2, the tracking unit 3123 The change amount information VI is also generated based on the three-dimensional position data WSD generated from the image data IMG_3D generated by the imaging device 22 capturing an image of the object OBJ at the third time t3.
  • the third time t3 at which the imaging device 22 images the object OBJ for the tracking process is the same time as the second time t2 at which the imaging device 21 images the object OBJ for the tracking process.
  • the tracking unit 3123 records the image data IMG_3D generated by the imaging device 22 imaging the object OBJ at the third time t3, which is the same as the second time t2.
  • the amount of change information VI is also generated based on the three-dimensional position data WSD generated from the three-dimensional position data WSD.
  • the state that "the second time t2 and the third time t3 are the same time” may include the state that "the second time t2 and the third time t3 are literally the same time”. good.
  • the state where "the second time t2 and the third time t3 are the same time” means “the second time t2 and the third time t3 are not completely the same time, but the second time t2 and the third time t3 are the same time”.
  • the second time t2 and the third time t3 can be considered to be substantially the same time because the time difference with t3 is smaller than the allowable upper limit value.
  • the allowable upper limit value may be an allowable upper limit value based on a control error of the robot arm 12.
  • the time difference between the second time t2 and the third time t3 causes an error in the calculation result of at least one of the position and orientation of the object OBJ, which will be described later (in other words, the calculated position and orientation of the object OBJ).
  • an error in the calculation of at least one of the position and orientation of the object OBJ may cause a control error in the robot arm 12.
  • a control error of the robot arm 12 becomes a movement error of the end effector 13, and the end effector 13 may not be able to appropriately perform a predetermined process on the object OBJ.
  • the allowable upper limit value can also be said to be the allowable upper limit value of the movement error of the end effector 13 by the robot arm 12. Further, for example, even if there is a time difference between the second time t2 and the third time t3 due to a synchronization error in the imaging processing of the imaging devices 21 and 22, the second time t2 and the third time t3 are substantially different from each other. It may be assumed that they are at the same time. Note that the synchronization error in the imaging processing of the imaging devices 21 and 22 may be a synchronization control error in the imaging processing of the imaging devices 21 and 22 by the control device 3.
  • the second time t2 and the third time t3 may be different times.
  • the state where "the second time t2 and the third time t3 are different times" means "the time difference between the second time t2 and the third time t3 is larger than the allowable upper limit, so the second time t2 and the third time t3 are different”.
  • the third time t3 may not be considered to be substantially the same time.
  • the imaging devices 21 and 22 when the imaging devices 21 and 22 image the object OBJ during a period in which each of the imaging devices 21 and 22 and the object OBJ are relatively displaced, the imaging devices 21 and 22 The object OBJ may be imaged so that the second time t2 and the third time t3 are the same time. That is, when the imaging devices 21 and 22 image the object OBJ during a period in which each of the imaging devices 21 and 22 and the object OBJ are relatively displaced, the control device 3 The imaging devices 21 and 22 may be controlled so that the third time t3 is the same time. The reason for this will be explained again when explaining the effects of the robot system SYS.
  • the imaging devices 21 and 22 image the object OBJ during a period in which each of the imaging devices 21 and 22 and the object OBJ are not relatively displaced, the imaging devices 21 and 22 capture images of the object OBJ at the second time. It is not necessary to image the object OBJ so that t2 and third time t3 are the same time. That is, the control device 3 does not have to control the imaging devices 21 and 22 so that the second time t2 and the third time t3 are the same time. For example, the imaging devices 21 and 22 may image the object OBJ so that the second time t2 and the third time t3 are different times.
  • the control device 3 may control the imaging devices 21 and 22 so that the second time t2 and the third time t3 are different times.
  • the imaging devices 21 and 22 image the object OBJ during a period in which each of the imaging devices 21 and 22 and the object OBJ are not relatively displaced, the imaging devices 21 and 22
  • the object OBJ may be imaged so that the third time t3 and the third time t3 are the same time. That is, the control device 3 may control the imaging devices 21 and 22 so that the second time t2 and the third time t3 are the same time.
  • the tracking unit 3123 In order to generate the change amount information VI based on the three-dimensional position data WSD, the tracking unit 3123 first generates position information PI3 indicating the position of at least one feature of the object OBJ based on the three-dimensional position data WSD. generate. Since the three-dimensional position data WSD is generated from the image data IMG_3D generated by the image capturing device 22 capturing an image of the object OBJ at the third time t3, the tracking unit 3123 calculates the first position based on the three-dimensional position data WSD. 3. Position information PI3 indicating the position of at least one characteristic point of object OBJ at time t3 is generated.
  • the tracking unit 3123 may calculate the position of at least one feature of the object OBJ in the Z-axis direction (3D) of the 3D imaging coordinate system. In this case, the tracking unit 3123 determines the position of the object OBJ in the Z-axis direction (3D) of the 3D imaging coordinate system based on the three-dimensional position of each of the plurality of points of the object OBJ in the 3D imaging coordinate system indicated by the three-dimensional position data WSD. The position of at least one feature point may be calculated. As a result, the tracking unit 3123 may generate position information PI3 indicating the position of at least one feature of the object OBJ in the Z-axis direction (3D).
  • the tracking unit 3123 may calculate the position of at least one feature point of the object OBJ in the Z-axis direction (2D) of the 2D imaging coordinate system.
  • the tracking unit 3123 stores the above coordinate system information (for example, the position in one of the 2D imaging coordinate system and the 3D imaging coordinate system, and the position in one of the 2D imaging coordinate system and the 3D imaging coordinate system).
  • conversion matrix for converting to a position in the coordinate system of It may also be converted into the three-dimensional position of each of a plurality of points on the OBJ.
  • the tracking unit 3123 may calculate the position of at least one feature of the object OBJ in the Z-axis direction (2D) based on the three-dimensional positions of each of the plurality of points of the object OBJ in the 2D imaging coordinate system. . As a result, the tracking unit 3123 may generate position information PI3 indicating the position of at least one feature of the object OBJ in the Z-axis direction (2D).
  • the tracking unit 3123 After that, the tracking unit 3123 generates change amount information based on the position information PI4 and PI5 generated from the two image data IMG_2D#t1 and IMG_2D#t2, respectively, and the position information PI3 generated from the three-dimensional position data WSD. Generate a VI. Specifically, the tracking unit 3123 calculates the position information PI5, which indicates the position of at least one characteristic part of the object OBJ at the second time t2, from a third point that is the same as the second time t2 (however, different depending on the case). Position information PI3 indicating the position of at least one feature point of object OBJ at time t3 is reflected.
  • the position information PI5 indicates the position of the feature in each of the X-axis direction (2D) and the Y-axis direction (2D), as described above
  • the position information PI3 indicates the position of the characteristic location in the Z-axis direction (2D) as described above.
  • the position of the feature point in the direction (2D) or the Z-axis direction (3D) is shown.
  • the tracking unit 3123 detects the feature in the Z-axis direction (2D) or the Z-axis direction (3D) indicated by the position information PI3 with respect to the position information PI5 that does not indicate the position of the feature in the Z-axis direction (2D).
  • the location of the part may also be added.
  • the tracking unit 3123 calculates the position information PI5 from the three-dimensional position data WSD in addition to the positions of the characteristic points in the X-axis direction (2D) and Y-axis direction (2D) calculated from the image data IMG_2D.
  • the position information PI3 is reflected in the position information PI5 so that the position information indicates the position of the characteristic location in the Z-axis direction (2D) or the Z-axis direction (3D).
  • both the position information PI5 and the position information PI3 indicate the position in the 2D imaging coordinate system.
  • the coordinate system of the position indicated by the position information PI5 is the same as the coordinate system of the position indicated by the position information PI3.
  • the tracking unit 3123 may add the position information PI3 indicating the position of the feature in the 2D imaging coordinate system to the position information PI5 indicating the position of the feature in the 2D imaging coordinate system. good.
  • the position information PI3 indicates the position of the feature in the Z-axis direction (3D)
  • the position information PI3 indicates the position in the 3D imaging coordinate system
  • the position information PI5 indicates the position in the 2D imaging coordinate system. It shows the position in the imaging coordinate system.
  • the coordinate system of the position indicated by the position information PI5 is different from the coordinate system of the position indicated by the position information PI3.
  • the tracking unit 3123 converts one of the position information PI3 and PI5 so that the coordinate system of the position indicated by the position information PI5 is the same as the coordinate system of the position indicated by the position information PI3, and then converts the coordinate system to the position information PI5.
  • Position information PI3 may be added to the information.
  • the tracking unit 3123 may store the above coordinate system information (for example, a position in one of the 2D imaging coordinate system and 3D imaging coordinate system, and a position in the other coordinate system of the 2D imaging coordinate system and 3D imaging coordinate system)
  • the position of the feature in the 2D imaging coordinate system indicated by the position information PI5 is converted to the position of the feature in the 3D imaging coordinate system using a transformation matrix for converting to a position in the coordinate system.
  • the tracking unit 3123 may add position information PI3 that has been converted to indicate the position of the feature in the 3D imaging coordinate system to position information PI5 indicating the position of the feature in the 3D imaging coordinate system. .
  • the tracking unit 3123 After that, the tracking unit 3123 generates change amount information VI based on the position information PI4 and the position information PI5 on which the position information PI3 is reflected. In other words, the tracking unit 3123 detects a change in at least one of the position and orientation of the object OBJ between the first time t1 and the second time t2 based on the position information POI4 and the position information PI5 reflecting the position information PI3. Calculate the amount.
  • the position information PI4 indicates the position of the characteristic location in each of the X-axis direction and the Y-axis direction.
  • the position information PI5 indicates the position of the characteristic location in each of the X-axis direction, Y-axis direction, and Z-axis direction.
  • the process of calculating the amount of change information VI indicating the amount of change in the position and orientation of the object OBJ can be considered to be substantially equivalent to the process of solving a PnP (Perspective n Point) problem.
  • PnP Perspective n Point
  • a general PnP problem involves solving the positions (three-dimensional positions) of n feature points of object OBJ existing in three-dimensional space and the positions of n feature points of object OBJ on a two-dimensional image.
  • This is a problem of estimating the position and orientation (specifically, the translation matrix and rotation matrix) of the imaging device 21 or 22 that images the object OBJ.
  • the estimated translation matrix and rotation matrix convert the position and orientation of the object OBJ existing in the three-dimensional space to the position and orientation of the object OBJ on the two-dimensional image, or conversely, convert the position and orientation of the object OBJ in the two-dimensional image.
  • the tracking unit 3123 uses position information PI4 indicating the positions (two-dimensional positions) of n feature points of the object OBJ at the first time t1 and the position information PI4 of the object OBJ at the second time t2 as a PnP problem.
  • position information PI4 indicating the positions (two-dimensional positions) of n feature points of the object OBJ at the first time t1 and the position information PI4 of the object OBJ at the second time t2 as a PnP problem.
  • a problem may be set up to estimate the required translation and rotation matrices.
  • the tracking unit 3123 may estimate the translation matrix and rotation matrix by solving a PnP problem. Thereafter, the tracking unit 3123 may calculate the amount of change in the position of the object OBJ between the first time t1 and the second time t2 based on the estimated translation matrix. The tracking unit 3123 may calculate the amount of change in the posture of the object OBJ between the first time t1 and the second time t2 based on the estimated rotation matrix.
  • the method for calculating the amount of change in the position and orientation of object OBJ is limited to tracking processing using two image data IMG_2D#t1 and IMG_2D#t2 (and three-dimensional position data WSD if necessary). do not have.
  • the tracking unit 3123 uses another well-known method to calculate the amount of change in the position and orientation of the object OBJ using the two image data IMG_2D#t1 and IMG_2D#t2 (and the three-dimensional position data WSD as necessary). may be used to calculate the amount of change in the position and orientation of the object OBJ.
  • a method for calculating the amount of change in the position and orientation of object OBJ is to use machine learning or deep learning to calculate two image data IMG_2D#t1 and IMG_2D#t2 (and three-dimensional position data WSD as necessary).
  • the amount of change in the position and orientation of the object OBJ may be calculated based on the above.
  • machine learning or deep learning when two image data IMG_2D#t1 and IMG_2D#t2 (and three-dimensional position data WSD if necessary) are input, the amount of change in the position and orientation of the object OBJ is output.
  • the position and orientation of the object OBJ can be determined by constructing a prediction model in advance for The amount of change may also be calculated.
  • the tracking unit 3123 may include this prediction model.
  • the tracking unit 3123 calculates the change amount information VI, it outputs the generated change amount information VI to the coordinate conversion unit 3125.
  • the tracking unit 3123 detects all of the changes in the positions and orientations of the 6 DOFs (that is, the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (2D), the amount of change ⁇ Rx (2D), The amount ⁇ Ry (2D) and the amount of change ⁇ Rz (2D), or the amount of change ⁇ Tx (3D), the amount of change ⁇ Ty (3D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and The amount of change ⁇ Rz (3D)) may be calculated, and the amount of change information VI indicating the amount of change in the calculated position and orientation of 6 DOF may be output to the coordinate conversion unit 3125.
  • the tracking unit 3123 calculates a part of the amount of change in the position and orientation of the 6DOF, and transfers the amount of change information VI indicating a part of the amount of change in the calculated position and orientation of the 6DOF to the coordinate conversion unit 3125. You can also output to
  • the tracking unit 3123 may calculate the amount of change in the position and orientation of the object OBJ at a predetermined tracking cycle. In other words, the tracking unit 3123 may calculate the amount of change in the position and orientation of the object OBJ each time the time corresponding to the tracking period elapses. Conversely, it may be assumed that a time period corresponding to the tracking cycle is required from when the tracking unit 3123 starts tracking processing until it completes the tracking process. In this case, the tracking unit 3123 may output change amount information VI indicating the amount of change in the position and orientation of the object OBJ to the coordinate conversion unit 3125 at a predetermined tracking cycle.
  • the tracking period may be the same as the 2D imaging period corresponding to the 2D imaging rate at which the imaging device 21 images the object OBJ.
  • the tracking cycle may be a cycle that allows the tracking unit 3123 to perform tracking processing several tens to hundreds of times (as an example, 500 times) per second.
  • the tracking unit 3123 determines whether the image data IMG_2D#ta is generated at time ta, at the same time as or after the image data IMG_2D#ta is generated at time ta and at time ta-1 before time ta.
  • Tracking processing may be performed using image data IMG_2D#ta and IMG_2D#ta-1 and three-dimensional position data WSD generated using image data IMG_3D#ta generated at time ta.
  • the tracking unit 3123 generates the image data IMG_2D#ta+1 at time ta+1 at the same time or after the image data IMG_2D#ta+1 is generated, at time ta+1 and at time ta before time ta+1.
  • Tracking processing may be performed using data IMG_2D#ta+1 and IMG_2D#ta and three-dimensional position data WSD generated using image data IMG_3D#ta+1 generated at time ta+1.
  • the amount of change in the position and orientation of the object OBJ between times ta and ta+1 is calculated.
  • the timing at which the image data IMG_2D shown in FIG. 18 is generated may be considered to be the timing at which the imaging device 21 images the object OBJ in order to generate the image data IMG_2D.
  • the tracking period may be the same as the 3D imaging period corresponding to the 3D imaging rate at which the imaging device 22 images the object OBJ.
  • the tracking unit 3123 generates the image data IMG_3D#ta using the image data IMG_3D#ta generated at the time ta, simultaneously with or after the image data IMG_3D#ta is generated at the time ta.
  • Tracking processing may be performed using the three-dimensional position data WSD generated at time ta and image data IMG_2D#ta and IMG_2D#ta-1 generated at time ta-1 before time ta.
  • the tracking unit 3123 generates image data IMG_3D#ta+1 using image data IMG_3D#ta+1 generated at time ta+1, simultaneously with or after image data IMG_3D#ta+1 is generated at time ta+1.
  • Tracking processing may be performed using the three-dimensional position data WSD and the image data IMG_2D#ta+1 and IMG_2D#ta generated at time ta+1 and time ta before time ta+1.
  • the timing at which the image data IMG_3D shown in FIG. 18 is generated may be considered to be the timing at which the imaging device 22 images the object OBJ in order to generate the image data IMG_3D.
  • the tracking period does not have to be the same as the 2D imaging period.
  • the tracking period may be shorter than the 2D imaging period.
  • the tracking period may be longer than the 2D imaging period.
  • the tracking period may not be the same as the 3D imaging period.
  • the tracking period may be shorter than the 3D imaging period.
  • the tracking period may be longer than the 3D imaging period.
  • the tracking period is shorter than the 2D matching period in which the 2D matching section 3121 described above performs 2D matching processing. That is, the 2D matching period is longer than the tracking period.
  • the arithmetic unit 31 only needs to perform calculations focusing on the data portions of the image data IMG_2D and the three-dimensional position data WSD that are related to the characteristic locations of the object OBJ, while in the matching process, The calculation device 31 needs to perform calculations to search for the object OBJ from the entire image data IMG_2D and three-dimensional position data WSD.
  • the amount of calculation required to perform the tracking process is usually smaller than the amount of calculation required to perform the matching process.
  • the tracking period is usually shorter than the 2D matching period. Therefore, the tracking unit 3123 may perform tracking processing multiple times from the start to the end of one 2D matching cycle. That is, the tracking unit 3123 may perform the tracking process multiple times from when the 2D matching unit 3121 starts the 2D matching process until it completes the 2D matching process.
  • the 2D matching period is considered to be equivalent to the time from when the object OBJ is imaged by the imaging device 21 until the position and orientation information POI1 is generated based on the image data IMG_2D. good.
  • the tracking period is determined by the two image data IMG_2D#t1 and IMG_2D#t2 generated at the first time t1 and the second time t2 after the imaging device 21 captures an image of the object OBJ at the second time t2. It may be considered that this is equivalent to the time until the change amount information VI is generated based on the three-dimensional position data WSD generated using the image data IMG_3D generated at time t2. For example, in the example shown in FIG.
  • the tracking period is an example of a first time t1 and a second time t2 after the object OBJ is imaged by the imaging device 21 at a time ta, which is an example of a second time t2.
  • Change amount information VI is generated based on two image data IMG_2D generated at time ta-1 and time ta and three-dimensional position data WSD generated using image data IMG_3D#ta generated at time ta. It may be assumed that the time required for
  • the tracking period is based on two image data IMG_2D#t1 and IMG_2D#t2 generated at the first time t1 and the second time t2 after the object OBJ is imaged at the first time t1 by the imaging device 21 and the second time. It may be considered that this is equivalent to the time until the change amount information VI is generated based on the three-dimensional position data WSD generated using the image data IMG_3D generated at t2.
  • the tracking period is from when the object OBJ is imaged by the imaging device 21 at time ta-1, which is an example of the first time t1, to the first time t1 and the second time t2, which is an example.
  • the amount of change information VI is based on the two image data IMG_2D generated at time ta-1 and time ta, and the three-dimensional position data WSD generated using the image data IMG_3D#ta generated at time ta. It may be considered that it is equivalent to the time until it is generated. Even in this case, the tracking period may be shorter than the 2D matching period.
  • the tracking unit 3123 performs the tracking process using image data IMG_2D that is different from the image data IMG_2D used for the 2D matching process. It's okay.
  • the 2D matching unit 3121 performs 2D matching processing using image data IMG_2D generated at time ta, and the 2D matching processing using image data IMG_2D generated at time ta ends. After that, 2D matching processing is performed using the image data IMG_2D generated at time tb.
  • the tracking unit 3123 uses two pieces of image data IMG_2D generated at two times different from times ta and tb, and image data IMG_3D generated at a time different from times ta and tb. Tracking processing may be performed using the three-dimensional position data WSD.
  • the tracking unit 3123 stores two image data IMG_2D and By performing tracking processing using the three-dimensional position data WSD generated using the image data IMG_3D generated at time ta+2 (or any time in the period between time ta and time tb), the time The amount of change in the position and orientation of the object OBJ between ta+1 and ta+2 may be calculated.
  • the tracking unit 3123 may perform the tracking process using the image data IMG_2D used for the 2D matching process. For example, in the example shown in FIG. 18, the tracking unit 3123 uses two image data IMG_2D generated at times ta and ta+1, and three-dimensional position data WSD generated using image data IMG_3D generated at time ta+1. The amount of change in the position and orientation of the object OBJ between times ta and ta+1 may be calculated by performing tracking processing using the following.
  • the tracking period is shorter than the 3D matching period in which the 3D matching section 3122 described above performs 3D matching processing. That is, the 3D matching period is longer than the tracking period. Therefore, the tracking unit 3123 may perform tracking processing multiple times from the start to the end of one 3D matching cycle. That is, the tracking unit 3123 may perform the tracking process multiple times from when the 3D matching unit 3122 starts the 3D matching process until it completes the 3D matching process.
  • the position and orientation information POI2 is generated based on the three-dimensional position data WSD generated using the image data IMG_3D after the object OBJ is imaged by the imaging device 22. It may be considered that the time is equivalent to
  • the tracking unit 3123 uses three-dimensional position data generated using image data IMG_3D different from image data IMG_3D used for the 3D matching process. Tracking processing may be performed using WSD.
  • the 3D matching unit 3122 performs 3D matching processing using the three-dimensional position data WSD generated from the image data IMG_3D generated at time ta, and After the 3D matching process using the three-dimensional position data WSD generated from IMG_3D is completed, the 3D matching process is performed using the three-dimensional position data WSD generated from the image data IMG_3D generated at time tb.
  • the tracking unit 3123 uses three-dimensional position data WSD generated from image data IMG_3D generated at a time different from times ta and tb, and two dimensional position data WSD generated at two times different from times ta and tb. Tracking processing may be performed using the image data IMG_2D. For example, in the example shown in FIG. 18, the tracking unit 3123 combines three-dimensional position data WSD generated from image data IMG_3D generated at time ta+2 (or any time in the period between time ta and time tb).
  • the position of object OBJ between time ta+1 and ta+2 is determined. and the amount of change in posture may be calculated.
  • the tracking unit 3123 may perform the tracking process using the three-dimensional position data WSD generated using the image data IMG_3D used for the 3D matching process.
  • the tracking unit 3123 uses three-dimensional position data WSD generated from image data IMG_3D generated at time ta, three-dimensional position data WSD generated from image data IMG_3D generated at time ta, and three-dimensional position data WSD generated at time ta and time ta-1 before time ta.
  • the amount of change in the position and orientation of the object OBJ between times ta-1 and ta may be calculated by performing tracking processing using the image data IMG_2D#ta and IMG_2D#ta-1.
  • the tracking period does not need to be shorter than the 2D matching period.
  • the tracking period may be the same as the 2D matching period.
  • the tracking period may be longer than the 2D matching period.
  • the tracking period need not be shorter than the 3D matching period.
  • the tracking period may be the same as the 3D matching period.
  • the tracking period may be longer than the 3D matching period.
  • the tracking unit 3123 does not need to calculate the amount of change in the position and orientation of the object OBJ within the 2D imaging coordinate system between the first time t1 and the second time t2. For example, the tracking unit 3123 performs the above-described matching process on each of the two image data IMG_2D#t1 and IMG_2D#t2, thereby determining the object OBJ in the 2D imaging coordinate system at the first time t1 and the second time t2. The position and orientation may also be calculated. Note that since the position and orientation of the object OBJ are calculated at different times (for example, times t1 and t2), the process performed by the tracking unit 3123 here can also be called a tracking process.
  • the tracking unit 3123 performs matching processing on each of the two image data IMG_2D#t1 and IMG_2D#t2 using a smaller number of feature locations than the feature locations of the object OBJ used for the matching processing by the 2D matching unit 3121 described above. You may do so. In this case, the tracking unit 3123 can shorten the calculation time required for the matching process, and therefore can perform the tracking process in a shorter period than the matching process by the 2D matching unit 3121.
  • the coordinate conversion unit 3125 calculates the position of object OBJ in the global coordinate system based on the results of the matching process and the results of the tracking process. That is, the coordinate conversion unit 3125 converts the position of the object OBJ in the global coordinate system into position Tx (GL), position Ty (GL), position Tz (GL), orientation Rx (GL), orientation Ry (GL), and orientation Rz. Calculate (GL).
  • the coordinate conversion unit 3125 generates position/orientation information POI0 indicating position Tx (GL), position Ty (GL), position Tz (GL), orientation Rx (GL), orientation Ry (GL), and orientation Rz (GL). .
  • the coordinate transformation unit 3125 corrects the result of the matching process based on the result of the tracking process. Specifically, as shown in FIG. 10, the coordinate conversion unit 3125 converts the position Tx (3D), position Ty (3D), position Tz (3D), and posture Rx from the 3D matching unit 3122 as a result of the matching process. (3D), position/orientation information POI2 indicating orientation Ry (3D) and orientation Rz (3D) is acquired.
  • the coordinate conversion unit 3125 receives the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (2D), the amount of change ⁇ Rx (2D), and the amount of change ⁇ Ry from the tracking unit 3123 as a result of the tracking process.
  • (2D) and change amount information VI indicating the change amount ⁇ Rz (2D), change amount ⁇ Tx (3D), change amount ⁇ Ty (3D), change amount ⁇ Tz (3D), change amount ⁇ Rx (3D), change amount ⁇ Ry (3D).
  • the coordinate conversion unit 3125 first matches the coordinate system of the position and orientation information POI2 with the coordinate system of the change amount information VI.
  • the coordinate conversion unit 3125 converts the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (2D), the amount of change ⁇ Rx (2D), the amount of change ⁇ Ry (2D), and the amount of change ⁇ Rz (2D).
  • the coordinate conversion unit 3125 uses the coordinate system information described above (for example, the transformation matrix M (3D to 2D) for converting a position in the 3D imaging coordinate system to a position in the 2D imaging coordinate system) to perform 3D imaging.
  • the coordinate system and amount of change of the position and orientation information POI2 can be changed.
  • the coordinate system of the information VI may be made to match.
  • the coordinate conversion unit 3125 converts the 2D imaging coordinates using the coordinate system information described above (for example, the transformation matrix M (2D to 3D) for converting a position in the 2D imaging coordinate system to a position in the 3D imaging coordinate system).
  • the position and orientation information POI2 is The coordinate system and the coordinate system of the change amount information VI may be made to match.
  • the imaging devices 21 and 22 are configured such that the origin of the 2D imaging coordinate system is the origin of the 3D imaging coordinate system. They may be arranged as close to the origin as possible.
  • the imaging devices 21 and 22 may be arranged so that the X-axis (2D) of the 2D imaging coordinate system is as close as possible to the X-axis (3D) of the 3D imaging coordinate system.
  • the imaging devices 21 and 22 may be arranged so that the Y-axis (2D) of the 2D imaging coordinate system is as close as possible to the Y-axis (3D) of the 3D imaging coordinate system.
  • the imaging devices 21 and 22 may be arranged so that the Z-axis (2D) of the 2D imaging coordinate system is as close as possible to the Z-axis (3D) of the 3D imaging coordinate system.
  • the coordinate conversion unit 3125 may align both the coordinate system of the position/orientation information POI2 and the coordinate system of the change amount information VI with the global coordinate system.
  • the coordinate conversion unit 3125 converts the position and orientation of the object OBJ in the 3D imaging coordinate system using a transformation matrix M (3DtoGL) for converting the position in the 3D imaging coordinate system to the position in the global coordinate system.
  • the position and orientation information POI2 shown may be converted into position and orientation information POI2 that represents the position and orientation of the object OBJ in the global coordinate system.
  • the coordinate conversion unit 3125 uses a transformation matrix M (2DtoGL) for converting a position in the 2D imaging coordinate system to a position in the global coordinate system, and changes the position and orientation of the object OBJ in the 2D imaging coordinate system.
  • the change amount information VI indicating the amount may be converted into change amount information VI indicating the amount of change in the position and orientation of the object OBJ in the global coordinate system.
  • the transformation matrix M (2DtoGL) includes, for example, the product of transformation matrices that reflect changes in the position coordinates of the imaging device 21 due to the rotation of the link 121 around the axis defined by each joint 122 of the robot arm 12. You can stay there.
  • the transformation matrix M (3DtoGL) includes, for example, a product of transformation matrices that reflects changes in the position coordinates of the imaging device 22 due to the rotation of the link 121 around the axis defined by each joint 122 of the robot arm 12. It's okay to stay.
  • This transformation matrix may be a so-called rotation matrix, a rotation matrix including a translation component, or a matrix based on Euler angles. Note that an existing transformation method may be used for the coordinate transformation of the robot arm using the transformation matrix, so a detailed explanation thereof will be omitted.
  • the coordinate conversion unit 3125 converts the amount of change ⁇ Tx (3D), the amount of change ⁇ Ty (3D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and the amount of change ⁇ Rz( 3D), both the coordinate system of the position/orientation information POI2 and the coordinate system of the change amount information VI are 3D imaging coordinate systems.
  • the coordinate system of the position/orientation information POI2 and the coordinate system of the change amount information VI already match.
  • the coordinate conversion unit 3125 does not need to separately perform processing for matching the coordinate system of the position and orientation information POI2 with the coordinate system of the change amount information VI.
  • the coordinate conversion unit 3125 converts the amount of change ⁇ Tx (3D), the amount of change ⁇ Ty (3D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and the amount of change ⁇ Rz (3D).
  • the coordinate transformation unit 3125 uses the transformation matrix M (3DtoGL) to convert the position and orientation information POI2 indicating the position and orientation of the object OBJ in the 3D imaging coordinate system to the position and orientation information POI2 indicating the position and orientation of the object OBJ in the global coordinate system. It may be converted into position and orientation information POI2.
  • the coordinate transformation unit 3125 uses the transformation matrix M (3D to GL) to transform the change amount information VI indicating the amount of change in the position and orientation of the object OBJ in the 3D imaging coordinate system into the position and orientation of the object OBJ in the global coordinate system. It may be converted into change amount information VI indicating the amount of change.
  • the coordinate conversion unit 3125 converts the amount of change ⁇ Tx(GL), the amount of change ⁇ Ty(GL), the amount of change ⁇ Tz(GL), the amount of change ⁇ Rx(GL), the amount of change ⁇ Ry(GL), and the amount of change ⁇ Rz( GL), the coordinate system of the position and orientation information POI2 is the 3D imaging coordinate system, while the coordinate system of the change amount information VI is the global coordinate system.
  • the coordinate transformation unit 3125 uses the transformation matrix M (3DtoGL) to convert the position and orientation information POI2 indicating the position and orientation of the object OBJ in the 3D imaging coordinate system to the position and orientation information POI2 indicating the position and orientation of the object OBJ in the global coordinate system.
  • the coordinate system of the position and orientation information POI2 may be made to match the coordinate system of the amount of change information VI.
  • the coordinate conversion unit 3125 uses a transformation matrix M (GLto3D) for converting a position in the global coordinate system to a position in the 3D imaging coordinate system, and calculates the amount of change in the position and orientation of the object OBJ in the global coordinate system.
  • the coordinate system of position and orientation information POI2 and the coordinate system of amount of change information VI may be matched.
  • the coordinate conversion unit 3125 After matching the coordinate system of the position and orientation information POI2 with the coordinate system of the amount of change information VI, the coordinate conversion unit 3125 converts the position and orientation that is the result of the matching process based on the amount of change information VI that is the result of the tracking process. Correct information POI2.
  • the coordinate conversion unit 3125 converts the position Tx (3D) based on the change amount ⁇ Tx (3D). By performing the correction, the position Tx' (3D) of the object OBJ in the X-axis direction (3D) of the 3D imaging coordinate system may be calculated.
  • the coordinate conversion unit 3125 calculates the position Ty' (3D) of the object OBJ in the Y-axis direction (3D) of the 3D imaging coordinate system by correcting the position Ty (3D) based on the amount of change ⁇ Ty (3D). You can.
  • the coordinate conversion unit 3125 calculates the position Tz' (3D) of the object OBJ in the Z-axis direction (3D) of the 3D imaging coordinate system by correcting the position Tz (3D) based on the amount of change ⁇ Tz (3D). You can.
  • the coordinate conversion unit 3125 calculates the posture Rx' (3D) of the object OBJ around the X axis (3D) of the 3D imaging coordinate system by correcting the posture Rx (3D) based on the amount of change ⁇ Rx (3D). You can.
  • the coordinate conversion unit 3125 calculates the posture Ry' (3D) of the object OBJ around the Y axis (3D) of the 3D imaging coordinate system by correcting the posture Ry (3D) based on the amount of change ⁇ Ry (3D). You can.
  • the coordinate conversion unit 3125 calculates the posture Rz' (3D) of the object OBJ around the Z axis (3D) of the 3D imaging coordinate system by correcting the posture Rz (3D) based on the amount of change ⁇ Rz (3D). You can.
  • the corrected position and orientation information POI2 includes position Tx' (3D), position Ty' (3D), position Tz' (3D), attitude Rx' (3D), attitude Ry' (3D), and attitude Rz'. (3D) is the position/orientation information POI2'.
  • the coordinate conversion unit 3125 converts the position Tx (2D) based on the change amount ⁇ Tx (2D). By correcting, the position Tx' (2D) of the object OBJ in the X-axis direction (2D) of the 2D imaging coordinate system may be calculated.
  • the coordinate conversion unit 3125 calculates the position Ty' (2D) of the object OBJ in the Y-axis direction (2D) of the 2D imaging coordinate system by correcting the position Ty (2D) based on the amount of change ⁇ Ty (2D). You can.
  • the coordinate conversion unit 3125 calculates the position Tz' (2D) of the object OBJ in the Z-axis direction (2D) of the 2D imaging coordinate system by correcting the position Tz (2D) based on the amount of change ⁇ Tz (2D). You can.
  • the coordinate conversion unit 3125 calculates the posture Rx' (2D) of the object OBJ around the X axis (2D) of the 2D imaging coordinate system by correcting the posture Rx (2D) based on the amount of change ⁇ Rx (2D). You can.
  • the coordinate conversion unit 3125 calculates the posture Ry' (2D) of the object OBJ around the Y axis (2D) of the 2D imaging coordinate system by correcting the posture Ry (2D) based on the amount of change ⁇ Ry (2D). You can.
  • the coordinate conversion unit 3125 calculates the posture Rz' (2D) of the object OBJ around the Z axis (2D) of the 2D imaging coordinate system by correcting the posture Rz (2D) based on the amount of change ⁇ Rz (2D). You can.
  • the corrected position and orientation information POI2 includes position Tx' (2D), position Ty' (2D), position Tz' (2D), attitude Rx' (2D), attitude Ry' (2D), and attitude Rz'. (2D) is the position/orientation information POI2'.
  • the coordinate conversion unit 3125 corrects the position Tx (GL) based on the change amount ⁇ Tx (GL). By doing so, the position Tx' (GL) of the object OBJ in the X-axis direction (GL) of the global coordinate system may be calculated.
  • the coordinate conversion unit 3125 calculates the position Ty' (GL) of the object OBJ in the Y-axis direction (GL) of the global coordinate system by correcting the position Ty (GL) based on the amount of change ⁇ Ty (GL). Good too.
  • the coordinate conversion unit 3125 calculates the position Tz' (GL) of the object OBJ in the Z-axis direction (GL) of the global coordinate system by correcting the position Tz (GL) based on the amount of change ⁇ Tz (GL). Good too.
  • the coordinate conversion unit 3125 calculates the posture Rx' (GL) of the object OBJ around the X-axis (GL) of the global coordinate system by correcting the posture Rx (GL) based on the amount of change ⁇ Rx (GL). Good too.
  • the coordinate conversion unit 3125 calculates the posture Ry' (GL) of the object OBJ around the Y axis (GL) of the global coordinate system by correcting the posture Ry (GL) based on the amount of change ⁇ Ry (GL). Good too.
  • the coordinate conversion unit 3125 calculates the posture Rz' (GL) of the object OBJ around the Z-axis (GL) of the global coordinate system by correcting the posture Rz (GL) based on the amount of change ⁇ Rz (GL). Good too.
  • the corrected position and orientation information POI2 includes position Tx' (GL), position Ty' (GL), position Tz' (GL), attitude Rx' (GL), attitude Ry' (GL), and attitude Rz'.
  • the position/orientation information POI2' indicates (GL).
  • the process of correcting the position and orientation information POI2 based on the amount of change information VI may include the process of adding the amount of change information VI to the position and orientation information POI2.
  • the coordinate conversion unit 3125 may calculate the position Tx' (3D) by adding the amount of change ⁇ Tx (3D) to the position Tx (3D).
  • the coordinate conversion unit 3125 may calculate the position Ty' (3D) by adding the amount of change ⁇ Ty (3D) to the position Ty (3D).
  • the coordinate conversion unit 3125 may calculate the position Tz' (3D) by adding the amount of change ⁇ Tz (3D) to the position Tz (3D).
  • the coordinate conversion unit 3125 may calculate the posture Rx' (3D) by adding the amount of change ⁇ Rx (3D) to the posture Rx (3D).
  • the coordinate conversion unit 3125 may calculate the posture Ry' (3D) by adding the amount of change ⁇ Ry (3D) to the posture Ry (3D).
  • the coordinate conversion unit 3125 may calculate the orientation Rz' (3D) by adding the amount of change ⁇ Rz (3D) to the orientation Rz (3D).
  • the tracking period which is the period in which the amount of change ⁇ Tx (3D) is calculated, is shorter than the 3D matching period, which is the period in which the position Tx (3D) is calculated.
  • the tracking unit 3123 calculates the amount of change ⁇ Tx (3D) multiple times from when the position Tx (3D) is newly calculated until the next time the position Tx (3D) is calculated.
  • the coordinate conversion unit 3125 converts the data based on the newly calculated position Tx (3D) and the newly calculated amount of change ⁇ Tx (3D) , the position Tx' (3D) may be calculated.
  • the coordinate conversion unit 3125 starts matching processing for calculating the position Tx (3D) for the position Tx (3D) every time the amount of change ⁇ Tx (3D) is newly calculated.
  • the position Tx' (3D) may be calculated by adding the sum of the amount of change ⁇ Tx (3D) calculated from the time to the current time.
  • the coordinate conversion unit 3125 converts the position Tx'(3D) at that time (that is, the position to which the calculated amount of change ⁇ Tx(3D) has been added)
  • a new position Tx' (3D) may be calculated by adding the newly calculated amount of change ⁇ Tx (3D) to Tx (3D)). Therefore, each time the amount of change ⁇ Tx(3D) is calculated, the total sum of the amount of change ⁇ Tx(3D) added to the position Tx(3D) increases.
  • the update frequency of the position Tx' (3D) becomes higher compared to the comparative example in which the position Tx (3D) is used as is as the position Tx' (3D).
  • the update frequency of the position Tx' (3D) depends on the 3D matching cycle, which is the cycle for calculating the position Tx (3D). do.
  • the update frequency of the position Tx' (3D) is based on the amount of change ⁇ Tx (3D) It depends on the tracking period, which is the period for calculating .
  • the update frequency of the position Tx' (3D) becomes higher. Since the position of the object OBJ in the global coordinate system is calculated based on the position Tx' (3D) as described later, the coordinate conversion unit 3125 converts the object OBJ in the global coordinate system at the same period as the tracking period. The position of OBJ can be calculated. Therefore, the position and orientation calculation unit 312 can calculate the position of the object OBJ in the global coordinate system in a shorter period. In other words, the position and orientation calculation unit 312 can calculate the position of the object OBJ in the global coordinate system more frequently (in other words, at high speed).
  • the amount of change ⁇ Tx (3D) calculated by the tracking process may include an error corresponding to the amount of deviation from the actual position of the object OBJ.
  • the error included in the sum of the amount of change ⁇ Tx (3D) which continues to increase each time the amount of change ⁇ Tx (3D) is calculated, may also become larger.
  • the discrepancy between the position Tx' (3D) of the object OBJ calculated by the control device 3 and the actual position of the object OBJ may increase.
  • the accuracy of the position calculated by the matching process is higher than the accuracy of the position calculated by the tracking process.
  • the coordinate conversion unit 3125 changes the amount of change ⁇ Tx (3D) to be added to the position Tx (3D) in order to calculate the position Tx' (3D). You may also reset the range. Specifically, when the position Tx (3D) at one time is newly calculated, the coordinate conversion unit 3125 converts the position Tx (3D) calculated from the one time to the current time. The position Tx' (3D) may be calculated by adding the sum of the amount of change ⁇ Tx (3D). When the position Tx (3D) at one time is newly calculated, the coordinate conversion unit 3125 performs matching for calculating the position Tx (3D) at the one time with respect to the position Tx (3D).
  • the position Tx' (3D) may be calculated by adding the sum of the amount of change ⁇ Tx (3D) calculated from the time when the process was started to the current time. As a result, the position/orientation calculation unit 312 can accurately calculate the position Tx' (3D).
  • FIG. 19 An example of the flow of calculating the position Tx' (3D) is shown in FIG. 19.
  • the imaging devices 21 and 22 capture an image of object OBJ at time t#0_0 to generate image data IMG_2D#0_0 and IMG_3D#0_0, and at time t#0_0, the 2D matching unit 3121 performs 2D matching processing.
  • An example of starting the is shown below.
  • the 2D matching unit 3121 completes the generation of position and orientation information POI1#0_0 indicating the position and orientation of the object OBJ at time t#0_0 at time t#1_0 when the 2D matching cycle has passed from time t#0_0. .
  • the three-dimensional position data generating unit 311 generates three-dimensional position data WSD#0_0 from the image data IMG_3D#0_0. Perform processing. For example, at the same time as the 2D matching unit 3121 starts the 2D matching process using the image data IMG_2D#0_0, the 3D position data generation unit 311 starts the process of generating the 3D position data WSD#0_0 from the image data IMG_3D#0_0. You may start.
  • the three-dimensional position data generating unit 311 starts the process of generating the three-dimensional position data WSD#0_0 from the image data IMG_3D#0_0. You may start. As a result, three-dimensional position data WSD#0_0 indicating the three-dimensional position of object OBJ at time t#0_0 is generated. Note that the three-dimensional position data generation unit 311 completes the generation of the three-dimensional position data WSD#0_0 when the 2D matching unit 3121 completes the 2D matching process using the image data IMG_2D#0_0. Processing may be performed to generate three-dimensional position data WSD#0_0 from data IMG_3D#0_0.
  • the 3D matching unit 3122 starts 3D matching processing at time t#1_0 based on the three-dimensional position data WSD#0_0 and the position and orientation information POI1#0_0. That is, the 3D matching unit 3122 performs a position and orientation determination process to determine the initial position and initial orientation of the three-dimensional model WM3 based on the position and orientation information POI1#0_0 generated at time t#1_0. 3122 performs matching processing using the three-dimensional position data WSD#0_0 and the three-dimensional model WM3 placed at the determined initial position and in the determined initial posture.
  • the 3D matching unit 3122 performs the position and orientation determination process at time t#1_0 based on the position and orientation information POI1#0_0. Good too. Thereafter, after the initial position and initial orientation of the three-dimensional model WM3 are determined by the position and orientation determination process, the 3D matching unit 3122 uses the three-dimensional position data WSD#0_0 and the three-dimensional model placed in the initial position and the initial orientation. WM3 may be used to start the 3D matching process. Therefore, if the 3D matching process does not include the position and orientation determination process, the 3D matching unit 3122 starts the 3D matching process after considering the time required to complete the position and orientation determination process. Good too.
  • the 3D matching unit 3122 generates position and orientation information POI2#0_0 indicating the position and orientation of the object OBJ at time t#0_0 at time t#2_0 when the 3D matching cycle has passed from time t#1_0 (see FIG. In the example shown in , the calculation of the position Tx (3D) #0_0 in the X-axis direction (3D) is completed.
  • the position and orientation of the object OBJ at time t#2_0 when the 3D matching unit 3122 completes generation of the position and orientation information POI2#0_0 are the time t#0_0 indicated by the position and orientation information POI2#0_0 (that is, the position and orientation information This may be different from the position and orientation of the object OBJ at the time when the matching process for generating POI2#0_0 was started.
  • the amount of change in the position and orientation of the object from time t#0_0 to time t#2_0 is calculated by tracking processing.
  • the coordinate conversion unit 3125 converts the position and orientation information POI2#0_0 indicating the position and orientation of the object at time t#0_0 from time t#0_0 to time t#2_0 corresponding to the current time. Add the total sum of the calculated amounts of change. For example, the coordinate conversion unit 3125 adds the total amount of change calculated from time t#0_0 to time t#2_0, which corresponds to the current time, to position Tx(3D)#0_0. , the position Tx' (3D) of the object OBJ at time t#2_0 is calculated. In the example shown in FIG.
  • position Tx(3D)#0_0+amount of change ⁇ Tx(3D)#0_0+amount of change ⁇ Tx(3D)#0_1+...+amount of change ⁇ Tx(3D)#0_N+amount of change ⁇ Tx(3D) ) #1_0+amount of change ⁇ Tx(3D) #1_1+...+amount of change ⁇ Tx(3D) #1_N is used as the position Tx'(3D).
  • the coordinate conversion unit 3125 further adds the newly calculated amount of change ⁇ Tx(3D) #2_0 to the current position Tx'(3D).
  • the coordinate conversion unit 3125 further adds the newly calculated amount of change ⁇ Tx(3D) #2_1 to the current position Tx'(3D).
  • position Tx (3D) #0_0 + amount of change ⁇ Tx (3D) #0_0 + amount of change ⁇ Tx (3D) #0_1+...+ amount of change ⁇ Tx (3D) #0_N + amount of change ⁇ Tx (3D) #1_0+change amount ⁇ Tx(3D) #1_1+...+change amount ⁇ Tx(3D) #1_N+change amount ⁇ Tx(3D) #2_0+change amount ⁇ Tx(3D) #2_1 is used as the position Tx'(3D).
  • the 2D matching unit 3121 completes generation of position and orientation information POI1#0_0 indicating the position and orientation of object OBJ at time t#0_0, and then at time t#1_0, the imaging device 21 A new 2D matching process using image data IMG_2D#1_0 generated by capturing an image of object OBJ is started. In this case, the 2D matching unit 3121 completes the generation of position and orientation information POI1#1_0 indicating the position and orientation of the object OBJ at time t#1_0 at time t#2_0 when the 2D matching cycle has elapsed from time t#1_0. .
  • the three-dimensional position data generating unit 311 generates three-dimensional position data WSD#1_0 from the image data IMG_3D#1_0. Perform processing. For example, at the same time that the 2D matching unit 3121 starts the 2D matching process using the image data IMG_2D#1_0, the three-dimensional position data generating unit 311 starts the process of generating the three-dimensional position data WSD#1_0 from the image data IMG_3D#1_0. You may start.
  • the three-dimensional position data generating unit 311 starts the process of generating the three-dimensional position data WSD#1_0 from the image data IMG_3D#1_0. You may start. As a result, three-dimensional position data WSD#1_0 indicating the three-dimensional position of object OBJ at time t#1_0 is generated. Note that the three-dimensional position data generation unit 311 completes the generation of the three-dimensional position data WSD#1_0 when the 2D matching unit 3121 completes the 2D matching process using the image data IMG_2D#1_0. Processing may be performed to generate three-dimensional position data WSD#1_0 from data IMG_3D#1_0.
  • the 3D matching unit 3122 starts 3D matching processing at time t#2_0 based on the three-dimensional position data WSD#1_0 and the position and orientation information POI1#1_0.
  • the 3D matching unit 3122 generates position and orientation information POI2#1_0 indicating the position and orientation of the object OBJ at time t#1_0 at time t#3_0 when the 3D matching cycle has passed from time t#2_0 (see FIG.
  • the calculation of the position Tx (3D) #1_0 in the X-axis direction (3D) is completed. Since the position and orientation information POI2 is newly generated in this way (in the example shown in FIG.
  • the coordinate conversion unit 3125 calculates the position Tx' (3D). Therefore, the range of the change amount ⁇ Tx (3D) to be added to the position Tx (3D) is newly set. Specifically, since the position Tx(3D) #1_0 at time t#1_0 is newly calculated, the coordinate conversion unit 3125 changes the amount of change ⁇ Tx(3D) calculated from time t#1_0 to the current time. The range of is newly set as the range of the amount of change ⁇ Tx(3D) to be added to the position Tx(3D) #1_0.
  • position Tx(3D)#1_0+amount of change ⁇ Tx(3D)#1_0+amount of change ⁇ Tx(3D)#1_1+...+amount of change ⁇ Tx(3D)#1_N+amount of change ⁇ Tx(3D) ) #2_0+amount of change ⁇ Tx(3D) #2_1+...+amount of change ⁇ Tx(3D) #2_N is used as the position Tx'(3D).
  • the coordinate conversion unit 3125 calculates the position Ty' (3D), the position Tz' (3D), and the orientation Rx, as in the case of calculating the position Tx' (3D). '(3D), attitude Ry' (3D), and attitude Rz' (3D). In other words, the coordinate conversion unit 3125 converts the coordinates calculated from the time when the matching process for calculating the position Ty (3D) at one time to the current time to calculate the position Ty (3D) at one time.
  • the position Ty' (3D) may be calculated by adding the sum of the changes ⁇ Ty (3D).
  • the coordinate conversion unit 3125 adds the sum of the amount of change ⁇ Ty (3D) calculated from one time to the current time to the position Ty (3D) at one time, thereby converting the position Ty' (3D). may be calculated.
  • the coordinate conversion unit 3125 converts the position Tz (3D) at one time to the position Tz (3D) calculated from the time when the matching process for calculating the position Tz (3D) at one time is started to the current time.
  • the position Tz' (3D) may be calculated by adding the sum of the changes ⁇ Tz (3D).
  • the coordinate conversion unit 3125 adds the sum of the amount of change ⁇ Tz (3D) calculated from one time to the current time to the position Tz (3D) at one time, thereby converting the position Tz' (3D).
  • the coordinate conversion unit 3125 converts the coordinates calculated from the time when the matching process for calculating the position Rx (3D) at one time to the current time to calculate the position Rx (3D) at one time.
  • the position Rx' (3D) may be calculated by adding the sum of the amounts of change ⁇ Rx (3D).
  • the coordinate conversion unit 3125 adds the sum of the amount of change ⁇ Rx (3D) calculated from one time to the current time to the position Rx (3D) at one time, thereby converting the position Rx' (3D). may be calculated.
  • the coordinate conversion unit 3125 converts the coordinates calculated from the time when the matching process for calculating the position Ry (3D) at one time to the current time to calculate the position Ry (3D) at one time.
  • the position Ry' (3D) may be calculated by adding the sum of the amounts of change ⁇ Ry (3D).
  • the coordinate conversion unit 3125 adds the sum of the amount of change ⁇ Ry (3D) calculated from one time to the current time to the position Ry (3D) at one time, thereby converting the position Ry' (3D). may be calculated.
  • the coordinate conversion unit 3125 converts the coordinates calculated from the time when the matching process for calculating the position Rz (3D) at one time to the current time to calculate the position Rz (3D) at one time.
  • the position Rz' (3D) may be calculated by adding the sum of the changes ⁇ Rz (3D).
  • the coordinate conversion unit 3125 adds the sum of the amount of change ⁇ Rz (3D) calculated from one time to the current time to the position Rz (3D) at one time, thereby converting the position Rz' (3D). may be calculated.
  • the coordinate conversion unit 3125 converts the coordinates into position Tx' (3D), position Ty' (3D), position Tz' (3D), attitude Rx' (3D), attitude Ry' (3D), and attitude Rz' (3D). Based on this, the position Tx (GL), position Ty (GL), position Tz (GL), orientation Rx (GL), orientation Ry (GL), and orientation Rz (GL) of object OBJ in the global coordinate system are calculated. In other words, the coordinate conversion unit 3125 converts the 6DOF position of the object OBJ in the 3D imaging coordinate system (or, in some cases, the 6DOF position of the object OBJ in the 2D imaging coordinate system) in the global coordinate system. Calculate the 6DOF position of object OBJ. The coordinate conversion unit 3125 generates position/orientation information POI0 indicating position Tx (GL), position Ty (GL), position Tz (GL), orientation Rx (GL), orientation Ry (GL), and orientation Rz (GL). .
  • the coordinate conversion unit 3125 uses a transformation matrix M (3DtoGL) for converting a position within the 3D imaging coordinate system (or 3D imaging coordinate system) to a position within the global coordinate system.
  • M 3DtoGL
  • the 6DOF position of the object OBJ in (or the 6DOF position of the object OBJ in the 2D imaging coordinate system) may be converted to the 6DOF position of the object OBJ in the global coordinate system.
  • the position and orientation information POI2' may be used as the position and orientation information POI0.
  • the coordinate transformation unit 3125 does not necessarily need to perform the coordinate transformation based on the transformation matrix M (3DtoGL) described above.
  • the control device 3 corrects the result of the matching process based on the result of the tracking process, thereby at least changing the position and orientation of the object OBJ in the global coordinate system.
  • One side is being calculated. Therefore, as will be explained in detail below, compared to the case where at least one of the position and orientation of the object OBJ in the global coordinate system is calculated based on the matching processing results without using the tracking processing results, the control The device 3 can calculate at least one of the position and orientation of the object OBJ in the global coordinate system in a shorter period (that is, more frequently).
  • the position and orientation calculation unit 312 can calculate at least one of the position and orientation of the object OBJ in the global coordinate system at higher speed.
  • the control device 3 can generate (that is, update) the robot control signal more frequently and output it to the robot 1. It can also be said that the control device 3 can generate (update) robot control signals at higher speed and output them to the robot 1.
  • control device 3 may generate a robot control signal every time at least one of the position and orientation of the object OBJ is calculated.
  • the control device 3 can generate the robot control signal at the same cycle as the cycle at which at least one of the position and orientation of the object OBJ is calculated (for example, the tracking cycle described above).
  • the control device 3 can output the robot control signal to the robot 1 at the same cycle as the cycle at which at least one of the position and orientation of the object OBJ is calculated (for example, the tracking cycle described above).
  • the control device 3 calculates the position and orientation of the object OBJ.
  • the robot control signal is generated and output at the same cycle as the 2D matching cycle or the 3D matching cycle (typically, a cycle shorter than the tracking cycle), at least one of which is the calculated cycle. Therefore, in this embodiment, compared to this case, the control device 3 can generate (that is, update) the robot control signal more frequently (at a higher speed) and output it to the robot 1. .
  • the control device 3 may generate and output the robot control signal at a period different from the period at which the position of the object OBJ is calculated (for example, at a period different from the tracking period described above).
  • Such an effect is particularly beneficial when at least one of the robot 1 and the object OBJ is moving (in particular, each of the imaging devices 21 and 22 and the object OBJ are displaced relative to each other). .
  • This is because the lower the frequency of calculating the position of the object OBJ in a situation where at least one of the robot 1 and the object OBJ is moving, the more the position of the object OBJ calculated by the control device 3 and the actual position of the object OBJ become different. This is because the period during which the degree of deviation is large becomes long.
  • the period during which the degree of deviation is large becomes long.
  • the control device 3 since the control device 3 calculates the position of the object OBJ in the global coordinate system more frequently (at higher speed), the position of the object OBJ calculated by the control device 3 and the actual position of the object OBJ are The degree of deviation from this becomes relatively small.
  • the control device 3 calculates the attitude of the object OBJ in the global coordinate system more frequently (at a higher speed), the attitude of the object OBJ calculated by the control device 3 is different from the actual attitude of the object OBJ.
  • the degree of deviation becomes relatively small. Therefore, by calculating at least one of the position and orientation of the object OBJ with high frequency, the control device 3 can substantially calculate at least one of the position and orientation of the object OBJ with high accuracy.
  • the control device 3 appropriately sends robot control signals for controlling the robot 1 so as to perform desired processing on the object OBJ, whose position and/or orientation are calculated with high precision and high frequency. It can be generated (that is, updated) and output to the robot 1. As a result, the possibility that the movement of the robot 1 will be different from the intended movement is reduced.
  • the control device 3 adjusts the calculation period of at least one of the position and orientation of the object OBJ.
  • the movement of the robot 1 that follows the moving workpiece W can be controlled (that is, a robot control signal can be generated and output). Therefore, even if the movement of the robot 1 deviates from the desired movement for following the moving workpiece W, the movement of the robot 1 will increase in accordance with the calculation cycle of at least one of the position and orientation of the workpiece W. Fixed to frequency (fast).
  • the movement of the robot 1 is modified frequently (at high speed) in accordance with the irregular movement of the workpiece W.
  • the movement of the robot 1 is frequently (highly) corrected in accordance with the high-speed movement of the workpiece W. Therefore, the robot 1 can follow the moving workpiece W while reducing unnecessary movements. As a result, the robot 1 can hold the moving workpiece W while reducing unnecessary movements.
  • the robot 1 is not limited to the examples shown in FIGS. 5A to 5D, and can move relative to the object OBJ while reducing unnecessary movements.
  • the robot 1 moves accurately and at high speed with respect to the object OBJ (for example, follows the object OBJ accurately and at high speed) without losing sight of the object OBJ, and the end effector 13 moves toward the object OBJ with the end effector 13. Desired processing can be performed.
  • the imaging devices 21 and 22 are The object OBJ may be imaged while at least one of the objects OBJ is moving. That is, the imaging devices 21 and 22 may image the object OBJ at a timing (in other words, at a time) when the imaging devices 21 and 22 and the object OBJ are moving relative to each other.
  • the imaging device 21 may generate the image data IMG_2D used in the above-described 2D matching process by capturing an image of the object OBJ at a time when the imaging device 21 and the object OBJ are moving relative to each other.
  • the imaging device 22 may generate the image data IMG_3D used in the above-described 3D matching process by capturing an image of the object OBJ at a time when the imaging device 22 and the object OBJ are moving relative to each other.
  • the imaging device 21 images the object OBJ at a first time t1 and a second time t2, which are times when the imaging device 21 and the object OBJ are moving relative to each other, thereby creating an image used in the tracking process described above.
  • Data IMG_2D may also be generated.
  • the imaging device 21 may generate the image data IMG_3D used in the tracking process described above by imaging the object OBJ at the third time t3, which is the time when the imaging device 21 and the object OBJ are moving relative to each other. good.
  • the control device 3 determines at least the initial position and initial orientation of the three-dimensional model WM3 used for performing the 3D matching process, based on the result of the 2D matching process (that is, the position and orientation information POI1). One can be decided. As a result, the control device 3 can determine the initial position and initial orientation of the three-dimensional model WM3 such that the degree of deviation from the actual position and orientation of the object OBJ is less than or equal to the allowable amount.
  • the position and orientation information POI1 obtained by the 2D matching process there is a lower possibility that information regarding at least one of the position and orientation of the object OBJ is missing compared to the position and orientation information POI2 obtained by the 3D matching process.
  • the 2D matching process detects the object OBJ using the image itself indicated by the image data IMG_2D, and therefore there is a low possibility that information regarding at least one of the position and orientation of the object OBJ will be missing. Therefore, the possibility that object OBJ will be successfully detected by the 2D matching process increases. In other words, the 2D matching process increases the possibility that an object OBJ that matches the two-dimensional model WM2 will be detected. In other words, it is less likely that an object different from the object OBJ corresponding to the two-dimensional model WM2 will be detected by the 2D matching process.
  • the 3D matching process is performed using three-dimensional position data WSD in which part of the information of the object OBJ may be missing (for example, point cloud data in which the point cloud corresponding to part of the object OBJ may be missing). ) is used to detect the object OBJ. Therefore, the lack of data in the three-dimensional position data WSD (for example, missing point group) has a large effect on the success of detecting the object OBJ by the 3D matching process. Therefore, the possibility that object OBJ will be successfully detected by 3D matching processing is lower than the possibility that object OBJ will be successfully detected by 2D matching processing.
  • the possibility that an object OBJ that matches the three-dimensional model WM3 will be detected by the 3D matching process is lower than the possibility that the object OBJ that matches the two-dimensional model WM2 will be detected by the 2D matching process.
  • the possibility that an object different from the object OBJ corresponding to the three-dimensional model WM3 will be detected by the 3D matching process is lower than the possibility that an object different from the object OBJ corresponding to the two-dimensional model WM2 will be detected by the 2D matching process. higher than sex.
  • the initial position of the three-dimensional model WM3 and the initial At least one of the postures can be determined.
  • the control device 3 detects a three-dimensional image in which the degree of deviation from the actual position and orientation of the object OBJ that should be detected by the 2D matching process (and should also be naturally detected by the 3D matching process) is less than the allowable amount.
  • the initial position and initial attitude of model WM3 can be determined reliably.
  • the control device 3 determines at least one of the position and orientation of the object OBJ corresponding to the three-dimensional model WM3 with higher precision than when at least one of the initial position and initial orientation of the three-dimensional model WM3 is not determined. It is possible to generate position and orientation information POI2 shown in FIG.
  • the control device 3 determines at least the initial position and initial orientation of the three-dimensional model WM3 used for performing the 3D matching process, based on the result of the 2D matching process (that is, the position and orientation information POI1).
  • the position and orientation information POI1 obtained by the 2D matching process indicates the position of the object OBJ in the X-axis direction (2D) and the Y-axis direction (2D) with high accuracy due to the characteristics of the 2D matching process using two-dimensional images.
  • the position and orientation information POI1 indicates the orientation of the object OBJ around the Z-axis (2D) with high accuracy due to the characteristics of 2D matching processing using two-dimensional images.
  • the position and orientation information POI1 is not as accurate as the position of the object OBJ in the X-axis direction (2D) and the Y-axis direction (2D), and the posture of the object OBJ around the Z-axis (2D), but it is accurate in the Y-axis direction.
  • the position of the object OBJ in (2D) and the attitude of the object OBJ around the X axis (2D) and the Y axis (2D) are also shown with a certain degree of accuracy.
  • the control device 3 performs 2D matching processing using the image data IMG_2D and 3D matching processing using the three-dimensional position data WSD in order to calculate at least one of the position and orientation of the object OBJ with higher accuracy. Combine with matching processing. Specifically, the control device 3 determines at least one of the initial position and initial orientation of the three-dimensional model WM3 based on the position and orientation information POI1. Therefore, in the 3D imaging coordinate system in which the 3D matching process is performed, the initial position of the three-dimensional model WM3 is unlikely to be significantly different from the actual position of the object OBJ.
  • the initial position of the three-dimensional model WM3 is likely to be close to the actual position of the object OBJ in each of the X-axis direction (3D) and the Y-axis direction (3D). Furthermore, in the Z-axis direction (3D), the initial position of the three-dimensional model WM3 is unlikely to be significantly different from the actual position of the object OBJ. Similarly, in the 3D imaging coordinate system, the initial orientation of the three-dimensional model WM3 is unlikely to be significantly different from the actual orientation of the object OBJ. Note that the initial posture of the three-dimensional model WM3 is likely to be close to the actual posture of the object OBJ, especially around the Z-axis (3D).
  • the initial posture of the three-dimensional model WM3 is unlikely to be significantly different from the actual posture of the object OBJ around the X-axis (3D) and the Y-axis (3D).
  • the 3D matching unit 3122 performs matching processing.
  • the 3D matching unit 3122 can accurately detect the object OBJ through the 3D matching process, compared to the case where the initial position and initial orientation of the three-dimensional model WM3 are not determined based on the position and orientation information POI1. This is because the three-dimensional model WM3 has already been placed near the object OBJ indicated by the three-dimensional position data WSD.
  • the 3D matching unit 3122 uses the three-dimensional position data WSD that indicates the position of the object OBJ in the Z-axis direction (3D) with high accuracy to determine the characteristic points of the three-dimensional model WM3 in the Z-axis direction (3D). It is possible to accurately approach the characteristic location of the object OBJ indicated by the three-dimensional position data WSD. Therefore, compared to the case where the three-dimensional position data WSD is not used, the 3D matching unit 3122 can generate position and orientation information POI2 that indicates at least one of the position and orientation of the object OBJ with higher accuracy. In particular, the 3D matching unit 3122 can generate position and orientation information POI2 that indicates the position Tz (3D), orientation Rx (3D), and orientation Ry (3D) with higher accuracy.
  • the position and orientation In order to improve the accuracy of information POI2, the 2D imaging time when the imaging device 21 images the object OBJ for the matching process and the 3D imaging time when the imaging device 22 images the object OBJ for the matching process are the same. It may be set to the time. Specifically, if the 2D imaging time and the 3D imaging time are different times, the 3D matching unit 3122 uses position and orientation information POI1 indicating at least one of the position and orientation of the object OBJ at the 2D imaging time.
  • At least one of the initial position and initial orientation of the three-dimensional model WM3 for calculating the position of the object OBJ at a 3D imaging time different from the 2D imaging time is determined.
  • the position of the object OBJ at the 2D imaging time is likely to be different from the position of the object OBJ at the 3D imaging time. Therefore, if the 2D imaging time and the 3D imaging time are different times, the initial position of the 3D model WM3 determined based on the position of the object OBJ at the 2D imaging time is the same as that of the object OBJ at the 3D imaging time. There is a high possibility that the location will be significantly different from the actual location.
  • the initial attitude of the three-dimensional model WM3 determined based on the attitude of the object OBJ at the 2D imaging time is likely to be significantly different from the actual attitude of the object OBJ at the 3D imaging time.
  • the accuracy of the position and orientation information POI2 generated using the three-dimensional model WM3 decreases compared to the case where the initial position and initial orientation of the three-dimensional model WM3 are the same as or close to the actual position and orientation of the object OBJ. there's a possibility that.
  • the position and orientation of the object OBJ at the 2D imaging time are the same as the position and orientation of the object OBJ at the 3D imaging time.
  • control device 3 can prevent the accuracy of the position and orientation information POI2 from decreasing by controlling the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same time.
  • the state where "the 2D imaging time and the 3D imaging time are the same time” means “the 2D imaging time and the 3D imaging time are not exactly the same time, but the time between the 2D imaging time and the 3D imaging time”
  • the 2D imaging time and the 3D imaging time can be considered to be substantially the same time because the deviation is smaller than the allowable upper limit value.
  • the allowable upper limit value of the time difference between the 2D imaging time and the 3D imaging time may be set to an appropriate value that can realize a state in which the accuracy of the position and orientation information POI2 is the desired accuracy.
  • the allowable upper limit value of the temporal deviation between the 2D imaging time and the 3D imaging time is the permissible amount of decrease in the accuracy of the position and orientation information POI2 due to the temporal deviation between the 2D imaging time and the 3D imaging time. It may be set to an appropriate value that can achieve a state within. For example, when a time difference between the 2D imaging time and the 3D imaging time causes an error in the calculation result of at least one of the position and orientation of the object OBJ (described later) (in other words, when the accuracy of the position and orientation information POI2 decreases) There is.
  • the allowable upper limit value of the time difference between the 2D imaging time and the 3D imaging time may be set to an appropriate value based on the accuracy of the position and orientation information POI2.
  • the 2D imaging time and the 3D imaging time may be different times. Even if it is, the position of the object OBJ at the 2D imaging time is the same as the position of the object OBJ at the 3D imaging time. Therefore, it is unlikely that the initial position and initial orientation of the three-dimensional model WM3 will be significantly different from the actual position and orientation of the object OBJ. Therefore, the control device 3 does not have to control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same time.
  • control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are different times. Even in this case, the accuracy of the position and orientation information POI2 is hardly reduced due to the time difference between the 2D imaging time and the 3D imaging time.
  • the control device 3 can perform tracking processing using the three-dimensional position data WSD in addition to the two image data IMG_2D#t1 and IMG_2D#t2.
  • the control device 3 performs the tracking process based on the position information PI3 indicating the position of the characteristic part of the object OBJ in the Z-axis direction (2D) or the Z-axis direction (3D) calculated from the three-dimensional position data WSD.
  • the accuracy of the position of the object OBJ in the Z-axis direction (2D) calculated from the image data IMG_2D representing a two-dimensional image is not necessarily high.
  • the three-dimensional position data WSD generated from the image data IMG_3D indicates the position of the object OBJ in the Z-axis direction (3D) with high accuracy. Therefore, when the control device 3 performs the tracking process using the position information PI3 as described in the present embodiment, the amount of change in at least one of the position and orientation of the object OBJ can be calculated with higher accuracy. .
  • the tracking unit 3123 uses change amount information VI indicating the change amount ⁇ Tz (3D), the change amount ⁇ Rx (3D), and the change amount ⁇ Ry (3D) with higher accuracy, or the change amount ⁇ Tz (2D), the change amount ⁇ Rx (2D) and change amount information VI indicating the change amount ⁇ Ry(2D) with higher accuracy can be generated.
  • the control device 3 can calculate at least one of the position and orientation of the object OBJ with high accuracy.
  • the control device 3 recognizes that the process of calculating the amount of change information VI indicating the amount of change in at least one of the position and orientation of the object OBJ is substantially equivalent to the process of solving the PnP problem. We are using. Therefore, the control device 3 can generate change amount information VI indicating the amount of change in at least one of the position and orientation of the object OBJ by solving the PnP problem using a mathematical calculation method. For this reason, the control device 3 generates the amount of change information VI that indicates the amount of change in at least one of the position and orientation of the object OBJ with high accuracy compared to the case where the amount of change information VI is generated without using the PnP problem. can be generated.
  • a second time t2 at which the imaging device 21 images the object OBJ for tracking processing and a third time t3 at which the imaging device 22 images the object OBJ for the tracking processing are determined.
  • the tracking unit 3123 transfers the position information PI3 indicating the position of the object OBJ at the third time t3 to the third time t3.
  • the position information PI5 indicating the position of the object OBJ at the second time t2, which is different from the second time t2.
  • the position of the object OBJ at the second time t2 is likely to be different from the position of the object OBJ at the third time t3. Therefore, when the second time t2 and the third time t3 are different times, the accuracy of the position information PI5 to which the position information PI3 is reflected is higher than the accuracy of the position information PI5 to which the position information PI3 is not reflected. may also become worse.
  • the position information PI5 reflecting the position information PI3 may indicate a position different from the original position of the object.
  • the accuracy of the change amount information VI may also deteriorate.
  • the control device 3 can prevent the accuracy of the variation information VI from decreasing by controlling the imaging devices 21 and 22 so that the second time t2 and the third time t3 are the same time.
  • the state where "the second time t2 and the third time t3 are the same time” means “the second time t2 and the third time t3 are not completely the same time, but the second time t2 and the third time t3 are the same time”. It has already been shown that the second time t2 and the third time t3 can be considered to be substantially the same time because the time difference with the third time t3 is smaller than the allowable upper limit value. As mentioned above. In this case, the allowable upper limit value of the time difference between the second time t2 and the third time t3 may be set to an appropriate value that can realize a state in which the accuracy of the amount of change information VI becomes a desired accuracy. .
  • the allowable upper limit of the time difference between the second time t2 and the third time t3 is the decrease in accuracy of the change amount information VI due to the time difference between the second time t2 and the third time t3. It may be set to an appropriate value that allows the amount to fall within an allowable range.
  • the allowable upper limit value may be an allowable upper limit value based on a control error of the robot arm 12. For example, an error may occur in the change amount information VI due to a time difference between the second time t2 and the third time t3. In this case, a control error of the robot arm 12 may occur due to an error occurring in the change amount information VI.
  • a control error of the robot arm 12 becomes a movement error of the end effector 13, and the end effector 13 may not be able to appropriately perform a predetermined process on the object OBJ.
  • the allowable upper limit value can also be said to be the allowable upper limit value of the movement error of the end effector 13 by the robot arm 12.
  • the second time t2 and the third time t3 are substantially different from each other. It may be assumed that they are at the same time.
  • the synchronization error in the imaging processing of the imaging devices 21 and 22 may be a synchronization control error in the imaging processing of the imaging devices 21 and 22 by the control device 3.
  • the second time t2 and the third time t3 are Even if the times are different, the position of the object OBJ at the second time t2 is the same as the position of the object OBJ at the third time t3. Therefore, the accuracy of the position information PI5 to which the position information PI3 is reflected does not become worse than the accuracy of the position information PI5 to which the position information PI3 is not reflected. Therefore, the control device 3 does not have to control the imaging devices 21 and 22 so that the second time t2 and the third time t3 are the same time.
  • control device 3 may control the imaging devices 21 and 22 so that the second time t2 and the third time t3 are different times. Even in this case, the accuracy of the variation information VI is hardly reduced due to the time difference between the second time t2 and the third time t3.
  • the control device 3 takes into consideration that the amount of change ⁇ Tx (3D) calculated by the tracking process includes an error, and when the position Tx (3D) is newly calculated, , the range of the amount of change ⁇ Tx (3D) to be added to the position Tx (3D) in order to calculate the position Tx' (3D) is reset. As a result, the control device 3 is able to calculate the position Tx' (3D) with higher accuracy compared to the case where the range of the amount of change ⁇ Tx (3D) added to the position Tx (3D) is not reset. Can be done.
  • the control device 3 controls the position Ty' (3D), the position Tz' (3D), the attitude Rx' (3D), the attitude Ry' (3D), the attitude Rz' (3D), and the position Tx' (2D), position Ty' (2D), position Tz' (2D), attitude Rx' (2D), attitude Ry' (2D), and attitude Rz' (2D) can be calculated with higher precision.
  • the control device 3 can calculate at least one of the position and orientation of the object OBJ with even higher accuracy.
  • the control device 3 acquires the image data IMG_2D from the imaging device 21.
  • image data IMG_3D can be acquired from the imaging device 22. Therefore, an imaging device 21 for generating image data IMG_2D and an imaging device 22 for generating image data IMG_3D are not prepared separately (for example, a single imaging device for generating image data IMG_2D and IMG_3D is not prepared separately).
  • the image data IMG_2D and IMG_3D are acquired more frequently than in the case where the image data IMG_2D and IMG_3D are prepared.
  • the control device 3 appropriately generates (that is, updates) a control signal for controlling the robot 1 to perform desired processing on the object OBJ whose position is frequently calculated. It can also be output to the robot 1.
  • FIG. 20 is a block diagram showing the configuration of the control device 3a in the first modification.
  • control device 3a in the first modification differs from the control device 3 described above in that the arithmetic device 31 includes an object determining section 314a as a logical processing block. .
  • Other features of the control device 3a may be the same as other features of the control device 3.
  • the object determination unit 314a causes the end effector 13 to perform predetermined processing on the object OBJ reflected in the image indicated by the image data IMG_2D, based on the image data IMG_2D generated by the imaging device 21 imaging the object OBJ.
  • the object is determined as the object OBJ_target.
  • the position/orientation calculation unit 312 performs 2D matching processing (particularly object detection processing) to find the template image indicated by the two-dimensional model data IMG_2M within the image indicated by the image data IMG_2D. Detect the object OBJ indicated by .
  • the target determining unit 314a determines whether or not to determine the object OBJ detected by the 2D matching process as the target object OBJ_target based on the image data IMG_2D.
  • the position and orientation calculation unit 312 determines the position of the object OBJ. and position/orientation information POI0 indicating the orientation.
  • the position and orientation calculation unit 312 It is not necessary to generate the position and orientation information POI0 indicating the position and orientation of the object OBJ. This is because the end effector 13 does not perform any predetermined processing on the object OBJ, so there is little need to calculate the position and orientation of the object OBJ.
  • the target determination unit 314a may determine the object OBJ that was not determined as the target object ONJ_target at one time point as the target object OBJ_target at another time point different from the one time point. For example, when the robot 1 performs the release process to place the plurality of workpieces W one by one on the mounting device T as described above, the target determination unit 314a sequentially sets the plurality of workpieces W to the target object OBJ_target. It may be determined (in this case, selected) as For example, the target determination unit 314a may determine one workpiece W as the target object OBJ_target at one point in time, while not determining another workpiece W different from the one workpiece W as the target object OBJ_target. Thereafter, after one workpiece W is placed on the mounting device T, the target determination unit 314a may determine another workpiece W as the target object OBJ_target.
  • FIG. 21 is a flowchart showing the flow of robot control processing in the first modification. Note that the same processes as those performed in the robot control process shown in FIG. 4 are given the same reference numerals, and detailed explanation thereof will be omitted.
  • the control device 3 uses the communication device 33 to obtain image data IMG_3D from the imaging device 22 (step S1). However, in the first modification, the control device 3 does not need to acquire the image data IMG_3D from the imaging device 22 at the stage of step S1.
  • the position and orientation calculation unit 312 performs the above-mentioned 2D matching process based on the image data IMG_2D acquired in step S1 (step S31a).
  • the 2D matching unit 3121 detects the object OBJ indicated by the template image indicated by the two-dimensional model data IMG_2M within the image indicated by the image data IMG_2D.
  • the 2D matching unit 3121 moves from one template image in the projection direction so that the characteristic points of the object OBJ reflected in the template image are brought closer to the characteristic points of the object OBJ reflected in the image indicated by the image data IMG_2D.
  • the object OBJ is detected by changing to another template image with a different imaging direction, and by translating, enlarging, reducing, and/or rotating the object OBJ reflected in the template image.
  • the 2D matching unit 3121 calculates the position and orientation of the detected object OBJ. That is, the 2D matching unit 3121 generates position and orientation information POI1 indicating the position and orientation of the detected object OBJ.
  • the 2D matching process in the first modification may be the same as the 2D matching process described above.
  • the 2D matching unit 3121 performs a process of bringing the characteristic points of the object OBJ reflected in the template image closer to the characteristic points of the object OBJ reflected in the image indicated by the image data IMG_2D.
  • the process may be performed sequentially for a plurality of objects OBJ reflected in the image indicated by the image data IMG_2D.
  • the 2D matching unit 3121 may detect multiple objects OBJ.
  • At least two of the plurality of objects OBJ detected using the same template image may have the same three-dimensional shape.
  • the state in which "the three-dimensional shapes of the two objects OBJ are the same” may include the state in which "the three-dimensional shapes of the two objects OBJ are literally completely the same.”
  • the "two objects OBJ have the same three-dimensional shape” state means "the two objects OBJ are manufactured to have the same three-dimensional shape, but due to manufacturing errors, the three-dimensional shape of the two objects OBJ is the same.” may include "different" states.
  • the "three-dimensional shapes of the two objects OBJ are the same” state means "the three-dimensional shapes of the two objects OBJ are different, but the image indicated by the image data IMG_2D generated by the imaging device 21 imaging the two objects OBJ
  • the above may include a state in which the shapes of the two objects OBJ can be considered to be the same.
  • "Two objects OBJ have the same three-dimensional shape” state means "Two objects OBJ are manufactured to have the same three-dimensional shape, but one of the two objects OBJ is different from the other of the two objects OBJ or It may also include a state in which the two objects OBJ have different three-dimensional shapes due to one of the two objects OBJ deforming due to contact with the object.
  • the state that "the three-dimensional shapes of the two objects OBJ are the same” means that "the parts of the two objects OBJ that are included in the imaging range (field of view) of the imaging device 21 have the same three-dimensional shape, It may also include a state in which the other parts of the two objects OBJ and which are not included in the imaging range (field of view) of the imaging device 21 have different three-dimensional shapes. Therefore, the three-dimensional shapes of at least two of the plurality of objects OBJ detected using the same template image may actually be different.
  • the 3D matching unit 3122 matches multiple objects OBJ using the same 3D model WM3. May be detected. That is, the 3D matching unit 3122 may detect a plurality of objects OBJ having the same three-dimensional shape indicated by the three-dimensional model WM3.
  • the state that "the three-dimensional shapes of two objects OBJ detected using the same three-dimensional model WM3 are the same” is the state that "the three-dimensional shapes of two objects OBJ detected using the same template image are the same” as described above. It may include "same" states and similar states.
  • the three-dimensional shapes of at least two of the plurality of objects OBJ detected using the same three-dimensional model WM3 may actually be different. This is because even if one object OBJ and another object OBJ different from the one object are detected using the same three-dimensional model WM3, since the one object OBJ and the other object OBJ are different objects. This is because the three-dimensional shape of one object OBJ and the three-dimensional shape of another object OBJ may actually be different.
  • the 2D matching unit 3121 may perform 2D matching processing using a plurality of template images each showing a plurality of mutually different two-dimensional shapes.
  • the 2D matching unit 3121 performs a 2D matching process using a first template image showing a first two-dimensional model, so that the object OBJ shown by the first template image (for example, a first three-dimensional shape OBJ) may be detected.
  • the 2D matching unit 3121 performs a 2D matching process using a second template image showing a second two-dimensional model different from the first two-dimensional model, so that the object OBJ shown by the second template image is (For example, an object OBJ having a second three-dimensional shape different from the first three-dimensional shape) may be detected.
  • the target determination unit 314a may determine (in this case, select) any one of the plurality of objects OBJ having different three-dimensional shapes as the target object OBJ_target by a process described later.
  • At least two of the plurality of objects OBJ detected using different template images may have different three-dimensional shapes.
  • at least two of the plurality of objects OBJ detected using different template images may have the same three-dimensional shape.
  • at least two of the plurality of objects OBJ detected using different template images may have the same three-dimensional shape.
  • the state "two objects OBJ detected using different template images have the same three-dimensional shape" is the above-mentioned state "two objects OBJ detected using the same template image have the same three-dimensional shape” may include states similar to the state.
  • the 3D matching unit 3122 matches multiple objects OBJ using different 3D models WM3. May be detected.
  • at least two of the plurality of objects OBJ detected using different three-dimensional models WM3 may have different three-dimensional shapes.
  • at least two of the plurality of objects OBJ detected using different three-dimensional models WM3 may have the same three-dimensional shape.
  • at least two of the plurality of objects OBJ detected using different three-dimensional models WM3 may have the same three-dimensional shape. good.
  • the state that "the three-dimensional shapes of two objects OBJ detected using different three-dimensional models WM3 are the same” refers to the state that "the three-dimensional shapes of two objects OBJ detected using the same template image are the same” as described above. It may include “same” states and similar states.
  • the 2D matching unit 3121 does not need to calculate the position and orientation of the detected object OBJ. That is, at the stage of step S31a, the 2D matching unit 3121 does not need to generate the position and orientation information POI1 indicating the position and orientation of the detected object OBJ. This is because, at the stage of step S31a, the object OBJ detected by the 2D matching process is not necessarily the target object OBJ_target. In this case, the 2D matching unit 3121 may calculate the position and orientation of the object OBJ detected in step S31a after the object OBJ is determined to be the target object OBJ_target. The 2D matching unit 3121 may generate position and orientation information POI1 indicating the position and orientation of the object OBJ detected in step S31a after the object OBJ is determined to be the target object OBJ_target.
  • the target determination unit 314a determines whether or not the object OBJ detected in the 2D matching process of step S31a is determined as the target object OBJ_target (steps S32a to S33a).
  • the target determining unit 314a may determine whether to determine the object OBJ as the target object OBJ_target by determining whether the object OBJ detected by the 2D matching process satisfies a predetermined matching condition. (Step S32a).
  • the matching conditions may include conditions regarding the results of the 2D matching process performed in step S31a.
  • the target determination unit 314a may be considered to be determining whether or not to determine the object OBJ detected in the 2D matching process as the target object OBJ_target based on the result of the 2D matching process.
  • the predetermined matching conditions may include conditions regarding matching similarity. Specifically, in the 2D matching process, the position/orientation calculation unit 312 moves the characteristic points of the object OBJ reflected in the template image closer to the characteristic points of the object OBJ reflected in the image indicated by the image data IMG_2D.
  • the matching similarity which is the similarity between the template image and the similar image portion in which the object OBJ of the template image is matched among the images indicated by the image data IMG_2D, is calculated as described above. be. In this case, the higher the matching similarity, the higher the possibility that the object OBJ reflected in the similar image portion is the same as the object OBJ indicated by the template image.
  • the higher the matching similarity the more the object OBJ detected by the 2D matching process is the same as the object OBJ that should be detected (that is, expected to be detected) by the 2D matching process. More likely.
  • the higher the matching similarity the higher the possibility that the object OBJ detected by the 2D matching process is the same as the object OBJ on which the end effector 13 should perform predetermined processing.
  • the object determining unit 314a determines whether the object OBJ detected by the 2D matching process satisfies the predetermined matching condition by determining whether the matching similarity exceeds a predetermined matching determination threshold. You may judge. If the matching similarity exceeds the matching determination threshold, the object determining unit 314a may determine that the object OBJ detected by the 2D matching process satisfies a predetermined matching condition (step S32a: Yes). On the other hand, if the matching similarity is less than the matching determination threshold, the object determining unit 314a may determine that the object OBJ detected by the 2D matching process does not satisfy the predetermined matching condition (step S32a: No).
  • the matching determination threshold is determined when the object OBJ detected by the 2D matching process is the same as the object OBJ for which the end effector 13 should perform a predetermined process, and the object OBJ detected by the 2D matching process is 13 may be set to an appropriate value that can appropriately distinguish between the object OBJ to which the predetermined process is to be performed and a different state from the matching similarity.
  • the target determining unit 314a determines whether the object OBJ detected by the 2D matching process satisfies a predetermined edge condition, thereby determining the object OBJ as the target object OBJ_target. It may be determined whether or not to decide (step S33a).
  • the edge conditions may include conditions regarding the result of the edge of the object OBJ detected based on the result of the 2D matching process.
  • the predetermined edge conditions may include conditions regarding edge similarity.
  • the edge similarity indicates the degree of similarity between an image edge, which is an edge of the image indicated by the image data IMG_2D acquired in step S1, and an object edge, which is an edge of the object OBJ detected by the 2D matching process.
  • the edge similarity may be an index value indicating the ratio of an overlapping edge portion that overlaps (that is, overlaps) an object edge among image edges to the object edge.
  • the edge similarity may be an index value indicating the ratio of the length of the overlapping edge portion to the length of the object edge. For this reason, edge similarity may be referred to as edge multiplicity.
  • the target determination unit 314a may calculate edge similarity. Specifically, the target determining unit 314a may detect an edge of the image (that is, an image edge) indicated by the image data IMG_2D acquired in step S1. For example, the target determining unit 314a may perform preprocessing (for example, gamma correction processing) on the image data IMG_2D as necessary, and then detect image edges using an edge detection filter such as a differential filter. good. Further, the object determining unit 314a may detect an edge of the object OBJ (that is, an object edge) detected by the 2D matching process.
  • preprocessing for example, gamma correction processing
  • the object determining unit 314a may detect the wire frame of the object OBJ (that is, the wire frame of the two-dimensional model) indicated by the template image matched to the image indicated by the image data IMG_2D as an object edge. . Thereafter, the target determination unit 314a may calculate edge similarity based on the detection results of image edges and object edges.
  • the higher the edge similarity the less the portion of the object OBJ reflected in the similar image portion that is hidden by the shielding object. This is because, when a part of the object OBJ is hidden by the shielding object, the part of the object edge hidden by the shielding object is no longer detected as an image edge. On the other hand, even if part of the object OBJ is hidden by the shield, a portion of the object edge that is hidden by the shield is detected as an object edge. This is because, as described above, the object determining unit 314a detects the wire frame of the object OBJ indicated by the template image (that is, the wire frame used in image processing that is not affected by an obstruction) as an object edge. Because there is.
  • an example of the shielding object is another object OBJ detected by the 2D matching process.
  • An example of an occluding object is any object that has not been detected by the 2D matching process. Therefore, the higher the edge similarity, the lower the possibility that the end effector 13 will interfere with an obstruction when the end effector 13 performs a predetermined process on the object OBJ. In other words, the higher the edge similarity, the higher the possibility that the end effector 13 will be able to perform a predetermined process on the object OBJ without being interfered with by an obstruction.
  • FIG. 22 shows two objects OBJ (specifically, objects OBJ#A and OBJ#B that differ in at least one of their positions and orientations) detected by the 2D matching process.
  • FIG. 22 shows an example in which object OBJ#A is stacked on object OBJ#B.
  • object OBJ#A is hidden by object OBJ#A.
  • object OBJ#B is hidden by object OBJ#A, so it is no longer detected as an image edge.
  • the number of overlapping edges that overlap the edge of object OBJ#B among the image edges is reduced.
  • the edge of object OBJ#A is detected as an image edge because it is not hidden by object OBJ#B.
  • the number of overlapping edges that overlap the edge of object OBJ#A among the image edges does not decrease. Therefore, the edge similarity of object OBJ#A that is not hidden by object OBJ#B is higher than the edge similarity of object OBJ#B that is hidden by object OBJ#A.
  • the end effector 13 that performs a predetermined process on the object OBJ#B may interfere with the object OBJ#A that partially hides the object OBJ#B.
  • the end effector 13 when the end effector 13 performs a holding process to hold the object OBJ#B, there is a possibility that the end effector 13 comes into contact with the object OBJ#A stacked on the object OBJ#B.
  • the end effector 13 that performs predetermined processing on the object OBJ#A is unlikely to interfere with the object OBJ#B below the object OBJ#A. Therefore, in this case, the object OBJ#A with higher edge similarity should be determined as the object OBJ on which the end effector 13 should perform predetermined processing than the object OBJ#B with lower edge similarity.
  • the object determining unit 314a determines whether the object OBJ detected by the 2D matching process satisfies the predetermined edge condition by determining whether the edge similarity exceeds a predetermined edge determination threshold. may be determined. If the edge similarity exceeds the edge determination threshold, the object determining unit 314a may determine that the object OBJ detected by the 2D matching process satisfies a predetermined edge condition (step S33a: Yes). On the other hand, if the edge similarity is less than the edge determination threshold, the object determining unit 314a may determine that the object OBJ detected by the 2D matching process does not satisfy the predetermined edge condition (step S33a: No).
  • the edge determination threshold is determined based on the state in which the object OBJ detected by the 2D matching process is an object OBJ on which the end effector 13 can perform predetermined processing without interference from other objects OBJ, and the 2D matching process.
  • a state in which an object OBJ detected by processing is an object OBJ for which it is difficult for an end effector 13 to perform predetermined processing without interference from other objects OBJ. may be set to an appropriate value that allows for.
  • the target determination unit 314a determines whether or not the object OBJ is determined as the target object OBJ_target by determining whether the object OBJ satisfies both the matching condition and the edge condition. An example of determination is shown. In this case, if it is determined that the object OBJ satisfies both the matching condition and the edge condition (step S32a: Yes and step S33a: Yes), the target determining unit 314a determines the object OBJ as the target object OBJ_target. (Step S34a).
  • the object determining unit 314a it is not necessary to determine the object OBJ as the object OBJ_target.
  • the target determining unit 314a uses either the matching condition or the edge condition in order to determine whether or not to determine the object OBJ as the target object OBJ_target, while using the other of the matching condition and the edge condition. It doesn't have to be used.
  • the target determination unit 314a may use the matching condition but not the edge condition in order to determine whether to determine the object OBJ as the target object OBJ_target. In this case, if it is determined that the object OBJ satisfies the matching condition (step S32a: Yes), the target determining unit 314a may determine the object OBJ as the target object OBJ_target (step S34a). On the other hand, if it is determined that the object OBJ does not satisfy the matching condition (step S32a: No), the target determining unit 314a does not need to determine the object OBJ as the target object OBJ_target.
  • the target determination unit 314a may use the edge condition but not the matching condition in order to determine whether to determine the object OBJ as the target object OBJ_target. In this case, if it is determined that the object OBJ satisfies the edge condition (step S33a: Yes), the target determination unit 314a may determine the object OBJ as the target object OBJ_target (step S34a). On the other hand, if it is determined that the object OBJ does not satisfy the edge condition (step S33a: No), the target determination unit 314a does not need to determine the object OBJ as the target object OBJ_target.
  • the target determining unit 314a determines (in this case, selects) any one of the multiple objects OBJ as the target object OBJ_target. good. For example, the object determining unit 314a selects the object OBJ with the highest matching similarity among the plurality of objects OBJ that satisfy the matching condition and the edge condition (the matching similarity exceeds the matching judgment threshold and the edge similarity exceeds the edge judgment threshold). One object OBJ may be determined (in this case, selected) as the target object OBJ_target.
  • the object determining unit 314a selects the object OBJ with the highest edge similarity from among the plurality of objects OBJ that satisfy the matching condition and the edge condition (the matching similarity exceeds the matching judgment threshold and the edge similarity exceeds the edge judgment threshold).
  • One object OBJ may be determined (in this case, selected) as the target object OBJ_target.
  • the target determining unit 314a selects one object OBJ with the highest matching similarity from among the plurality of objects OBJ detected by the 2D matching process, without determining whether the object OBJ satisfies the matching condition and the edge condition.
  • the object OBJ may be determined (in this case, selected) as the object OBJ_target.
  • the target determining unit 314a selects one object OBJ with the highest edge similarity from among the plurality of objects OBJ detected by the 2D matching process, without determining whether the object OBJ satisfies the matching condition and the edge condition. may be determined (in this case, selected) as the object OBJ_target.
  • the target determination unit 314a determines any one of the plurality of objects OBJ as the target object OBJ_target (in this case, the selected ) may be done.
  • the target determination unit 314a determines one object OBJ selected from among the plurality of objects OBJ detected by the 2D matching process based on the matching similarity and edge similarity as the target object OBJ_target (in this case, , selection).
  • the target object OBJ_target one object OBJ selected based on the matching similarity and edge similarity from the first object OBJ#1 and the second object OBJ#2 detected by the 2D matching process.
  • the matching similarity of the first object OBJ#1 is expressed as a variable "MS#1"
  • the matching similarity of the second object OBJ#2 is expressed as a variable "MS#2”.
  • the edge similarity of the first object OBJ#1 is expressed as a variable "ES#1”
  • the edge similarity of the second object OBJ#2 is expressed as a variable "ES#2”.
  • the target determining unit 314a selects the matching similarity MS#1, the matching similarity MS#2, the edge similarity ES#1, and the edge similarity from the first object OBJ#1 and the second object OBJ#2. Based on the degree ES#2, one selected object OBJ is determined (in this case, selected) as the target object OBJ_target.
  • the target is determined.
  • the unit 314a determines (in this case, selects) the one object OBJ as the target object OBJ_target. Specifically, the edge similarity ES#1 exceeds the edge determination threshold, the edge similarity ES#1 is higher than the edge similarity ES#2, the matching similarity MS#1 exceeds the matching determination threshold, and When the matching similarity MS#1 is higher than the matching similarity MS#2, the target determining unit 314a determines (in this case, selects) the first object OBJ#1 as the target object OBJ_target.
  • the target determining unit 314a determines (in this case, selects) the second object OBJ#2 as the target object OBJ_target.
  • the object determining unit 314a calculates the matching ratio and the edge ratio.
  • the matching ratio indicates a value obtained by dividing the smaller of the matching similarities MS#1 and MS#2 by the larger of the matching similarities MS#1 and MS#2. In other words, the matching ratio indicates the ratio of the smaller of the matching similarities MS#1 and MS#2 to the larger of the matching similarities MS#1 and MS#2.
  • the edge ratio indicates a value obtained by dividing the smaller of edge similarities ES#1 and ES#2 by the larger of edge similarities ES#1 and ES#2. That is, the edge ratio indicates the ratio of the smaller of the edge similarities ES#1 and ES#2 to the larger of the edge similarities ES#1 and ES#2.
  • the target determining unit 314a determines one object OBJ selected from the first object OBJ#1 and the second object OBJ#2 based on the matching ratio and the edge ratio as the target object OBJ_target ( In this case, select). Specifically, when the matching ratio is smaller than the edge ratio, the difference between the matching similarities MS#1 and MS#2 is larger than the difference between the edge similarities ES#1 and ES#2.
  • the target determining unit 314a selects one of the first object OBJ#1 and second object OBJ#2, which has a higher matching similarity than the other object OBJ, as the target object OBJ_target. Decide (in this case, choose).
  • the target determining unit 314a selects one of the first object OBJ#1 and second object OBJ#2, which has a higher degree of edge similarity than the other object OBJ, as the target object OBJ_target. Decide (in this case, choose).
  • the edge similarity ES#1 is higher than the edge determination threshold
  • the edge similarity ES#1 is higher than the edge similarity ES#2
  • the matching similarity MS#1 is higher than the matching determination threshold
  • the matching similarity is lower than the matching similarity degree MS#2
  • the matching ratio and edge ratio are calculated using the formula #2/edge similarity ES#1.
  • the target determining unit 314a determines the second object OBJ#2 having a matching similarity MS#2 higher than the matching similarity MS#1 as the target object OBJ_target. (in this case, select).
  • the target determination unit 314a sets the first object OBJ#1 having the edge similarity ES#1 higher than the edge similarity RS#2 as the target object OBJ_target. Decide (in this case, choose).
  • the edge similarity ES#2 exceeds the edge determination threshold
  • the edge similarity ES#2 is higher than the edge similarity ES#1
  • the matching similarity MS#2 exceeds the matching determination threshold
  • the matching ratio and edge ratio are calculated using the formula: degree ES#1/edge similarity ES#2.
  • the target determining unit 314a determines the first object OBJ#1 having a matching similarity MS#1 higher than the matching similarity MS#2 as the target object OBJ_target. (in this case, select).
  • the target determining unit 314a sets the second object OBJ#2 having the edge similarity ES#2 higher than the edge similarity RS#1 as the target object OBJ_target. Decide (in this case, choose).
  • the matching similarity and edge similarity of the object OBJ determined as the target object OBJ_target exceed the matching determination threshold and the edge determination threshold, respectively. That is, when determining (in this case, selecting) one object OBJ selected based on the matching similarity and edge similarity as the target object OBJ_target, the target determining unit 314a determines that the matching similarity and edge similarity are The object OBJ that exceeds the matching determination threshold and the edge determination threshold, respectively, is determined as the target object OBJ_target. Specifically, the case where edge similarity ES#1 exceeds the edge judgment threshold, matching similarity MS#1 exceeds the matching judgment threshold, and edge similarity ES#2 exceeds the edge judgment threshold.
  • the target determining unit 314a determines (in this case, selects) the first object OBJ#1 as the target object OBJ_target.
  • the target determining unit 314a determines (in this case, selects) the second object OBJ#2 as the target object OBJ_target.
  • the target determining unit 314a determines both the first object OBJ#1 and the second object OBJ#2 as the target object OBJ_target. (In this case, select) No.
  • step S34a if the object OBJ detected by the 2D matching process is determined as the target object OBJ_target, the position and orientation calculation unit 312 calculates the position and orientation of the object OBJ determined as the target object OBJ_target. Specifically, the three-dimensional position data generation unit 311 generates three-dimensional position data WSD based on the image data IMG_3D acquired in step S1 (step S2). However, the three-dimensional position data generation unit 311 generates the three-dimensional position data WSD based on the image data IMG_3D acquired in step S1 before the object OBJ detected in the 2D matching process has been determined as the target object OBJ_target. may be generated in advance.
  • the three-dimensional position data generation unit 311 generates the three-dimensional position data WSD based on the image data IMG_3D acquired in step S1 during the period in which the processes from step S31a to step S33a are performed. You can leave it there.
  • the position and orientation calculation unit 312 calculates at least one of the position and orientation of the object OBJ based on the image data IMG_2D acquired in step S1 and the three-dimensional position data WSD generated in step S2 (step S3 ). As a result, the position and orientation calculation unit 312 generates position and orientation information POI0 indicating at least one of the position and orientation of the object OBJ (step S3).
  • step S3 of FIG. 21 the position and orientation calculation unit 312 does not need to perform the 2D matching process again.
  • the position and orientation calculation unit 312 determines the initial position and initial orientation of the three-dimensional model WM3 used to perform the tracking process based on the result of the 2D matching process in step S31a (that is, the position and orientation information POI1). You may.
  • the position and orientation calculation unit 312 may perform the 2D matching process again.
  • the imaging device 21 is moved for matching processing.
  • the 2D imaging time at which the object OBJ is imaged and the 3D imaging time at which the imaging device 22 images the object OBJ for matching processing may be set to different times.
  • the imaging device 22 may image the object OBJ determined as the target object OBJ_target after the object OBJ is determined as the target object OBJ_target. That is, the imaging device 22 may image the object OBJ after the imaging device 21 images the object OBJ in order to determine whether the object OBJ is determined as the target object OBJ_target.
  • the position and orientation calculation unit 312 uses the result of the 2D matching process (that is, position and orientation information POI1) performed to determine whether or not the object OBJ is determined as the target object OBJ_target, and the At least one of the position and orientation of the object OBJ may be calculated based on the three-dimensional position data WSD generated from the image data IMG_3D generated by the imaging device 22 imaging the object OBJ after imaging the object OBJ. .
  • the target determination unit 314a may determine the object OBJ that has not been determined as the target object OBJ_target at one point in time as the target object OBJ_target at another point in time different from the one point in time. be. For example, when the robot 1 performs a release process to place multiple workpieces W one by one on the mounting device T, the target determination unit 314a sequentially determines the multiple workpieces W as the target object OBJ_target (this (selection) may be made as described above. In this case, even after the object OBJ detected in the 2D matching process is determined as the target object OBJ_target, the target determining unit 314a performs the steps S1 to S4 in FIG.
  • the target determining unit 314a determines the object OBJ as the target object OBJ_target until it is determined that the robot control process is finished. You may repeat the process of determining whether or not.
  • the image capturing range (field of view) of the imaging device 21 is , object OBJ may not exist.
  • the imaging device 21 may not be imaging the object OBJ from an appropriate position.
  • the imaging device 21 may not be imaging the object OBJ in an appropriate posture. In other words, the imaging device 21 may not be imaging the object OBJ from an appropriate direction.
  • the control device 3 may control the robot 1 to move the imaging device 21 with respect to the object OBJ (step S36a). Specifically, the control device 3 may generate a robot control signal for controlling the robot 1 (in particular, the robot arm 12) to move the imaging device 21 with respect to the object OBJ. Thereafter, the control device 3 may output the generated robot control signal to the robot 1. As a result, the robot 1 (particularly the robot arm 12) may move the imaging device 21 with respect to the object OBJ.
  • the imaging device 21 moves relative to the object OBJ, there is a possibility that the object OBJ will newly exist in the imaging range (field of view) of the imaging device 21, where the object OBJ did not exist before the imaging device 21 moved. It's coming.
  • the imaging device 21 moves relative to the object OBJ, the imaging device 21, which was not imaging the object OBJ from an appropriate position before the imaging device 21 moved, can now image the object OBJ from an appropriate position. Gender comes out.
  • the imaging device 21 moves relative to the object OBJ, the imaging device 21, which was not imaging the object OBJ in an appropriate attitude before the imaging device 21 moved, can now image the object OBJ in an appropriate attitude. Gender comes out.
  • the imaging device 21 moves relative to the object OBJ
  • the imaging device 21 which had not imaged the object OBJ from an appropriate direction before the imaging device 21 moved, can now image the object OBJ from an appropriate direction. There is a possibility that it can be done. Therefore, compared to the case where the imaging devices 21 and 22 are not moved relative to the object OBJ, the control device 3 is more likely to be able to determine the object OBJ detected by the 2D matching process as the object OBJ_target. .
  • step S1 to step S33a in FIG. 21 may be performed again.
  • the control device 3 performs an operation of moving the imaging device 21 until the object OBJ detected by the 2D matching process is determined as the target object OBJ_target, and the control device 3 performs an operation of moving the imaging device 21 until the object OBJ detected by the 2D matching process is determined as the target object OBJ_target.
  • the operation of determining whether or not it is possible may be repeated alternately.
  • the control device 3 generates a robot control signal for controlling the robot 1 (in particular, the robot arm 12) so that the imaging device 21 rotates around a desired rotation axis, as shown in FIG. 23(a). It's okay.
  • the control device 3 may rotate the imaging device 21 around at least one of a rotation axis along the X-axis (2D), a rotation axis along the Y-axis (2D), and a rotation axis along the Z-axis (2D). may be rotated.
  • the control device 3 rotates the imaging device 21 around at least one of a rotation axis along the X-axis (3D), a rotation axis along the Y-axis (3D), and a rotation axis along the Z-axis (3D).
  • the control device 3 rotates the imaging device 21 around at least one of a rotation axis along the X-axis (GL), a rotation axis along the Y-axis (GL), and a rotation axis along the Z-axis (GL). You may move it.
  • the control device 3 may rotate the imaging device 21 around a rotation axis that intersects the optical axis AX21 of the imaging device 21.
  • the control device 3 since the Z-axis (2D) is an axis along the optical axis AX21, the control device 3 may rotate the imaging device 21 around the rotation axis that intersects the Z-axis (2D).
  • the control device 3 may rotate the imaging device 21 around the X-axis (2D) intersecting the Z-axis (2D).
  • the control device 3 may rotate the imaging device 21 around the Y-axis (2D) intersecting the Z-axis (2D).
  • FIG. 23(a) the direction in which the imaging device 21 images the object OBJ changes. For this reason, the possibility that the imaging device 21, which did not image the object OBJ from an appropriate direction before the imaging device 21 moved, will be able to image the object OBJ from an appropriate direction tends to increase.
  • the control device 3 performs a first rotational movement operation of rotating the imaging device 21 in a positive rotational direction around the X-axis (2D) and a negative rotational movement around the X-axis (2D).
  • a second rotational movement operation that rotationally moves the imaging device 21 in the rotational direction
  • a third rotational movement operation that rotationally moves the imaging device 21 in the positive rotational direction around the Y-axis (2D)
  • a third rotational movement operation that rotationally moves the imaging device 21 in the positive rotational direction around the Y-axis (2D
  • At least one of the fourth rotational movement operation of rotationally moving the imaging device 21 in the negative rotational direction may be performed.
  • the control device 3 may sequentially perform at least two of the first to fourth rotational movement operations until the object OBJ detected in the 2D matching process is determined as the target object OBJ_target. For example, the control device 3 may perform the first rotational movement operation, and then determine whether the object OBJ detected in the 2D matching process is determined as the object OBJ_target. As a result, if it is still determined that the object OBJ detected in the 2D matching process is not determined as the target object OBJ_target, the control device 3 performs a second rotational movement operation, and then the object OBJ detected in the 2D matching process is It may be determined whether the object OBJ is determined as the object OBJ_target.
  • the control device 3 performs the third rotational movement operation, and then the object OBJ detected in the 2D matching process is It may be determined whether the object OBJ is determined as the object OBJ_target. As a result, if it is still determined that the object OBJ detected in the 2D matching process is not determined as the target object OBJ_target, the control device 3 performs a fourth rotational movement operation, and then the object OBJ detected in the 2D matching process is It may be determined whether the object OBJ is determined as the object OBJ_target.
  • control device 3 controls the robot 1 (in particular, the robot arm 12) so that the imaging device 21 translates along a desired translation axis, as shown in FIG. 23(b).
  • Robot control signals may also be generated.
  • the control device 3 may translate the imaging device 21 along at least one of the X axis (2D), the Y axis (2D), and the Z axis (2D).
  • the control device 3 may translate the imaging device 21 along at least one of the X axis (3D), the Y axis (3D), and the Z axis (3D).
  • the control device 3 may translate the imaging device 21 along at least one of the X-axis (GL), the Y-axis (GL), and the Z-axis (GL).
  • the control device 3 may translate the imaging device 21 along a translational axis that intersects the optical axis AX21 of the imaging device 21.
  • the control device 3 since the Z-axis (2D) is an axis along the optical axis AX21, the control device 3 may translate the imaging device 21 along the translational axis that intersects the Z-axis (2D).
  • the control device 3 may translate the imaging device 21 along the X-axis (2D) that intersects the Z-axis (2D).
  • the control device 3 may translate the imaging device 21 along the Y axis (2D) that intersects the Z axis (2D).
  • the position at which the imaging device 21 images the object OBJ changes. For this reason, the possibility that the imaging device 21, which did not image the object OBJ from an appropriate position before the imaging device 21 moved, will be able to image the object OBJ from an appropriate position is likely to increase.
  • the control device 3 performs a first translation operation to translate the imaging device 21 in a positive translation direction along the X-axis (2D) and a first translation operation to translate the imaging device 21 in a positive translation direction along the X-axis (2D)
  • a second translational movement operation that translates the imaging device 21 in a negative translational direction; a third translational movement action that translates the imaging device 21 in a positive translational direction along the Y-axis (2D);
  • the control device 3 may sequentially perform at least two of the first translation operation to the fourth translation operation until the object OBJ detected in the 2D matching process is determined as the target object OBJ_target.
  • control device 3 may perform the first translation operation, and then determine whether the object OBJ detected in the 2D matching process is determined as the object OBJ_target. As a result, if it is still determined that the object OBJ detected in the 2D matching process is not determined as the target object OBJ_target, the control device 3 performs the second translation operation, and then the object OBJ detected in the 2D matching process is It may be determined whether the object OBJ is determined as the object OBJ_target.
  • the control device 3 performs a third translation operation, and then the object OBJ detected in the 2D matching process is It may be determined whether the object OBJ is determined as the object OBJ_target. As a result, if it is still determined that the object OBJ detected in the 2D matching process is not determined as the target object OBJ_target, the control device 3 performs a fourth translation operation, and then the object OBJ detected in the 2D matching process is It may be determined whether the object OBJ is determined as the object OBJ_target.
  • the control device 3 may alternately perform an operation of rotating the imaging device 21 and an operation of translating the imaging device 21. For example, the control device 3 may sequentially perform the above-described first to fourth rotational movement operations until the object OBJ detected in the 2D matching process is determined as the target object OBJ_target. Thereafter, even after the first to fourth rotational movement operations described above are performed, if it is still determined that the object OBJ detected in the 2D matching process is not determined as the object OBJ_target, the control device 3 may perform any one of the first translation operation to the fourth translation operation so that the imaging device 21 moves from one position to another position.
  • control device 3 uses the imaging device 21 located at another position to perform the first to fourth rotational movement operations until the object OBJ detected in the 2D matching process is determined as the object OBJ_target. may be performed in order. Thereafter, the control device 3 may repeat the same operation.
  • the control device 3 performs an operation of rotating the imaging device 21 and an operation of translating the imaging device 21. You may give priority to This is because the imaging device 21 is more likely to be able to properly image the object OBJ if the direction in which the imaging device 21 images the object OBJ changes than if the position in which the imaging device 21 images the object OBJ changes. This is because it becomes expensive.
  • control device 3 of the first modification can appropriately determine whether to determine the object OBJ detected by the 2D matching process as the object OBJ_target. Therefore, even if a plurality of objects OBJ are detected by the 2D matching process, the control device 3 can appropriately determine one of the plurality of objects OBJ as the target object OBJ_target.
  • the control device 3 can determine the object OBJ that satisfies the matching condition as the target object OBJ_target. In this case, the control device 3 can appropriately determine the object OBJ that is expected to be detected by the 2D matching process as the target object OBJ_target. That is, the control device 3 does not erroneously determine the object OBJ, which should not be detected by the 2D matching process, as the target object OBJ_target. Therefore, the end effector 13 can appropriately perform predetermined processing on the object OBJ to be detected by 2D matching processing.
  • the control device 3 can determine the object OBJ that satisfies the edge condition as the target object OBJ_target. In this case, the control device 3 can preferentially determine the object OBJ that is not hidden by the shielding object as the target object OBJ_target, rather than the object OBJ that is partially hidden by the shielding object.
  • the object OBJ that is partially hidden by the shield is assumed to be the object OBJ on which the shield overlaps. Therefore, if the end effector 13 attempts to perform a predetermined process (for example, the above-mentioned holding process) on such an object OBJ, there is a possibility that the shielding object will interfere with the process of the end effector 13.
  • an object OBJ that is not hidden by a shield is an object OBJ on which a shield does not overlap. Therefore, even if the end effector 13 attempts to perform a predetermined process (for example, the above-described holding process) on such object OBJ, there is a low possibility that the process of the end effector 13 will be hindered. Therefore, the control device 3 can determine, as the target object OBJ_target, an object OBJ with which the end effector 13 is likely to be able to appropriately perform a predetermined action. That is, the control device 3 does not determine, as the target object OBJ_target, an object OBJ for which there is a high possibility that the end effector 13 cannot appropriately perform a predetermined operation. Therefore, the end effector 13 can appropriately perform predetermined processing on the object OBJ.
  • a predetermined process for example, the above-described holding process
  • the effect that can be achieved by determining the object OBJ that satisfies such edge conditions as the target object OBJ_target is that the end effector 13 is able to perform a predetermined position on a plurality of workpieces W stacked randomly on the mounting device T. This is particularly effective when performing processing (for example, holding processing). This is because, when a plurality of works W are stacked randomly on the mounting device T, there is a high possibility that a part of one work W is blocked by another work W.
  • the control device 3 moves the imaging device 21 (for example, rotationally and/or translationally) and then It can be determined again whether or not the object OBJ detected by the 2D matching process is determined as the object OBJ_target. Therefore, the control device 3 can appropriately find the object OBJ determined as the target object OBJ_target.
  • the control device 3 when the object OBJ detected by the 2D matching process is determined as the target object OBJ_target, the control device 3 normally controls the robot arm 12 so that the end effector 13 approaches the object OBJ. Generate robot control signals. However, before the object OBJ detected by the 2D matching process is determined as the target object OBJ_target, the control device 3 controls the robot arm so that the end effector 13 approaches the object OBJ detected by the 2D matching process. 12 may be generated. For example, when a plurality of objects OBJ are detected by the 2D matching process, the control device 3 causes the end effector 13 to approach the object OBJ that has been detected by the 2D matching process and has the highest matching similarity. A robot control signal for controlling the robot arm 12 may also be generated.
  • the control device 3 when a plurality of objects OBJ are detected by the 2D matching process, the control device 3 causes the end effector 13 to approach the object OBJ detected by the 2D matching process and having the highest edge similarity.
  • a robot control signal for controlling the robot arm 12 may also be generated.
  • the control device 3 controls the robot arm 12 so that the end effector 13 approaches the object OBJ detected by the 2D matching process.
  • a robot control signal for controlling the robot may be generated. In this case, the control device 3 performs the robot control process shown in FIG.
  • the 21 determines the object OBJ as the target object OBJ_target during a period in which the end effector 13 approaches the object OBJ detected by the 2D matching process. (a process of determining whether or not) may be performed. As a result, after the object OBJ temporarily detected by the 2D matching process is determined as the target object OBJ_target, the time required for the end effector 13 to reach the object OBJ becomes shorter.
  • the control device 3 performs matching.
  • a robot control signal may be generated to control the robot arm 12 so that the end effector 13 approaches another object OBJ with a higher degree of similarity or edge similarity.
  • the end effector 13 is controlled to move toward the object OBJ#1 whose matching similarity is "MR1" before the object OBJ#1 is determined as the target object OBJ_target.
  • the robot arm 12 is shown in FIG. Assume that under the situation shown in FIG. 24(a), as shown in FIG.
  • an object OBJ#2 whose matching similarity is "MR2" which is higher than "MR1" is newly detected.
  • the control device 3 controls the robot arm 12 so that the end effector 13 approaches the object OBJ#2 instead of the object OBJ#1, as shown in FIG. 24(b).
  • a control signal may also be generated.
  • the control device 3 controls the robot 1 (e.g., robot arm 12) may generate a robot control signal for controlling.
  • control device 3 controls the robot 1 to move the imaging device 21 relative to the object OBJ when the object OBJ detected in the 2D matching process is not determined as the target object OBJ_target. is being controlled (step S36a in FIG. 21).
  • control device 3 may control the robot 1 to move the imaging device 21 with respect to the object OBJ even when the object OBJ detected in the 2D matching process is determined as the target object OBJ_target. .
  • control device 3 may move the imaging device 21 relative to the object OBJ so that as many edges of the object OBJ as possible are reflected in the image indicated by the image data IMG_2D.
  • the control device 3 moves the imaging device 21 relative to the object OBJ so that the number of edges of the object OBJ reflected in the image indicated by the image data IMG_2D increases compared to before moving the imaging device 21.
  • the control device 3 may move the imaging device 21 relative to the object OBJ so that the sum of the lengths of the edges of the object OBJ reflected in the image indicated by the image data IMG_2D is as large as possible.
  • the control device 3 moves the imaging device 21 relative to the object OBJ so that the sum of the lengths of the edges of the object OBJ reflected in the image indicated by the image data IMG_2D becomes larger than before moving the imaging device 21. You may move it.
  • the control device 3 may calculate the position and orientation information POI0 by performing the processes of steps S1, S2, and S3 in FIG. 21 again. Note that in this case, since the object OBJ determined as the target object OBJ_target has not changed, the control device 3 does not need to perform the processes from step 31a to step S34a in FIG. 21. In this case, compared to the case where the imaging device 21 is not moved with respect to the object OBJ, the variation in the orientation of the object OBJ indicated by the position and orientation information POI0 becomes smaller.
  • FIG. 25 shows an imaging device 21 that images the object OBJ from a first position P1, and an imaging device 21 that images the object OBJ from a second position P2 different from the first position P1. ing.
  • the edge of the object OBJ reflected in the image indicated by the image data IMG_2D generated by the imaging device 21 that images the object OBJ from the first position P1 images the object OBJ from the second position P2.
  • the number of edges is smaller than the number of edges of the object OBJ reflected in the image shown by the image data IMG_2D generated by the imaging device 21.
  • the control device 3 may move the imaging device 21 from the first position P1 to the second position P2.
  • the number of edges of the object OBJ appearing in the image increases (for example, the total length of the edges increases). Therefore, the amount of information that can be used by 2D matching processing increases. Therefore, variations in the position and orientation information POI1 generated by the 2D matching process are reduced. As a result, variations in the position and orientation information POI0 generated based on the position and orientation information POI1 are also reduced (reproducibility is increased).
  • the imaging device 22 also moves as the imaging device 21 moves, the amount of information included in the image data IMG_3D generated by the imaging device 21 and that can be used by the 3D matching process also increases. As a result, the amount of information included in the three-dimensional position data WSD generated from the image data IMG_3D and that can be used by the 3D matching process (for example, the number of point clouds included in the point cloud data) also increases. Therefore, variations in the position and orientation information POI2 generated by the 3D matching process are reduced. As a result, variations in the position and orientation information POI0 generated based on the position and orientation information POI2 are also reduced (reproducibility is increased).
  • FIG. 26 is a block diagram showing the configuration of the control device 3b in the first modification.
  • control device 3b in the second modification is different from the above-described control device 3 or 3a in that the arithmetic device 31 includes a pre-processing section 315b as a logical processing block. different.
  • the pre-processing unit 315b performs pre-processing before matching processing and tracking processing are performed.
  • Other features of the control device 3b may be the same as other features of the control device 3 or 3a.
  • the pre-processing unit 315b may perform pre-processing on the three-dimensional position data WSD generated by the three-dimensional position data generating unit 311.
  • the 3D matching unit 3122 may perform 3D matching processing using the pre-processed three-dimensional position data WSD.
  • the tracking unit 3123 may perform tracking processing using the pre-processed three-dimensional position data WSD.
  • the pre-processing performed on the three-dimensional position data WSD may include data deletion processing for deleting a part of the data portion of the three-dimensional position data WSD.
  • the three-dimensional position data WSD includes not only the data portion indicating the three-dimensional position of each of multiple points of the object OBJ whose position and orientation are to be calculated, but also the three-dimensional position data of other objects different from the object OBJ. It may contain a data portion indicating the original position.
  • the data deletion process may include a process of deleting, from the three-dimensional position data WSD, a data portion indicating the three-dimensional position of another object different from the object OBJ.
  • the imaging device 22 generates image data IMG_3D by capturing an image of the workpiece W placed on the mounting device T. Therefore, the three-dimensional position data WSD generated from the image data IMG_3D may include not only a data part indicating the three-dimensional position of each of the plurality of points on the workpiece W, but also a data part indicating the shape of the mounting device T. There is sex. In this case, as shown in FIG. 27, the preprocessing unit 315b may delete the data portion indicating the three-dimensional position of the mounting device T from the three-dimensional position data WSD.
  • the mounting surface of the mounting device T on which the workpiece W is mounted may be a flat surface.
  • the pre-processing unit 315b may perform plane removal processing, which is an example of data deletion processing, on the three-dimensional position data WSD.
  • plane removal processing is at least one of the plane removal process using RANSAC (Random Sample Consensus) and the plane removal process using the least squares method.
  • Such data deletion processing reduces the data size of the three-dimensional position data WSD.
  • the amount of calculation required for the 3D matching process using the three-dimensional position data WSD and the amount of calculation required for the tracking process using the three-dimensional position data WSD are reduced. Therefore, the 3D matching cycle, which is the time required for the 3D matching unit 3122 to complete the 3D matching process, and the tracking cycle, which is the time required for the tracking unit 3123 to complete the tracking process, can be shortened.
  • the position/orientation calculation unit 312 replaces the data portion indicating the three-dimensional position of an object different from the object OBJ whose position and orientation are to be calculated with the data portion indicating the three-dimensional position of the object OBJ.
  • the possibility of misrecognition as a data part is reduced. This is because the data portion indicating the three-dimensional position of an object different from the object OBJ, which may become noise in the tracking processing and matching processing described above, is deleted. Therefore, the position and orientation calculation unit 312 can appropriately recognize the data portion indicating the three-dimensional position of the object OBJ.
  • the position/orientation calculation unit 312 can appropriately recognize the point cloud regarding the object OBJ.
  • the position and orientation calculation unit 312 can appropriately recognize depth information regarding the object OBJ. As a result, the position and orientation calculation unit 312 can calculate the position and orientation of the object OBJ with higher accuracy than when data deletion processing is not performed.
  • the preprocessing unit 315b based on a user's instruction specifying a data portion of the three-dimensional position data WSD that indicates the three-dimensional position of another object different from the object OBJ, extracts the following from the three-dimensional position data WSD: A data portion indicating the shape of an object different from object OBJ may be deleted.
  • the control device 3 uses an output device 35 including a display device to display a part of the data portion of the three-dimensional position data WSD indicating the three-dimensional position of another object different from the object OBJ.
  • a GUI Graphic User Interface
  • the control device 3 uses the output device 35 including a display device to display a plurality of points indicated by point cloud data that is an example of the three-dimensional position data WSD, and displays a plurality of points that are different from the object OBJ among the plurality of points.
  • a GUI that can be operated by the user may be displayed to specify a point indicating the three-dimensional position of another object.
  • the control device 3 uses the output device 35 including a display device to display a depth image indicated by depth image data that is an example of the three-dimensional position data WSD, and displays other objects different from the object OBJ in the depth image.
  • a GUI that can be operated by the user may be displayed to specify pixels that represent the shape of the object.
  • the pre-processing unit 315b may perform pre-processing on at least one of the image data IMG_2D and IMG_3D.
  • the 2D matching unit 3121 may perform 2D matching processing using the image data IMG_2D that has been subjected to pre-processing.
  • the tracking unit 3123 may perform tracking processing using two pieces of image data IMG_2D, each of which has been pre-processed.
  • the three-dimensional position data generation unit 311 may generate the three-dimensional position data WSD using the pre-processed image data IMG_3D.
  • the 3D matching unit 3122 may perform 3D matching processing using the three-dimensional position data WSD generated from the pre-processed image data IMG_3D.
  • the tracking unit 3123 may perform tracking processing using the three-dimensional position data WSD generated from the pre-processed image data IMG_3D.
  • the pre-processing performed on at least one of the image data IMG_2D and IMG_3D may include gamma correction processing.
  • Gamma correction processing emphasizes the edges of object OBJ reflected in the image represented by image data IMG_2D (or image data IMG_3D) by adjusting the contrast of the image represented by image data IMG_2D (or image data IMG_3D). It may also include correction processing.
  • the imaging unit 2 may include a photometric device 24b capable of measuring the brightness (that is, luminance) of the object OBJ. . At least one of the imaging devices 21 and 22 may automatically adjust the exposure based on the measurement result of the photometry device 24b. Alternatively, at least one of the imaging devices 21 and 22 may calculate the luminance value (for example, of a plurality of pixels) of the actually generated image data IMG_2D (or image data IMG_3D) in addition to or instead of the measurement result of the photometric device 24b.
  • the luminance value for example, of a plurality of pixels
  • the exposure may be automatically adjusted based on the average brightness value.
  • at least one of the imaging devices 21 and 22 may automatically adjust the exposure so that the brightness value of the image data IMG_2D (or image data IMG_3D) falls within a desired range.
  • the contrast of the image data IMG_2D (or image data IMG_3D) can be adjusted more appropriately by gamma correction processing so as to emphasize the edge of the workpiece W reflected in the image data IMG_2D (or image data IMG_3D). becomes.
  • the pre-processing performed on at least one of the image data IMG_2D and IMG_3D may include high dynamic range synthesis processing.
  • high dynamic range processing involves compositing multiple image data IMG_2D (or multiple image data IMG_3D) generated by capturing images of the workpiece W under multiple imaging environments with different exposures.
  • the process may include processing for generating image data IMG_2D (or image data IMG_3D) having a wide dynamic range with less blown-out highlights and blown-up shadows.
  • the pre-processing performed on at least one of the image data IMG_2D and IMG_3D may include noise removal processing. As shown in FIG. 30, the noise removal process generates image data IMG_2D (or image data IMG_3D) with less noise by removing noise included in image data IMG_2D (or image data IMG_3D). It may be a process.
  • the pre-processing unit 315b may perform known filter processing on at least one of the image data IMG_2D and IMG_3D as the noise removal processing. For example, the pre-processing unit 315b may perform filter processing on at least one of the image data IMG_2D and IMG_3D using at least one of an averaging filter, a median filter, and an expansion filter.
  • the pre-processing unit 315b may perform known filter processing on at least one of the image data IMG_2D and IMG_3D.
  • the pre-processing unit 315b may perform filter processing using at least one of a smoothing filter and an edge enhancement filter on at least one of the image data IMG_2D and IMG_3D.
  • FIG. 31 is a block diagram showing the configuration of the robot system SYS in the third modification.
  • the third modification of the robot system SYS will be referred to as a "robot system SYSc" to distinguish it from the robot system SYS described above.
  • the robot system SYSc in the third modification is different from the robot systems SYS, SYSa, or SYSb described above in that it includes an imaging unit 2c instead of the imaging unit 2. It's okay.
  • Other characteristics of the robot system SYSc may be the same as other characteristics of the robot system SYS, SYSa or SYSb.
  • the imaging unit 2c in the third modification differs from the imaging unit 2 described above in that it includes an illumination device 24c, a filter 25c, and a filter 26c. Other features of the imaging unit 2c may be the same as other features of the imaging unit 2.
  • the illumination device 24c is a device that can irradiate illumination light onto the object OBJ.
  • the illumination device 24c is a device that can illuminate the object OBJ with illumination light by irradiating the object OBJ with illumination light.
  • the illumination device 24c may illuminate the object OBJ with illumination light that includes a light component in a second wavelength band (for example, a wavelength band of red light).
  • the projection light that the projection device 23 irradiates onto the object OBJ may include a light component in a first wavelength band (for example, a wavelength band of blue light) that is different from the second wavelength band.
  • the projection device 23 may irradiate the object OBJ with projection light that includes a light component in a first wavelength band (for example, a wavelength band of blue light) that is different from the second wavelength band.
  • the projection device 23 is a device that projects a desired projection pattern onto the object OBJ by irradiating the object OBJ with projection light.
  • the projection device 23 may be considered to be illuminating the object OBJ with projection light.
  • the projection device 23 may be considered to be an illumination device that illuminates the object OBJ with projection light. Note that, as described above, when the projection device 23 is considered to be an illumination device, the projection light does not need to be light that can project a desired projection pattern onto the target object.
  • the filter 25c can attenuate the optical component in the first wavelength band.
  • the imaging device 21 images the object OBJ by having the imaging element receive light from the object OBJ via the filter 25c.
  • the filter 25c attenuates the projection light. Therefore, return light (for example, at least one of reflected light and scattered light of the projection light) from the object OBJ to which the projection light has been irradiated (that is, onto which the projection pattern has been projected) is attenuated by the filter 25c.
  • the imaging device 21 is affected by the influence of the projection light emitted from the projection device 23.
  • the object OBJ illuminated with the illumination light emitted from the illumination device 24c can be appropriately imaged without being affected by the illumination light.
  • the filter 26c is capable of attenuating optical components in the second wavelength band.
  • the imaging device 22 images the object OBJ (in other words, the projection pattern projected onto the object OBJ) by having the imaging element receive light from the object OBJ via the filter 26c.
  • the filter 26c attenuates the illumination light. Therefore, return light (for example, at least one of reflected light and scattered light of the illumination light) from the object OBJ irradiated with the illumination light from the illumination device 24c is attenuated by the filter 26c.
  • the image pickup device 22 is unable to use the illumination light emitted from the illumination device 24c.
  • the object OBJ illuminated by the projection light emitted from the projection device 23 (in other words, the projection pattern projected onto the object OBJ) can be appropriately imaged without being affected.
  • the imaging device 21 can image the object OBJ without being affected by the imaging of the object OBJ (the projection pattern projected onto the object OBJ) by the imaging device 22.
  • the imaging device 22 can image the object OBJ (the projection pattern projected onto the object OBJ) without being affected by the imaging of the object OBJ by the imaging device 21. Therefore, the imaging devices 21 and 22 can simultaneously image the object OBJ. In other words, even if the time at which the imaging device 21 images the object OBJ and the time at which the imaging device 22 images the object OBJ (the projection pattern projected onto the object OBJ) are the same, each of the imaging devices 21 and 22 , it is possible to appropriately image the object OBJ.
  • At least one of the robot systems SYS, SYSa, SYSb, and SYSc may adopt other modifications described below.
  • the end effector 13 (for example, a hand gripper or a vacuum clipper) that performs at least one of the holding process and the release process is attached to the robot arm 12. ing.
  • the end effector 13 is not limited to a device that performs at least one of holding processing and release processing, but may be a device that performs other processing on object OBJ.
  • a processing device for processing the object OBJ which is an example of the end effector 13, may be attached to the robot arm 12.
  • the processing device performs at least one of an additional process that adds a new model to the object OBJ, a removal process that removes a part of the object OBJ, a welding process that joins two objects OBJ, and a cutting process that cuts the object OBJ. You may do one.
  • the processing device may process the object OBJ using a tool. In this case, a processing device including tools may be attached to the robot arm 12.
  • the processing device may process the object OBJ by irradiating the object OBJ with an energy beam (for example, light, electromagnetic waves, and charged particle beam).
  • an energy beam for example, light, electromagnetic waves, and charged particle beam
  • a processing device including an irradiation device that irradiates the object OBJ with an energy beam may be attached to the robot arm 12.
  • a processing device as an example of the end effector 13 may perform soldering processing to solder parts to the object OBJ.
  • the processing device may solder the component to the object OBJ using a soldering iron.
  • a processing device including a soldering iron may be attached to the robot arm 12.
  • the processing device may solder the component to the object OBJ by irradiating the solder with an energy beam (eg, light, electromagnetic waves, and charged particle beam).
  • an energy beam eg, light, electromagnetic waves, and charged particle beam
  • a processing device including an irradiation device that irradiates the object OBJ with an energy beam may be attached to the robot arm 12.
  • a measuring device for measuring the object OBJ which is an example of the end effector 13, may be attached to the robot arm 12.
  • the measuring device may be capable of measuring the characteristics of the object OBJ. Examples of the characteristics of the object OBJ include at least one of the shape of the object OBJ, the size of the object OBJ, and the temperature of the object OBJ.
  • the measuring device may measure the object OBJ using a touch probe. In this case, a measurement device including a touch probe may be attached to the robot arm 12.
  • the measuring device may measure the object OBJ by irradiating the object OBJ with an energy beam (for example, light, electromagnetic waves, or charged particle beam).
  • an energy beam for example, light, electromagnetic waves, or charged particle beam
  • a measuring device including an irradiation device that irradiates the object OBJ with an energy beam may be attached to the robot arm 12.
  • the control device 3 When at least one of a processing device and a measuring device as an example of the end effector 13 is attached to the robot arm 12, the control device 3 outputs a control signal for controlling the operation of at least one of the processing device and the measuring device. may be generated.
  • the control device 3 may generate a control signal for controlling the rotation of a tool included in the processing device.
  • the control device 3 may generate a control signal for controlling on/off of an energy beam by an irradiation device included in at least one of the processing device and the measurement device.
  • a discharge device which is an example of the end effector 13, may be attached to the robot arm 12.
  • the discharge device may discharge at least one of an adhesive, a sealant, a paint, and a solder.
  • the discharge device may discharge at least one of an adhesive, a sealant, a paint, and a solder toward the object OBJ.
  • the control device 3 may generate a control signal for controlling at least one of turning on/off the ejection from the ejection device and the amount of ejection.
  • discharging at least one of an adhesive, a sealant, a paint, and a solder onto the object OBJ can also be said to process the object OBJ, so the discharging device may also be referred to as a processing device.
  • the imaging unit 2 includes the imaging device 22 and the projection device 23 in order to generate the image data IMG_3D. However, the imaging unit 2 does not need to include the projection device 23 in order to generate the image data IMG_3D. This is because, as described above, since the imaging device 22 is a stereo camera, a three-dimensional image indicating the three-dimensional position of each of the plurality of points on the object OBJ is obtained from two image data IMG_3D generated by the two image sensors of the stereo camera. This is because the position data WSD can be generated.
  • the three-dimensional position data generation unit 311 calculates the parallax by associating each part (for example, each pixel) of the images indicated by each of the two image data included in the image data IMG_3D using a well-known process.
  • the three-dimensional position data WSD may be generated using a well-known method based on the principle of triangulation using the calculated parallax.
  • the imaging device 22 does not need to be a stereo camera.
  • the imaging device 22 may be a monocular camera that images the object OBJ using a single imaging element.
  • the object OBJ on which the projection pattern is projected is reflected in the image indicated by the image data IMG_3D.
  • the shape of the projection pattern reflected in the image indicated by the image data IMG_3D reflects the three-dimensional shape of the object OBJ onto which the projection pattern is projected. Therefore, even if the imaging device 22 is not a stereo camera, the three-dimensional position data generation unit 311 generates three-dimensional position data using well-known processing based on the projection pattern reflected in the image indicated by the image data IMG_3D. WSD can be generated.
  • the imaging unit 2 may include one of the imaging devices 21 and 22 (that is, a single imaging device) and a projection device 23.
  • image data generated by a single imaging device capturing an image of the object OBJ during a period when the projection device 23 is not projecting the desired projection pattern onto the object OBJ may be used as the image data IMG_2D.
  • image data generated by a single imaging device capturing an image of the object OBJ during a period when the projection device 23 is projecting a desired projection pattern onto the object OBJ may be used as the image data IMG_3D. .
  • the position and orientation calculation unit 312 calculates a position indicating the position and orientation of the object OBJ based on the image data IMG_2D and the three-dimensional position data WSD generated from the image data IMG_3D in step S3 of FIG. Posture information POI0 is generated. Specifically, by performing 2D matching processing, the 2D matching unit 3121 determines the position of the object OBJ in the 2D imaging coordinate system as position Tx (2D), position Ty (2D), position Tz (2D), and orientation Rx. (2D), posture Ry (2D), and posture Rz (2D).
  • the 3D matching unit 3122 determines the position of the object OBJ in the 3D imaging coordinate system as position Tx (3D), position Ty (3D), position Tz (3D), orientation Rx (3D), and orientation. At least one of Ry (3D) and posture Rz (3D) may be calculated.
  • the tracking unit 3123 performs tracking processing to determine the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (2D), the amount of change ⁇ Rx (2D), the amount of change ⁇ Ry (2D), and the amount of change ⁇ Rz.
  • the control device 3 uses the above-mentioned coordinate system information (for example, a transformation matrix) to determine the position in either the 2D imaging coordinate system or the 3D imaging coordinate system. It is not necessarily necessary to perform a process of converting the position to a position within the other coordinate system, either the imaging coordinate system or the 3D imaging coordinate system.
  • the imaging unit 2 includes an imaging device 22 that is a stereo camera, it does not need to include the imaging device 21.
  • image data generated by imaging the object OBJ by one of the two monocular cameras included in the imaging device 22 may be used as the image data IMG_2D.
  • image data representing two images generated by both of the two monocular cameras included in the imaging device 22 capturing the object OBJ may be used as the image data IMG_3D.
  • the position and orientation calculation unit 312 calculates a position indicating the position and orientation of the object OBJ based on the image data IMG_2D and the three-dimensional position data WSD generated from the image data IMG_3D in step S3 of FIG. Posture information POI0 is generated.
  • the 2D matching unit 3121 determines the position of the object OBJ in the 2D imaging coordinate system as position Tx (2D), position Ty (2D), position Tz (2D), and orientation Rx. (2D), posture Ry (2D), and posture Rz (2D).
  • the 3D matching unit 3122 determines the position of the object OBJ in the 3D imaging coordinate system as position Tx (3D), position Ty (3D), position Tz (3D), orientation Rx (3D), and orientation. At least one of Ry (3D) and posture Rz (3D) may be calculated.
  • the tracking unit 3123 performs tracking processing to determine the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (2D), the amount of change ⁇ Rx (2D), the amount of change ⁇ Ry (2D), and the amount of change ⁇ Rz. (2D), or the amount of change ⁇ Tx (3D), the amount of change ⁇ Ty (3D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and the amount of change ⁇ Rz (3D) may be calculated.
  • the control device 3 uses the above-mentioned coordinate system information (for example, a transformation matrix) to determine the position within either the 2D imaging coordinate system or the 3D imaging coordinate system. It is not necessarily necessary to perform a process of converting the position to a position within the other coordinate system, either the 2D imaging coordinate system or the 3D imaging coordinate system.
  • both the imaging devices 21 and 22 are provided on the robot 1 (particularly on the robot arm 12). However, at least one of the imaging devices 21 and 22 may be provided in a different part from the robot 1 (in particular, the robot arm 12). For example, at least one of the imaging devices 21 and 22 may be placed on the ceiling of a building where the robot system SYS is placed.
  • the imaging unit 2 is provided in the robot 1 that includes the robot arm 12.
  • the imaging unit 2 may be provided in a robot that is not equipped with the robot arm 12.
  • the imaging unit 2 may be provided in the robot 1 including a support device capable of suspending the imaging unit 2 from above the object OBJ.
  • the imaging unit 2 may be provided on a support device.
  • the support device may include a plurality of leg members extending upward from the support surface S, and a beam member that connects the plurality of leg members via the upper ends of the plurality of leg members or the vicinity thereof.
  • the robot 1 may be able to move the support device.
  • control device 3 (in particular, the three-dimensional position data generation unit 311) generates the three-dimensional position data WSD from the image data IMG_3D.
  • a device different from the control device 3 may generate the three-dimensional position data WSD from the image data IMG_3D.
  • the imaging unit 2 that generates the image data IMG_3D may generate the three-dimensional position data WSD from the image data IMG_3D.
  • the control device 3 may acquire the three-dimensional position data WSD from a device different from the control device 3.
  • the control device 3 may perform at least one of the above-described 3D matching process and tracking process using the three-dimensional position data WSD acquired from a device different from the control device 3. For example, when the imaging unit 2 generates the three-dimensional position data WSD from the image data IMG_3D, the control device 3 acquires the three-dimensional position data WSD from the imaging unit 2, and performs at least one of the above-mentioned 3D matching processing and tracking processing. You may do one.
  • the control device 3 performs both matching processing and tracking processing. However, the control device 3 does not have to perform the tracking process.
  • the control device 3 may generate the result of the 3D matching process performed by the 3D matching unit 3122 (that is, the position and orientation information POI2) as the position and orientation information POI0.
  • the control device 3 may generate the result of the 2D matching process performed by the 2D matching unit 3121 (that is, the position and orientation information POI1) as the position and orientation information POI0.
  • the control device 3 may generate a robot control signal based on the position and orientation information POI1 or POI2. In this case, the control device 3 may generate and output the robot control signal at the same cycle as the 2D matching cycle or 3D matching cycle, which is the cycle at which the position and orientation of the object OBJ are calculated.
  • the control device 3 when the end effector 13 holds the workpiece W, the control device 3 further generates position and orientation information POI0 indicating at least one of the position and orientation of the workpiece W held by the end effector 13. As mentioned above, this may be allowed.
  • the positional relationship between the end effector 13 and the workpiece W does not change.
  • the positional relationship between the imaging devices 21 and 22 and the work W does not change.
  • the degree of deviation between the position of the workpiece W calculated by the control device 3 and the actual position of the workpiece W is similar to the case where the tracking process described above is performed. It never gets bigger.
  • the control device 3 may generate the position and orientation information POI0 indicating at least one of the position and orientation of the workpiece W held by the end effector 13, without performing the tracking process.
  • the tracking process described above is Even if it is not performed, the degree of deviation between the position of the object OBJ calculated by the control device 3 and the actual position of the object OBJ does not become large, as in the case where the above-described tracking process is performed.
  • the above-described tracking process is not performed.
  • the control device 3 may generate the position and orientation information POI0 indicating at least one of the position and orientation of the object OBJ, without performing the tracking process.
  • the control device 3 can control the position and orientation of the object OBJ to indicate at least one of the position and orientation of the object OBJ without performing tracking processing.
  • Information POI0 may be generated.
  • control device 3 may generate the result of the 3D matching process performed by the 3D matching unit 3122 (that is, the position and orientation information POI2) as the position and orientation information POI0.
  • the position and orientation calculation unit 312 uses the 3D matching unit 3122 After the position and orientation information POI0 is once generated using the result of the 3D matching process (that is, the position and orientation information POI2) performed by Good too. Even in this case, the positional relationship between the imaging devices 21 and 22 and the work W does not change, so the position of the object OBJ does not differ greatly from the position indicated by the generated position and orientation information POI0.
  • the control device 3 may generate the robot control signal based on the initially generated position and orientation information POI0.
  • control device 3 generates a robot control signal for controlling at least one of the robot arm 12 and the end effector 13 to hold the object OBJ based on the initially generated position and orientation information POI0 of the object OBJ. It's okay.
  • control device 3 controls at least the robot arm 12 and the end effector 13 to release (for example, place or fit) another object onto the object OBJ based on the initially generated position and orientation information POI0 of the object OBJ.
  • a robot control signal may be generated to control one of the two.
  • the position and orientation calculation unit 312 calculates the results of the 3D matching process performed by the 3D matching unit 3122 (i.e., After once generating the position and orientation information POI0 using the position and orientation information POI2), it is not necessary to periodically perform the 2D matching process and the 3D matching process (furthermore, the tracking process).
  • control device 3 While the control device 3 performs 2D matching processing, it does not need to perform 3D matching processing and tracking processing. In this case, the control device 3 may generate the result of the 2D matching process performed by the 2D matching unit 3121 (that is, the position and orientation information POI1) as the position and orientation information POI0.
  • the control device 3 may generate the result of the 2D matching process performed by the 2D matching unit 3121 (that is, the position and orientation information POI1) as the position and orientation information POI0.
  • the control device 3 While the control device 3 performs the 3D matching process, it is not necessary to perform the 2D matching process and the tracking process.
  • the 3D matching unit 3122 does not need to determine the initial position and initial orientation of the three-dimensional model WM3 based on the result of the 2D matching process (that is, the position and orientation information POI1). That is, the 3D matching unit 3122 may perform matching processing using the three-dimensional position data WSD as the 3D matching processing without determining the initial position and initial orientation of the three-dimensional model WM3.
  • the control device 3 may generate the result of the 3D matching process performed by the 3D matching unit 3122 (that is, the position and orientation information POI2) as the position and orientation information POI0.
  • the initial position of the three-dimensional model WM3 may be set in advance at the origin of the coordinate system of the three-dimensional model WM3 (the coordinate system of the three-dimensional model data WMD), and the initial posture of the three-dimensional model WM3 is , may be set to a predetermined posture in the coordinate system of the three-dimensional model WM3.
  • the initial position of the three-dimensional model WM3 may be set in advance to the origin of the 3D imaging coordinate system or the 2D imaging coordinate system
  • the initial posture of the three-dimensional model WM3 may be set in advance to the origin of the 3D imaging coordinate system or the 2D imaging coordinate system. may be set to a predetermined posture.
  • the control device 3 As a tracking process, the control device 3 generates two image data IMG_2D#t1 and IMG_2D#t2 that are generated by the imaging device 21 capturing images of the object OBJ at the first time t1 and the second time t2, respectively.
  • a tracking process is performed using the three-dimensional position data WSD generated from the image data IMG_3D generated by the imaging device 22 capturing an image of the object OBJ at the third time t3.
  • control device 3 controls the three-dimensional position data WSD generated from the image data IMG_3D generated by the imaging device 22 capturing an image of the object OBJ at the first time t1 (for example, at least 3D position information of one feature point) and 3D position data WSD (for example, 3D position data
  • the amount of change in at least one of the position and orientation of the object OBJ between the first time t1 and the second time t2 by performing tracking processing using the three-dimensional position information of at least one feature point indicated by the WSD. may be calculated as the change amount information VI.
  • the control device 3 uses the three-dimensional position information of each of the plurality of feature points indicated by the three-dimensional position data WSD generated from the image data IMG_3D at the first time t1 and the second time t2. between the first time t1 and the second time t2 based on the displacement with the three-dimensional position information of each of the plurality of feature points indicated by the three-dimensional position data WSD generated from the image data IMG_3D of the imaging device 22.
  • the amount of change in at least one of the position and orientation of object OBJ may be calculated as change amount information VI. Even in this case, the control device 3 may correct the results of the above-described matching process based on the results of the tracking process.
  • the control device 3 may correct the result of the 3D matching process (that is, the position and orientation information POI2) based on the change amount information VI that is the result of the tracking process.
  • the control device 3 may correct the result of the 2D matching process (that is, the position and orientation information POI1) based on the change amount information VI that is the result of the tracking process.
  • the control device 3 may include a robot control device 14.
  • the control device 3 may function as a robot control device 14.
  • a logical functional block that can function as the robot control device 14 may be implemented within the arithmetic device 31 of the control device 3.
  • the control device 3 may control the operation of the robot 1 based on the robot control signal generated by the control device 3. In this case, the robot 1 does not need to include the robot control device 14.
  • the three-dimensional position data generation unit 311 of the control device 3 generates three-dimensional data from one image data IMG_3D (for example, if the imaging device 22 is a stereo camera, one image data IMG_3D including two image data). Position data WSD is generated. However, the three-dimensional position data generation unit 311 may generate three-dimensional position data WSD from a plurality of image data IMG_3D. For example, the three-dimensional position data generation unit 311 may generate the three-dimensional position data WSD from a plurality of image data IMG_3D captured from different directions of the object OBJ.
  • the signal generation unit 313 changes the object OBJ to a different position with the position of the object OBJ as a starting point.
  • the imaging unit 2 that is, the robot arm 12
  • the signal generation unit 313 generates a robot control signal for operating the robot arm 12 so that the object OBJ can be imaged from different directions using the position of the object OBJ as a starting point, and outputs it to the robot control device 14. It's okay.
  • the imaging device 22 may generate a plurality of image data IMG_3D obtained by imaging the object OBJ from different directions by performing imaging multiple times as the imaging unit 2 moves from the position of the object OBJ as a starting point.
  • the three-dimensional position data generation unit 311 may generate three-dimensional position data WSD for each of the plurality of image data IMG_3D through the above-described processing.
  • the three-dimensional position data generation unit 311 may combine the plurality of generated three-dimensional position data WSD using existing processing.
  • the three-dimensional position data generation unit 311 synthesizes the plurality of point cloud data by performing ICP (Iterative Closest Point) processing. It's okay.
  • the three-dimensional position data generation unit 311 performs a process of aligning the positions of feature points on the depth image indicated by the respective depth image data.
  • a plurality of depth image data may be combined by performing the following steps.
  • the position and orientation calculation unit 312 uses the three-dimensional position data WSD generated by the synthesis process by the three-dimensional position data generation unit 311 to calculate the position and orientation of the object OBJ by performing at least one of the matching process and the tracking process described above.
  • the posture may also be calculated.
  • the signal generation unit 313 generates a robot control signal based on the position and orientation of the object OBJ calculated by the position and orientation calculation unit 312, and outputs it to the robot control device 14 for processing on the object OBJ by the end effector 13. It's okay.
  • a control device that generates a control signal for controlling a robot provided with an imaging unit, the control device comprising: The control device includes: a calculation unit that generates the control signal; an output section that outputs the control signal generated by the calculation section; The arithmetic unit is generating first information indicating at least one of the position and orientation of the object using first image data generated by the imaging unit imaging the object; determined based on the first information and three-dimensional position data that is generated from second image data that is generated when the imaging unit images the object and indicates three-dimensional positions of each of the plurality of points on the object.
  • the first information indicates at least one of the position and orientation of the object within a first coordinate system
  • the position and orientation of the three-dimensional model indicated by the determined three-dimensional model data are within a second coordinate system different from the first coordinate system
  • the control device according to supplementary note 1, wherein the second information indicates at least one of the position and orientation of the object within the second coordinate system.
  • the arithmetic unit is converting the first information within the first coordinate system to at least one of a position and an orientation within the second coordinate system;
  • the control device according to supplementary note 2 wherein the position and orientation of the three-dimensional model are determined based on at least one of the position and orientation in the transformed second coordinate system.
  • the control device according to supplementary note 3 wherein the calculation unit sets the position and orientation in the transformed second coordinate system as the position and orientation of the three-dimensional model.
  • the arithmetic unit is setting at least one of the position and orientation of the three-dimensional model indicated by the three-dimensional model data within the first coordinate system based on the first information; the second coordinate system by converting at least one of the position and orientation of the three-dimensional model within the set first coordinate system to at least one of the position and orientation within the second coordinate system; The control device according to supplementary note 2, wherein at least one of the position and orientation of the three-dimensional model is determined within the three-dimensional model.
  • the calculation unit sets at least one of the position and orientation of the object within the first coordinate system indicated by the first information as at least one of the position and orientation of the three-dimensional model within the first coordinate system. The control device described in Appendix 5.
  • Additional Note 1 The time at which the imaging section images the object in order to generate the first image data and the time at which the imaging section images the object in order to generate the second image data are the same time. 7. The control device according to any one of 6 to 6. [Additional note 8] The control device according to any one of Supplementary Notes 1 to 6, wherein the output unit outputs the control signal generated by the calculation unit based on the second information in a first cycle. [Additional note 9] The calculation unit is configured to generate two first image data generated by the imaging unit imaging the object at first and second times that are different from each other, and the imaging unit imaging the object at a third time.
  • the second time is a time later than the first time, The control device according to appendix 9 or 10, wherein the second time and the third time are the same.
  • the arithmetic unit is Using the three-dimensional position data generated from the second image data generated by the imaging at the third time by the imaging unit, the first axis along the optical axis of the optical system of the imaging unit, the first axis A characteristic point of the object in a direction parallel to the first axis within a first coordinate system defined by a second axis perpendicular to the first axis and a third axis perpendicular to the first and second axes.
  • the control device is generating third information indicating the position, The control device according to any one of Supplementary Notes 9 to 11, wherein the amount of change is calculated using the two pieces of first image data and the third information.
  • the arithmetic unit is generating fourth information indicating the position of the characteristic location at the first time using the first image data generated by the imaging unit imaging the object at the first time; fifth information indicating the position of the characteristic location at the second time using the first image data generated by the imaging unit imaging the object at the second time and the third information; generate, The control device according to appendix 12, wherein the amount of change is calculated using the fourth information and the fifth information.
  • each of the first time, the second time, and the third time is a time when the imaging unit and the object are moving relative to each other.
  • the output unit outputs the control signal generated by the calculation unit based on the second information and the amount of change in a second cycle.
  • the time from when the object is imaged by the imaging unit to when the second information is generated based on the first and second image data is such that the object is
  • the control device according to any one of appendices 9 to 15, which is longer than the time from when the image is captured at the earlier of the times until the amount of change is calculated based on the two first image data.
  • the arithmetic unit is generating the second information in a third cycle;
  • the control device according to any one of Supplementary Notes 9 to 16, wherein the amount of change is calculated in a fourth period that is shorter than the third period.
  • the arithmetic unit calculates that the amount of change is new in the fourth period after the second information is newly generated as one period of the third period until the second information is generated next time.
  • the output unit outputs the control signal generated by the calculation unit based on the second information and the amount of change in a second cycle, The control device according to appendix 17 or 18, wherein the second period is the same as the fourth period.
  • the robot is provided with a processing device, The control device according to any one of Supplementary Notes 1 to 19, wherein the calculation unit determines the object as a target object to be processed by the processing device based on the first image data.
  • the robot is provided with a processing device, The control device according to any one of Supplementary Notes 1 to 20, wherein the calculation unit determines, based on the first image data, whether or not the object is determined as a target object to be processed by the processing device. .
  • the arithmetic unit is generating the first information by an object detection process that detects the object in a first image indicated by the first image data;
  • the arithmetic unit is As the object detection process, a matching process is performed using two-dimensional model data representing a two-dimensional model of the object and the first image data, detecting an edge of the object from the object detected in the matching process; detecting edges in the first image using the first image data; The control device according to attachment 22, wherein the control device determines whether or not to determine the object as the target object based on a first similarity between the edge of the object and the edge in the first image.
  • the arithmetic unit is As the object detection process, a matching process is performed using two-dimensional model data representing a two-dimensional model of the object and the first image data, Based on a second degree of similarity between the two-dimensional model of the object and the first image indicated by the first image data, which is calculated in the matching process, determine whether the object is to be determined as the target object.
  • the control device according to appendix 22 or 23, which determines.
  • the arithmetic unit is As the object detection process, a matching process is performed using two-dimensional model data representing a two-dimensional model of the object and the first image data, Based on a second degree of similarity between the two-dimensional model of the object and the first image indicated by the first image data, which is calculated in the matching process, determine whether the object is to be determined as the target object. judge, determining that the object is determined to be the target object when the first and second similarities exceed a predetermined first threshold; The control device according to attachment 23, wherein the control device determines that the object is not determined as the target object when at least one of the first and second similarities is less than a predetermined first threshold.
  • the arithmetic unit is generating the first information by an object detection process that detects the object in a first image indicated by the first image data; Before determining the object as the target object, a first signal for controlling the robot so that the processing device moves so that the processing device approaches the object detected by the object detection processing is sent to the robot. Generate a control signal based on the first information, of the object in the first image shown by the first image data generated by the imaging unit imaging the object during at least part of the period during which the robot is controlled based on the first signal; The control device according to any one of Supplementary Notes 19 to 25, wherein it is determined whether or not the object is determined as the target object based on the result of the object detection process.
  • the arithmetic unit is generating the first information by an object detection process that detects the first object in a first image indicated by the first image data; A first step for controlling the robot so that the processing device moves so that the processing device approaches the first object detected by the object detection process before determining the first object as the target object. generating a signal as the control signal based on the first information; In a first image shown by the first image data generated by the imaging unit capturing images of the first and second objects during at least a part of the period during which the robot is controlled based on the first signal.
  • the control device Based on the results of the object detection processing for each of the first and second objects in , it is determined whether the first object or the second object is determined as the target object.
  • the control device according to item 1.
  • the arithmetic unit is During at least part of the period during which the robot is controlled based on the first signal, the second object is determined to be the target object by the determination as to whether the first object or the second object is determined as the target object. If the second object is determined as a target object, in order to process the second object with the processing device, a second signal for controlling the robot to change at least one of the position and orientation of the processing device is sent to the control device.
  • the control device according to appendix 27, which generates the signal as a signal.
  • a control device that generates a control signal for controlling a robot provided with an imaging unit, the control device comprising: The control device includes: a calculation unit that generates the control signal; an output section that outputs the control signal generated by the calculation section; The arithmetic unit is Two first image data generated by the imaging unit imaging the object at mutually different first and second times, and two first image data generated by the imaging unit imaging the object at a third time. the amount of change in at least one of the position and orientation of the object between the first and second times based on three-dimensional position data generated from second image data and indicating three-dimensional positions of each of the plurality of points on the object; Calculate, A control device that generates the control signal based on the amount of change.
  • the second time is a time later than the first time
  • the arithmetic unit is Using the three-dimensional position data, a first axis along the optical axis of the optical system of the imaging unit, a second axis perpendicular to the first axis, and a third axis perpendicular to the first and second axes are determined. generating information indicating the position of a feature of the object in a direction parallel to the first axis within a first coordinate system defined by the axis; The control device according to attachment 29 or 30, wherein the amount of change is calculated using the two pieces of first image data and the information.
  • the arithmetic unit is generating second information indicating the position of the characteristic location at the first time using the first image data generated by the imaging unit imaging the object at the first time; third information indicating the position of the characteristic location at the second time using the first image data generated by the imaging unit imaging the object at the second time and the first information; generate, The control device according to attachment 31, wherein the amount of change is calculated using the second information and the third information.
  • control device according to any one of appendices 29 to 33, wherein the output unit outputs the control signal generated by the calculation unit based on the amount of change at a predetermined period.
  • the calculation unit generates the first information by a matching process using the first image data and two-dimensional model data representing a two-dimensional model of the object.
  • Control device [Appendix 36] The control device according to any one of Supplementary Notes 1 to 28 and 35, wherein the calculation unit determines the position and orientation of the three-dimensional model of the object indicated by the three-dimensional model data, based on the first information. .
  • the timing at which the imaging unit images the object to generate image data is the timing when the imaging unit and the object are moving relative to each other.
  • control device [Additional note 40]
  • the second time is a time later than the first time,
  • the control device controls the timing of imaging by the first and second imaging devices so that the second time and the third time are the same. Control device as described.
  • the three-dimensional position data generated from the second image data generated by imaging at the third time indicates the three-dimensional position of each of the plurality of points of the object within the second coordinate system
  • the arithmetic unit is converting the three-dimensional position within the second coordinate system to a three-dimensional position within the first coordinate system
  • the three-dimensional position data generated from the second image data generated by imaging at the third time indicates the three-dimensional position of each of the plurality of points of the object within the first coordinate system
  • the control device according to appendix 12 or 13, wherein the calculation unit generates the third information indicating the position of the characteristic location in a direction parallel to the first axis from the three-dimensional position within the first coordinate system.
  • the three-dimensional position data indicates the three-dimensional position of each of the plurality of points of the object within a second coordinate system
  • the arithmetic unit is converting the three-dimensional position within the second coordinate system to a three-dimensional position within the first coordinate system
  • the three-dimensional position data indicates the three-dimensional position of each of the plurality of points of the object within the first coordinate system
  • the arithmetic unit is determining that the object is determined to be the target object when the first similarity exceeds a predetermined first threshold;
  • the control device according to attachment 23 wherein the control device determines that the object is not determined as the target object when the first similarity is less than a predetermined first threshold.
  • the arithmetic unit is determining that the object is determined to be the target object when the second similarity exceeds a predetermined second threshold; The control device according to attachment 24, wherein the control device determines that the object is not determined as the target object when the second similarity is less than a predetermined second threshold.
  • the arithmetic unit is When it is determined that the object is not determined as the target object, a signal for controlling the robot to move the imaging unit with respect to the object is generated as the control signal. Supplementary notes 20 to 28 and 45 47. The control device according to any one of 46 to 46.
  • the arithmetic unit is generating, as the control signal, a signal for controlling the robot to rotate the imaging device around a desired rotation axis;
  • the control device according to attachment 47 wherein after the robot is controlled based on the signal, the determination is made based on the first image data generated by the imaging unit capturing an image of the object.
  • the arithmetic unit is If it is determined that the object is not selected as the target object as a result of making the determination after the robot is controlled to rotate the imaging unit based on the signal, the image capturing unit is rotated along the desired translation axis. 49.
  • the control device according to attachment 48 wherein a signal for controlling the robot to translate the imaging unit is generated as the control signal.
  • the robot is provided with a holding device capable of holding the object,
  • the control signal is a signal for controlling the holding operation of the holding device, Based on the control signal generated by the calculation unit, the robot is controlled so that the holding device approaches the object, and the holding device is controlled so that the object is held by the holding device,
  • the arithmetic unit is After the object is held by the holding device, the position and orientation of the object held by the holding device are determined using the first image data generated by the imaging unit capturing an image of the object. generating sixth information indicating at least one side; Based on the sixth information, the robot is configured to move the object held by the holding device to a desired position and/or change the posture of the object held by the holding device to a desired posture. 50.
  • a control device that generates a control signal for controlling a robot that is provided with a processing device that processes an object and an imaging unit, The control device includes: a calculation unit that generates the control signal; an output section that outputs the control signal generated by the calculation section; The arithmetic unit is Determining whether or not the object is to be determined as a target object for which the processing is performed by the processing device, based on image data generated by the imaging unit imaging the object; A control device that generates the control signal for performing the processing on the object.
  • the arithmetic unit is The control device according to attachment 51, wherein it is determined whether or not the object is determined as the target object based on a result of an object detection process that detects the object in an image indicated by the image data.
  • the arithmetic unit is As the object detection process, a matching process is performed using two-dimensional model data representing a two-dimensional model of the object and the image data, detecting an edge of the object from the object detected in the matching process; detecting edges in the image using the image data; The control device according to attachment 52, wherein the control device determines whether or not to determine the object as the target object based on a first similarity between the edge of the object and the edge in the image.
  • the arithmetic unit is As the object detection process, a matching process is performed using two-dimensional model data representing a two-dimensional model of the object and the image data, Supplementary note 52: determining whether or not the object is determined as the target object based on a second degree of similarity between the two-dimensional model of the object and the image indicated by the image data, which is calculated in the matching process. or the control device according to 53.
  • the arithmetic unit is As the object detection process, a matching process is performed using two-dimensional model data representing a two-dimensional model of the object and the first image data, determining whether to determine the object as the target object based on a second similarity between the two-dimensional model of the object and the image indicated by the image data, which is calculated in the matching process; determining that the object is determined to be the target object when the first and second similarities exceed a predetermined first threshold; The control device according to attachment 53, wherein the control device determines that the object is not determined as the target object when at least one of the first and second similarities is less than a predetermined first threshold.
  • the arithmetic unit is generating first information indicating at least one of a position and an orientation of the object by an object detection process that detects the object in an image indicated by the image data; Before determining the object as the target object, a first signal for controlling the robot so that the processing device moves so that the processing device approaches the object detected by the object detection processing is sent to the robot.
  • the arithmetic unit is generating first information indicating at least one of a position and an orientation of the first object by an object detection process that detects the first object in an image indicated by the image data; A first step for controlling the robot so that the processing device moves so that the processing device approaches the first object detected by the object detection process before determining the first object as the target object. generating a signal as the control signal based on the first information; The first object in the image shown by the image data generated by the imaging unit taking images of the first and second objects during at least a part of the period during which the robot is controlled based on the first signal.
  • the arithmetic unit is During at least part of the period during which the robot is controlled based on the first signal, the second object is determined to be If the second object is determined as a target object, in order to process the second object with the processing device, a second signal for controlling the robot to change at least one of the position and orientation of the processing device is sent to the control device.
  • the control device according to appendix 57, which generates the signal as a signal.
  • the arithmetic unit is If it is determined that the object is not determined as the target object, a signal for controlling the robot to move the imaging unit with respect to the object is generated as the control signal.
  • the control device according to item 1.
  • the arithmetic unit is generating, as the control signal, a signal for controlling the robot to rotate the imaging device around a desired rotation axis; 59.
  • the control device according to attachment 59 wherein after the robot is controlled based on the signal, the determination is made based on the image data generated by the image capturing unit capturing an image of the object.
  • the arithmetic unit is If it is determined that the object is not selected as the target object as a result of making the determination after the robot is controlled to rotate the imaging unit based on the signal, the image capturing unit is rotated along the desired translation axis.
  • the control device according to attachment 60 wherein a signal for controlling the robot to translate the imaging unit is generated as the control signal.
  • the processing device includes a holding device capable of holding the object,
  • the control signal is a signal for controlling the holding operation of the holding device, Based on the control signal generated by the calculation unit, the robot is controlled so that the holding device approaches the object, and the holding device is controlled so that the object is held by the holding device,
  • the arithmetic unit is After the object is held by the holding device, the imaging unit uses the image data generated by imaging the object to indicate the position and orientation of the object held by the holding device. 1 generate information, Based on the first information, the robot is configured to move the object held by the holding device to a desired position and/or change the posture of the object held by the holding device to a desired posture.
  • the control device according to any one of appendices 51 to 61, which generates the control signal for controlling.
  • the processing device performs processing on at least one of a first object and a second object having at least one of a position and an orientation different from the first object, as the object. Control device as described.
  • the arithmetic unit is The object detection process is a matching process using first two-dimensional model data indicating a two-dimensional model of the first object, second two-dimensional model data indicating a two-dimensional model of the second object, and the image data.
  • the second object is It is determined that it is selected as an object,
  • the first similarity is higher than a predetermined first threshold
  • the first similarity is higher than the second similarity
  • the third similarity is higher than a predetermined second threshold
  • the third similarity is higher than the second similarity.
  • the first object is It is determined that it is selected as an object,
  • the second similarity is higher than a predetermined first threshold, the second similarity is higher than the first similarity, the fourth similarity is higher than a predetermined second threshold, and the fourth similarity is higher than the first similarity.
  • the first object is It is determined that it is selected as an object,
  • the second similarity is higher than a predetermined first threshold, the second similarity is higher than the first similarity, the fourth similarity is higher than a predetermined second threshold, and the fourth similarity is higher than the first similarity.
  • the second object is It is determined that it is selected as an object, the first similarity exceeds a predetermined second threshold, the third similarity exceeds a predetermined first threshold, and the second similarity is below a predetermined second threshold, and/ or determining that the first object is selected as the target object when the fourth similarity is less than a predetermined first threshold; the second similarity exceeds a predetermined second threshold, the fourth similarity exceeds a predetermined first threshold, and the first similarity is less than a predetermined second threshold; and/or determining that the second object is selected as the target object when the third similarity is less than a predetermined first threshold; the first similarity is less than a predetermined second threshold, and/or the third similarity is less than a predetermined first threshold, and the second similarity is less than a predetermined second threshold; and/or when the fourth
  • the second object has the same three-dimensional shape as the first object, The control device according to attachment 64, wherein the first two-dimensional model data and the second two-dimensional model data are two-dimensional model data indicating a two-dimensional model of the three-dimensional shape.
  • a control device that generates a control signal for controlling a robot that is provided with a processing device that processes at least one of a first object and a second object that is different from the first object, and an imaging unit.
  • the control device includes: a calculation unit that generates the control signal; an output section that outputs the control signal generated by the calculation section;
  • the arithmetic unit is Whether or not the first or second object is determined as a target object to be subjected to the processing by the processing device, based on image data generated by the imaging unit capturing images of the first and second objects. Determine, A control device that generates the control signal for performing the processing on the object.
  • the arithmetic unit is The control device according to attachment 66, wherein the control device determines whether or not the object is determined as the target object based on the result of object detection processing that detects the first and second objects in the image indicated by the image data. .
  • the arithmetic unit is The object detection process is a matching process using first two-dimensional model data indicating a two-dimensional model of the first object, second two-dimensional model data indicating a two-dimensional model of the second object, and the image data. and detecting an edge of the first object and an edge of the second object from the first and second objects detected in the matching process; detecting edges in the image using the image data; the first similarity between the edge of the first object and the edge in the image; and the second similarity between the edge of the second object and the edge in the image.
  • the control device according to attachment 67 determines whether or not to determine the second object as the target object.
  • the arithmetic unit is The object detection process is a matching process using first two-dimensional model data indicating a two-dimensional model of the first object, second two-dimensional model data indicating a two-dimensional model of the second object, and the image data. and a third degree of similarity between the two-dimensional model of the first object and the image indicated by the image data, which is calculated in the matching process; and a third similarity between the two-dimensional model of the second object and the image indicated by the image data.
  • the control device according to attachment 67 or 68, wherein the control device determines whether to determine the first or second object as the target object based on a fourth similarity with the object.
  • the arithmetic unit is The object detection process is a matching process using first two-dimensional model data indicating a two-dimensional model of the first object, second two-dimensional model data indicating a two-dimensional model of the second object, and the image data.
  • the arithmetic unit is generating first information indicating at least one of a position and an orientation of the first object by an object detection process that detects the first object in an image indicated by the image data; A first step for controlling the robot so that the processing device moves so that the processing device approaches the first object detected by the object detection process before determining the first object as the target object.
  • the arithmetic unit is generating first information indicating at least one of a position and an orientation of the first object by an object detection process that detects the first object in an image indicated by the image data; A first step for controlling the robot so that the processing device moves so that the processing device approaches the first object detected by the object detection process before determining the first object as the target object. generating a signal as the control signal based on the first information; In the first image shown by the image data generated by the imaging unit capturing images of the first and second objects during at least a part of the period during which the robot is controlled based on the first signal.
  • the control device described in . [Additional note 73]
  • the arithmetic unit is During at least part of the period during which the robot is controlled based on the first signal, the second object is determined to be If the second object is determined as a target object, in order to process the second object with the processing device, a second signal for controlling the robot to change at least one of the position and orientation of the processing device is sent to the control device.
  • the control device according to attachment 72 which generates the signal as a signal.
  • the arithmetic unit is When it is determined that the first and second objects are not determined as the target objects, the controller sends a signal for controlling the robot to move the imaging unit with respect to the first and second objects.
  • the control device according to any one of appendices 66 to 73, wherein the control device generates a signal.
  • the arithmetic unit is generating, as the control signal, a signal for controlling the robot to rotate the imaging device around a desired rotation axis;
  • the control device according to attachment 74 wherein after the robot is controlled based on the signal, the image capturing section performs the determination based on the image data generated by capturing images of the first and second objects.
  • the arithmetic unit is When it is determined that the first and second objects are not selected as the target objects as a result of performing the determination after the robot is controlled to rotate the imaging unit based on the signal, the desired The control device according to attachment 75, wherein the control device generates, as the control signal, a signal for controlling the robot to translate the imaging unit along a translation axis.
  • the processing device includes a holding device capable of holding the first or second object, The control signal is a signal for controlling the holding operation of the holding device, Based on the control signal generated by the calculation unit, the robot is controlled so that the holding device approaches the first or second object, and the holding device holds the first or second object.
  • the holding device is controlled so that The arithmetic unit is After the first or second object is held by the holding device, the holding device uses the image data generated by the imaging unit to image the first or second object. generating first information indicating the position and orientation of the first or second object; Based on the first information, moving the first or second object held by the holding device to a desired position and/or changing the posture of the first or second object held by the holding device 77.
  • the control device according to any one of appendices 66 to 76, wherein the control device generates the control signal for controlling the robot to change to a desired posture.
  • the arithmetic unit is The object detection process is a matching process using first two-dimensional model data indicating a two-dimensional model of the first object, second two-dimensional model data indicating a two-dimensional model of the second object, and the image data. and detecting an edge of the first object and an edge of the second object from the first and second objects detected in the matching process; detecting edges in the image using the image data; a third degree of similarity between the two-dimensional model of the first object and the image indicated by the image data, which is calculated in the matching process; and a third similarity between the two-dimensional model of the second object and the image indicated by the image data.
  • a fourth degree of similarity between the edge of the first object and the edge in the image a fourth degree of similarity between the edge of the first object and the edge in the image; and a second degree of similarity between the edge of the second object and the edge in the image.
  • the first similarity is higher than a predetermined first threshold, the first similarity is higher than the second similarity, the third similarity is higher than a predetermined second threshold, and the
  • the second object is It is determined that it is selected as an object,
  • the first similarity is higher than a predetermined first threshold
  • the first similarity is higher than the second similarity
  • the third similarity is higher than a predetermined second threshold
  • the third similarity is higher than the second similarity.
  • the first object is It is determined that it is selected as an object,
  • the second similarity is higher than a predetermined first threshold, the second similarity is higher than the first similarity, the fourth similarity is higher than a predetermined second threshold, and the fourth similarity is higher than the first similarity.
  • the first object is It is determined that it is selected as an object,
  • the second similarity is higher than a predetermined first threshold, the second similarity is higher than the first similarity, the fourth similarity is higher than a predetermined second threshold, and the fourth similarity is higher than the first similarity.
  • the second object is It is determined that it is selected as an object, the first similarity exceeds a predetermined second threshold, the third similarity exceeds a predetermined first threshold, and the second similarity is below a predetermined second threshold, and/ or determining that the first object is selected as the target object when the fourth similarity is less than a predetermined first threshold; the second similarity exceeds a predetermined second threshold, the fourth similarity exceeds a predetermined first threshold, and the first similarity is less than a predetermined second threshold; and/or determining that the second object is selected as the target object when the third similarity is less than a predetermined first threshold; the first similarity is less than a predetermined second threshold, and/or the third similarity is less than a predetermined first threshold, and the second similarity is less than a predetermined second threshold; and/or when the fourth
  • the control device described in the appendix has the same three-dimensional shape as the first object, The control device according to attachment 78, wherein the first two-dimensional model data and the second two-dimensional model data are two-dimensional model data indicating a two-dimensional model of the three-dimensional shape.
  • the imaging unit includes a first imaging device that outputs the first image data, and a second imaging device that outputs the second image data.
  • the first imaging device is a monocular camera
  • the control device according to attachment 80, wherein the second imaging device is a stereo camera having two monocular cameras different from the monocular camera.
  • the imaging unit includes a first imaging device that outputs the first image data and a second imaging device that outputs the second image data, a first filter that attenuates a light component in a first wavelength band of the light from the object; further comprising: a second filter that attenuates a light component in a second wavelength band different from the first wavelength band of the light from the object;
  • the first imaging device images the object by receiving light from the object via the first filter,
  • an illumination device that illuminates the object with illumination light that includes a light component in the second wavelength band; further comprising a light projection device that projects pattern light including a light component in the first wavelength band onto the object,
  • the control system according to attachment 84 wherein the second imaging device images the object onto which pattern light is projected from the light projection device.
  • a control method for generating a control signal for controlling a robot provided with an imaging unit, the method comprising: Generating first information indicating at least one of a position and an orientation of the object using first image data generated by the imaging unit capturing an image of the object; determined based on the first information and three-dimensional position data that is generated from second image data that is generated when the imaging unit images the object and indicates the three-dimensional positions of each of the plurality of points on the object. generating second information indicating at least one of the position and orientation of the object using three-dimensional model data of the object at the position and orientation; and generating the control signal based on the second information.
  • a control method for generating a control signal for controlling a robot provided with an imaging unit, the method comprising: Two first image data generated by the imaging unit imaging the object at first and second times that are different from each other, and two first image data generated by the imaging unit imaging the object at a third time. the amount of change in at least one of the position and orientation of the object between the first and second times based on three-dimensional position data generated from second image data and indicating three-dimensional positions of each of the plurality of points on the object; and A control method comprising: generating the control signal based on the amount of change.
  • a control method for generating a control signal for controlling a robot equipped with a processing device that processes an object and an imaging unit comprising: Determining whether or not the object is to be determined as a target object to be subjected to the processing by the processing device, based on image data generated by the imaging unit imaging the object; and generating the control signal for performing the process on the target object.
  • SYS Robot system 1 Robot 12 Robot arm 13 End effector 14 Robot control device 2 Imaging unit 21, 22 Imaging device 23 Projection device 24c Illumination device 25c, 26c Filter 3 Control device 31
  • Arithmetic device 311 Three-dimensional position data generation unit 312 Position and orientation calculation Section 3121 2D matching section 3122 3D matching section 3123 Tracking section 3125 Coordinate transformation section 313 Signal generation section 314a Object determination section 315b
  • Pre-processing section 33 Communication device OBJ Object W Work T Mounting device IMG_2D, IMG_3D Image data IMG_2M Two-dimensional model data WSD 3D position data WMD 3D model data WM 3D model

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Human Computer Interaction (AREA)
  • Signal Processing (AREA)
  • Quality & Reliability (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Manipulator (AREA)
  • Length Measuring Devices By Optical Means (AREA)
PCT/JP2022/019364 2022-04-28 2022-04-28 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム Ceased WO2023209974A1 (ja)

Priority Applications (6)

Application Number Priority Date Filing Date Title
JP2024517781A JPWO2023209974A1 (https=) 2022-04-28 2022-04-28
CN202280096481.3A CN119300961A (zh) 2022-04-28 2022-04-28 控制装置、控制系统、机器人系统、控制方法及计算机程序
EP22940248.2A EP4516465A4 (en) 2022-04-28 2022-04-28 CONTROL DEVICE, CONTROL SYSTEM, ROBOTIC SYSTEM, CONTROL METHOD, AND COMPUTER PROGRAM
PCT/JP2022/019364 WO2023209974A1 (ja) 2022-04-28 2022-04-28 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム
KR1020247036541A KR20250003671A (ko) 2022-04-28 2022-04-28 제어 장치, 제어 시스템, 로봇 시스템, 제어 방법 및 컴퓨터 프로그램
US18/859,185 US20250326125A1 (en) 2022-04-28 2022-04-28 Control apparatus, control system, robot system, control method, and computer program

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/019364 WO2023209974A1 (ja) 2022-04-28 2022-04-28 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム

Publications (1)

Publication Number Publication Date
WO2023209974A1 true WO2023209974A1 (ja) 2023-11-02

Family

ID=88518160

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/019364 Ceased WO2023209974A1 (ja) 2022-04-28 2022-04-28 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム

Country Status (6)

Country Link
US (1) US20250326125A1 (https=)
EP (1) EP4516465A4 (https=)
JP (1) JPWO2023209974A1 (https=)
KR (1) KR20250003671A (https=)
CN (1) CN119300961A (https=)
WO (1) WO2023209974A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2026069666A1 (ja) * 2024-09-30 2026-04-02 株式会社ニコン 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5088278B2 (ja) * 2008-09-17 2012-12-05 株式会社安川電機 物体検出方法と物体検出装置およびロボットシステム
US20130230235A1 (en) 2010-11-19 2013-09-05 Canon Kabushiki Kaisha Information processing apparatus and information processing method
WO2020022041A1 (ja) * 2018-07-23 2020-01-30 オムロン株式会社 制御システム、制御方法およびプログラム
JP6784991B2 (ja) * 2016-06-07 2020-11-18 国立大学法人 岡山大学 ワーク検知システム及び衣服検知システム
JP2021115651A (ja) * 2020-01-23 2021-08-10 三菱重工業株式会社 マニピュレータ、自動化方法及びプログラム

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9279661B2 (en) * 2011-07-08 2016-03-08 Canon Kabushiki Kaisha Information processing apparatus and information processing method
US9437005B2 (en) * 2011-07-08 2016-09-06 Canon Kabushiki Kaisha Information processing apparatus and information processing method
JP6444027B2 (ja) * 2013-12-09 2018-12-26 キヤノン株式会社 情報処理装置、情報処理装置の制御方法、情報処理システムおよびプログラム
WO2023127021A1 (ja) * 2021-12-27 2023-07-06 株式会社ニコン 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5088278B2 (ja) * 2008-09-17 2012-12-05 株式会社安川電機 物体検出方法と物体検出装置およびロボットシステム
US20130230235A1 (en) 2010-11-19 2013-09-05 Canon Kabushiki Kaisha Information processing apparatus and information processing method
JP6784991B2 (ja) * 2016-06-07 2020-11-18 国立大学法人 岡山大学 ワーク検知システム及び衣服検知システム
WO2020022041A1 (ja) * 2018-07-23 2020-01-30 オムロン株式会社 制御システム、制御方法およびプログラム
JP2021115651A (ja) * 2020-01-23 2021-08-10 三菱重工業株式会社 マニピュレータ、自動化方法及びプログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4516465A4

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2026069666A1 (ja) * 2024-09-30 2026-04-02 株式会社ニコン 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム

Also Published As

Publication number Publication date
EP4516465A1 (en) 2025-03-05
JPWO2023209974A1 (https=) 2023-11-02
US20250326125A1 (en) 2025-10-23
EP4516465A4 (en) 2026-03-11
CN119300961A (zh) 2025-01-10
KR20250003671A (ko) 2025-01-07

Similar Documents

Publication Publication Date Title
US10805546B2 (en) Image processing system, image processing device, and image processing program
JP5949242B2 (ja) ロボットシステム、ロボット、ロボット制御装置、ロボット制御方法、およびロボット制御プログラム
JP5602392B2 (ja) 情報処理装置、情報処理方法およびプログラム
US11667036B2 (en) Workpiece picking device and workpiece picking method
CN110136208A (zh) 一种机器人视觉伺服系统的联合自动标定方法及装置
JP5665333B2 (ja) 情報処理装置および情報処理装置の制御方法
WO2023127021A1 (ja) 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム
JP7331527B2 (ja) ロボットの動作を決定する装置、方法およびプログラム
CN110587592A (zh) 机器人控制装置、机器人控制方法及计算机可读记录介质
US10656097B2 (en) Apparatus and method for generating operation program of inspection system
US12058468B2 (en) Image capturing apparatus, image processing apparatus, image processing method, image capturing apparatus calibration method, robot apparatus, method for manufacturing article using robot apparatus, and recording medium
JP2014188617A (ja) ロボット制御システム、ロボット、ロボット制御方法及びプログラム
CN109773777B (zh) 把持方法、把持系统以及存储介质
WO2023209974A1 (ja) 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム
JP6841297B2 (ja) ビジュアルサーボシステム
JP2012078105A (ja) 姿勢制御装置、制御方法及びプログラム
CN117957097A (zh) 位置姿势测量系统
CN119784839A (zh) 机械臂位姿确定方法、设备、存储介质及程序产品
JP7660686B2 (ja) ロボット制御装置、ロボット制御システム、及びロボット制御方法
CN114902281B (zh) 图像处理系统
US20260102920A1 (en) Arithmetic apparatus, arithmetic system, robot system, arithmetic method, and computer program
JP7401250B2 (ja) 画像処理装置、制御方法およびプログラム
WO2026069666A1 (ja) 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム
US20250196351A1 (en) Robot system
JP7757756B2 (ja) 画像処理装置、ロボット制御システムおよび制御方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22940248

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2024517781

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 18859185

Country of ref document: US

WWE Wipo information: entry into national phase

Ref document number: 202280096481.3

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 2022940248

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022940248

Country of ref document: EP

Effective date: 20241128

WWP Wipo information: published in national office

Ref document number: 202280096481.3

Country of ref document: CN

WWP Wipo information: published in national office

Ref document number: 18859185

Country of ref document: US