WO2023127021A1 - 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム - Google Patents

制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム Download PDF

Info

Publication number
WO2023127021A1
WO2023127021A1 PCT/JP2021/048572 JP2021048572W WO2023127021A1 WO 2023127021 A1 WO2023127021 A1 WO 2023127021A1 JP 2021048572 W JP2021048572 W JP 2021048572W WO 2023127021 A1 WO2023127021 A1 WO 2023127021A1
Authority
WO
WIPO (PCT)
Prior art keywords
axis
amount
change
imaging
image data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/048572
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
幸司 細見
日雅留 繁野
アシク イフテカル
牧人 宮野
智志 長谷川
晃平 三村
大輔 小柳
友介 森野
英之 佐野
颯 清水
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nikon Corp
Original Assignee
Nikon Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nikon Corp filed Critical Nikon Corp
Priority to KR1020247018276A priority Critical patent/KR20240130076A/ko
Priority to PCT/JP2021/048572 priority patent/WO2023127021A1/ja
Priority to US18/724,200 priority patent/US20250214249A1/en
Priority to EP21969905.5A priority patent/EP4458526A4/en
Priority to JP2023570515A priority patent/JPWO2023127021A1/ja
Priority to CN202180104846.8A priority patent/CN118369192A/zh
Publication of WO2023127021A1 publication Critical patent/WO2023127021A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1694—Program controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J15/00—Gripping heads and other end effectors
    • B25J15/0052—Gripping heads and other end effectors multiple gripper units or multiple end effectors
    • B25J15/0066—Gripping heads and other end effectors multiple gripper units or multiple end effectors with different types of end effectors, e.g. gripper and welding gun
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
    • B25J19/02—Sensing devices
    • B25J19/021—Optical sensing devices
    • B25J19/023—Optical sensing devices including video camera means
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1656—Program controls characterised by programming, planning systems for manipulators
    • B25J9/1664—Program controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40613—Camera, laser scanner on end effector, hand eye manipulator, local

Definitions

  • the present invention for example, relates to the technical field of control devices, control systems, robot systems, control methods, and computer programs capable of generating control signals for controlling robots.
  • Patent Document 1 describes an example of a control device that calculates the position of an object to be processed by the robot and controls the robot based on the calculated position. Such a control device is required to accurately calculate the position of an object.
  • a processing device that processes an object, a first imaging device that outputs first image data, and a second imaging device that outputs second image data
  • the processing A control device that generates a control signal for controlling a robot arm that moves the device and the first and second imaging devices, comprising: a computing unit that generates the control signal; an output unit for outputting a control signal, wherein the computing unit receives the image obtained by the first imaging device capturing an image of the object during relative movement between the first and second imaging devices and the object; Based on the first image data, a first direction parallel to the first axis and the second axis within a first coordinate system defined by a first axis, a second axis, and a third axis orthogonal to each other and a position of the object in at least one of a second direction parallel to and a third direction parallel to the third axis, and one of the first axis, the second axis, and the third axis
  • the second image data obtained by calculating a
  • a second coordinate system defined by mutually orthogonal fourth, fifth, and sixth axes and different from the first coordinate system, based on shape data representing the three-dimensional shape of the object generated from in at least one of a fourth direction parallel to the fourth axis, a fifth direction parallel to the fifth axis, and a sixth direction parallel to the sixth axis; calculating a second position including at least one of a posture of the object about at least one of the fourth axis, the fifth axis, and the sixth axis; A control device for generating the control signal based on the position, wherein the output unit outputs the control signal generated by the calculation unit based on the first position and the second position at a first predetermined cycle. be done.
  • a control system comprising the control device provided by the first aspect, the first imaging device, and the second imaging device.
  • a robot system including the control device provided by the first aspect, the first imaging device, the second imaging device, and the robot arm.
  • a processing device that processes an object, a first imaging device that outputs first image data, and a second imaging device that outputs second image data
  • the processing A control method for generating control signals for controlling a robot arm that moves an apparatus and said first and second imaging devices, wherein said within a first coordinate system defined by a first axis, a second axis, and a third axis that are orthogonal to each other, based on the first image data obtained by imaging the object by the first imaging device , a position of the object in at least one of a first direction parallel to the first axis, a second direction parallel to the second axis, and a third direction parallel to the third axis; calculating a first position comprising at least one of a pose of the object about at least one of a first axis, the second axis, and the third axis; A fourth axis, a fifth axis, and a sixth axis that are orthogonal to each other are generated from the second
  • calculating a second position comprising one; generating the control signal based on the first position and the second position; and generating the control signal based on the first position and the second position. and outputting a signal at a predetermined cycle.
  • a computer program that causes a computer to execute the control method provided by the fourth aspect.
  • FIG. 1 is a block diagram showing the configuration of the robot system of this embodiment.
  • FIG. 2 is a side view showing the appearance of the robot of this embodiment.
  • FIG. 3 is a block diagram showing the configuration of the control device of this embodiment.
  • FIG. 4 is a flowchart showing the flow of robot control processing.
  • Each of FIGS. 5(a) to 5(d) shows the state at a certain point in time during the gripping process for gripping the workpiece placed on the placing device that is moving on the support surface.
  • FIG. 4 is a side view showing the positional relationship between the robot and the work;
  • FIGS. 6(a) to 6(d) illustrates the robot and workpiece at a point in time during the placement process for placing the workpiece onto the placement device moving on the support surface.
  • FIG. 7 is a block diagram showing logical processing blocks for calculating the position of the workpiece W by performing matching processing and tracking processing.
  • FIG. 8 schematically shows the 2D matching process.
  • FIG. 9 is a timing chart showing the relationship between the timing when the imaging device generates the workpiece image and the timing when the 2D matching section performs the 2D matching process.
  • FIG. 10 schematically shows the 3D matching process.
  • FIG. 11 is a timing chart showing the relationship between the timing at which the imaging device generates a work image and the timing at which the 3D matching section performs 3D matching processing.
  • FIG. 12 schematically shows 3D matching target data, which is a part of work shape data.
  • FIG. 13 schematically shows the 2D tracking process.
  • FIG. 14 is a timing chart showing the relationship between the timing at which the imaging device generates a work image and the timing at which the 2D tracking unit performs 2D tracking processing.
  • FIG. 15 schematically shows 3D tracking processing.
  • FIG. 16 is a timing chart showing the relationship between the timing at which the imaging device generates a workpiece image and the timing at which the 3D tracking section performs 3D tracking processing.
  • FIG. 17 is a timing chart showing the relationship between the timing at which the 2D matching section performs 2D matching processing and the timing at which the 2D tracking section performs 2D tracking processing.
  • FIG. 18 is a block diagram showing the configuration of the control device in the first modified example.
  • FIG. 19 schematically shows the plane removal process.
  • FIG. 20 is a block diagram showing the configuration of the imaging unit.
  • FIG. 21 schematically shows high dynamic range synthesis processing.
  • FIG. 22 schematically shows noise removal processing.
  • FIG. 23 is a block diagram showing the configuration of the robot system in the second modified example.
  • Embodiments of a control device, a control system, a robot system, a control method, and a computer program will be described below with reference to the drawings. Embodiments of the control device, the control system, the robot system, the control method, and the computer program will be described below using the robot system SYS.
  • FIG. 1 is a block diagram showing the overall configuration of the robot system SYS.
  • the robot system SYS includes a robot 1, an imaging unit 2, and a control device 3.
  • the robot 1 is a device capable of performing predetermined processing on a work W, which is an example of an object.
  • An example of a robot 1 is shown in FIG.
  • FIG. 2 is a side view showing the appearance of the robot 1.
  • the robot 1 includes, for example, a base 11, a robot arm 12, an end effector 13, and a robot controller .
  • the base 11 is a base member of the robot 1 .
  • the base 11 is arranged on a support surface S such as a floor surface.
  • the base 11 may be fixed to the support surface S.
  • the base 11 may be movable with respect to the support surface S.
  • the base 11 may be self-propelled on the support surface S.
  • the base 11 may be installed on an automatic guided vehicle (AGV).
  • FIG. 2 shows an example in which the base 11 is fixed to the support surface S. As shown in FIG.
  • the robot arm 12 is attached to the base 11.
  • the robot arm 12 is a device in which a plurality of links 121 are connected via joints 122 .
  • the joint 122 contains an actuator.
  • Link 121 may be rotatable about an axis defined by joint 122 by an actuator built into joint 122 .
  • At least one link 121 may be extendable along the direction in which the link 121 extends.
  • a device including a device in which a plurality of links 121 are connected via joints 122 and the base 11 may be called a robot arm 12 .
  • the end effector 13 is attached to the robot arm 12.
  • the end effector 13 is attached to the tip of the robot arm 12 .
  • the end effector 13 is movable by movement of the robot arm 12 . That is, the robot arm 12 moves the end effector 13 .
  • the end effector 13 is a device that performs a predetermined process on the workpiece W.
  • the end effector 13 may perform gripping processing for gripping the workpiece W.
  • the end effector 13 may perform placement processing for placing the workpiece W gripped by the end effector 13 at a desired position.
  • the end effector 13 may perform a fitting process for fitting the work W gripped by the end effector 13 into another object different from the work W.
  • a hand gripper capable of gripping the work W by physically sandwiching the work W and a work At least one vacuum gripper capable of gripping W is included.
  • FIG. 2 shows an example in which the end effector 13 is a hand gripper.
  • FIG. 2 shows an example in which the robot 1 is a robot having a robot arm 12 (that is, a vertical articulated robot).
  • the robot 1 may be a robot different from the vertical articulated robot.
  • the robot 1 may be a SCARA robot (that is, a horizontal articulated robot).
  • robot 1 may be a parallel link robot.
  • the robot 1 may be a dual-arm robot having two robot arms 12 .
  • the robot 1 may be a Cartesian robot.
  • the work W to be processed by the end effector 13 may be placed on the placement device 4 .
  • the mounting device 4 may be arranged on the support surface S.
  • the mounting device 4 may be fixed to the support surface S.
  • the mounting device 4 may be movable with respect to the support surface S.
  • the mounting device 4 may be self-propelled on the support surface S.
  • FIG. 2 shows an example in which the mounting device 4 can move on the support surface S by itself.
  • the mounting device 4 may be called an automatic guided vehicle (AGV).
  • the mounting device 4 may be called a belt conveyor.
  • the robot control device 14 controls the motion of the robot 1.
  • robot controller 14 may control the movement of robotic arm 12 .
  • robot controller 14 may control motion of robot arm 12 such that desired link 121 rotates about an axis defined by desired joint 122 .
  • the robot controller 14 may control the operation of the robot arm 12 so that the end effector 13 attached to the robot arm 12 is positioned at a desired position (in other words, moves).
  • the robot controller 14 may control the motion of the end effector 13 .
  • the robot control device 14 may control the operation of the end effector 13 so that the end effector 13 grips the workpiece W at desired timing.
  • the robot controller 14 may control the operation of the end effector 13 so that the end effector 13 places the work W at a desired position at a desired timing (that is, releases the gripped work W). .
  • the robot controller 14 may control the timing of opening and closing the hand gripper.
  • the robot controller 14 may control the timing of turning on/off the vacuum of the vacuum gripper.
  • the imaging unit 2 images a target object (for example, work W).
  • the imaging unit 2 includes an imaging device 21 , an imaging device 22 , and a projection device 23 to capture an image of the workpiece W.
  • the imaging device 21 is a camera capable of imaging a target object (for example, work W).
  • the image capturing device 21 captures an image of the work W to generate work image data IMG_2D.
  • Work image data IMG_2D generated by the imaging device 21 is output from the imaging device 21 to the control device 3 .
  • the imaging device 21 is a monocular camera.
  • the imaging device 21 can image the workpiece W using a monocular camera (in other words, an imaging device).
  • the imaging device 21 is not limited to a monocular camera.
  • the imaging device 21 may be a stereo camera capable of imaging the workpiece W using two monocular cameras, or may include three or more monocular cameras.
  • the imaging device 21 may be at least one of a light field camera, a plenoptic camera, and a multispectral camera.
  • the imaging device 22 is a camera capable of imaging a target object (for example, work W).
  • the imaging device 22 is a stereo camera.
  • the imaging device 22 is a stereo camera capable of imaging the workpiece W using two monocular cameras (in other words, two imaging elements).
  • the image capturing device 22 captures an image of the work W to generate work image data IMG_3D.
  • the imaging device 22 since the imaging device 22 is a stereo camera, the imaging device 22 generates work image data IMG_3D including two image data generated by two monocular cameras.
  • Work image data IMG_3D generated by the imaging device 22 is output from the imaging device 22 to the control device 3 .
  • the imaging device 22 is not limited to a stereo camera.
  • the imaging device 22 may be a monocular camera or may include three or more monocular cameras.
  • the imaging device 22 may be at least one of a light field camera, a plenoptic camera, and a multispectral camera.
  • the projection device 23 is a device capable of irradiating a target object (for example, a work W) with projection light.
  • the projection device 23 is a device capable of projecting a desired projection pattern onto the work W by irradiating the work W with projection light.
  • the desired projected pattern may include, for example, random patterns.
  • the random pattern may be a projected pattern having a different pattern for each unit irradiation area.
  • the random pattern may include random dot patterns.
  • the desired projected pattern is not limited to a random pattern, and may include, for example, a one-dimensional or two-dimensional grid pattern, or a pattern different from the grid pattern.
  • the imaging device 22 images the work W on which the projection pattern is projected.
  • the image indicated by the workpiece image data IMG_3D includes the workpiece W on which the projection pattern is projected.
  • the imaging device 21 does not have to image the work W on which the projection pattern is projected.
  • the imaging device 21 may capture an image of the work W on which no projection pattern is projected.
  • the image indicated by the workpiece image data IMG_2D may not include the workpiece W on which the projection pattern is projected.
  • the image indicated by the workpiece image data IMG_2D may include the workpiece W on which the projection pattern is not projected.
  • the projection device 23 may be regarded as illuminating the work W with the projection light by irradiating the work W with the projection light.
  • the projection device 23 may function as an illumination device that illuminates the workpiece W.
  • the projection light may be referred to as illumination light.
  • the projection light does not have to be light that can project a desired projection pattern onto the workpiece W.
  • the imaging unit 2 is provided on the robot arm 12, similar to the end effector 13. That is, the imaging devices 21 and 22 and the projection device 23 are attached to the robot arm 12 .
  • the imaging devices 21 and 22 and the projection device 23 may be attached to the tip of the robot arm 12 in the same manner as the end effector 13 .
  • imaging devices 21 and 22 and projection device 23 are movable by the movement of robot arm 12 . That is, the robot arm 12 moves the imaging devices 21 and 22 and the projection device 23 .
  • the imaging unit 2 may not be provided on the robot arm 12 .
  • the imaging unit 2 may be provided at any position where the workpiece W can be irradiated with the projection light and the workpiece W can be imaged.
  • At least one of the imaging device 21, the imaging device 22, and the projection device 23 of the imaging unit 2 is provided on the robot arm 12, and the imaging device 21, the imaging device 22, and the projection device 23 of the imaging unit 2 are provided. may be provided at a location different from the robot arm 12 .
  • the imaging devices 21 and 22 may image the workpiece W while synchronizing with each other.
  • the imaging devices 21 and 22 may image the work W at the same time. That is, the imaging devices 21 and 22 may image the work W so that the time when the imaging device 21 images the work W and the time when the imaging device 22 images the work W are the same.
  • the imaging devices 21 and 22 do not have to image the work W at the same time. That is, the imaging devices 21 and 22 may image the work W such that the time when the imaging device 21 images the work W and the time when the imaging device 22 images the work W are different.
  • the control device 3 performs robot control processing.
  • the robot control process is a process of generating a robot control signal for controlling the robot 1 .
  • the control device 3 generates robot control signals based on the workpiece image data IMG_2D and IMG_3D output from the imaging unit 2 .
  • the control device 3 calculates the position of the workpiece W within the global coordinate system of the robot system SYS based on the workpiece image data IMG_2D and IMG_3D, and controls the robot based on the calculated position of the workpiece W.
  • Generate a signal For example, the global coordinate system can also be said to be a coordinate system based on the robot 1 .
  • the control device 3 Based on the work image data IMG_2D and IMG_3D, the control device 3 operates within a coordinate system different from the global coordinate system of the robot system SYS (for example, within a 2D imaging coordinate system, a 3D imaging coordinate system, or a common coordinate system, which will be described later).
  • the position of the workpiece W within the system may be calculated, and the robot control signal may be generated based on the calculated position of the workpiece W.
  • control device 3 and the imaging unit 2 are used to control the robot 1. Therefore, the system including the control device 3 and the imaging unit 2 may be referred to as a robot control system or control system.
  • the robot control signal generated by the control device 3 is output to the robot control device 14 of the robot 1.
  • the robot control device 14 controls the operation of the robot 1 (for example, controls the driving of the actuator built in the joint 122 of the robot arm 12) based on the robot control signal generated by the control device 3. Therefore, the robot control signal may include a signal for controlling the motion of the robot 1 .
  • the robot control signal may include a signal that can be used as is for the robot controller 14 to control the operation of the robot 1.
  • the robot control signal may include a signal that can be used as it is as a robot drive signal used by the robot controller 14 to control the operation of the robot 1 .
  • the robot control device 14 may use the robot control signal as it is to control the motion of the robot 1 .
  • the control device 3 generates a drive signal for an actuator built in the joint 122 of the robot arm 12 as a robot control signal
  • the robot control device 14 generates a robot control signal generated by the control device 3 ( That is, the drive signal of the actuator built in the joint 122 of the robot arm 12 may be used as it is to control the actuator built in the joint 122 of the robot arm 12 .
  • the robot controller 14 can move the robot arm 12 .
  • the control device 3 generates, as a robot control signal, a drive signal for an actuator that moves the end effector 13 attached to the robot arm 12, and the robot control device 14 generates a robot control signal generated by the control device 3 (that is, an end effector).
  • the drive signal for the actuator that moves the effector 13 may be used as is to control the actuator that moves the end effector 13 .
  • the robot controller 14 can move the end effector 13 .
  • the robot control signals may include signals that the robot controller 14 can use to generate robot drive signals for controlling the operation of the robot 1 .
  • a signal that can be used by the robot control device 14 to generate a robot drive signal for controlling the motion of the robot 1 can be said to be a signal that the robot control device 14 controls the motion of the robot 1 as a basis.
  • the robot control device 14 may generate a signal for controlling the motion of the robot 1 based on the robot control signal, and may control the motion of the robot 1 based on the generated signal.
  • the robot control device 14 generates a drive signal for an actuator built in the joint 122 of the robot arm 12 based on the robot control signal, and based on the generated signal, drives the actuator built in the joint 122 of the robot arm 12.
  • Actuators may be controlled.
  • the robot controller 14 can move the robot arm 12 .
  • the robot control device 14 may generate a drive signal for an actuator that moves the end effector 13 based on the robot control signal, and control the actuator that moves the end effector 13 based on the generated signal.
  • the robot controller 14 can move the end effector 13 .
  • a signal that can be used by the robot control device 14 to generate a signal for controlling the operation of the robot 1 may be a signal representing the position of the workpiece W in the global coordinate system.
  • the robot control device 14 causes the end effector 13 to approach the work W in the global coordinate system represented by the robot control signal (that is, the robot 1 and the work W
  • a drive signal for the actuator built in the robot joint 122 of the robot arm 12 may be generated so that the positional relationship of the robot arm 12 becomes a desired positional relationship, and the motion of the robot 1 may be controlled based on the generated signal.
  • a signal that can be used by the robot control device 14 to generate a signal for controlling the operation of the robot 1 is a signal representing a desired positional relationship between the robot 1 and the workpiece W in the global coordinate system. good too.
  • the robot control device 14 controls the robot arm 12 based on the robot control signal so that the desired positional relationship between the robot 1 and the workpiece W in the global coordinate system is represented by the robot control signal.
  • a drive signal for an actuator built in the robot joint 122 may be generated, and the motion of the robot 1 may be controlled based on the generated signal.
  • the signals that the robot controller 14 can use to generate signals for controlling the motion of the robot 1 may include signals representing the desired position of the end effector 13 in the global coordinate system.
  • the robot control device 14 moves the end effector 13 to a desired position in the global coordinate system represented by the robot control signal (i.e., the robot 1 and the workpiece W).
  • a drive signal for the actuator built in the robot joint 122 of the robot arm 12 is generated so that the positional relationship between the robot arm 12 and the robot joint 122 becomes a desired positional relationship, and the operation of the robot 1 is controlled based on the generated signal. good.
  • the robot control device 14 moves the end effector 13 to a position where the end effector 13 can grip the workpiece W located at the position indicated by the robot control signal.
  • a drive signal for an actuator built in the robot joint 122 of the robot arm 12 may be generated, and the motion of the robot 1 may be controlled based on the generated signal.
  • the robot control device 14 causes the end effector 13 that has gripped the work W positioned at the position represented by the robot control signal to move the end effector 13 to the placement device on which the work W was placed. 4
  • a drive signal for an actuator built in the robot joint 122 of the robot arm 12 may be generated, and the motion of the robot 1 may be controlled based on the generated signal.
  • a signal that can be used by the robot controller 14 to generate a signal for controlling the operation of the robot 1 is, for example, a signal representing a desired position of the tip of the robot arm 12 in the global coordinate system.
  • the coordinate system used as a reference in the robot control signal may be a coordinate system other than the global coordinate system (for example, within a 2D imaging coordinate system, within a 3D imaging coordinate system, or within a common coordinate system, which will be described later).
  • FIG. 3 is a block diagram showing the configuration of the control device 3. As shown in FIG.
  • the control device 3 includes an arithmetic device 31, a storage device 32, and a communication device 33. Furthermore, the control device 3 may comprise an input device 34 and an output device 35 . However, the control device 3 does not have to include at least one of the input device 34 and the output device 35 . Arithmetic device 31 , storage device 32 , communication device 33 , input device 34 , and output device 35 may be connected via data bus 36 .
  • the computing device 31 includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array).
  • Arithmetic device 31 reads a computer program.
  • arithmetic device 31 may read a computer program stored in storage device 32 .
  • the computing device 31 may read a computer program stored in a computer-readable non-temporary recording medium using a recording medium reading device (not shown) included in the control device 3 .
  • the computing device 31 may acquire (that is, download) a computer program from a device (not shown) arranged outside the control device 3 via the communication device 33 (or other communication device). or read).
  • Arithmetic device 31 executes the read computer program.
  • logical functional blocks for executing the processing (for example, the robot control processing described above) that should be performed by the control device 3 are realized in the arithmetic device 31 .
  • the arithmetic device 31 can function as a controller for realizing logical functional blocks for executing the processing that the control device 3 should perform.
  • a computing model that can be constructed by machine learning may be implemented in the computing device 31 by the computing device executing a computer program.
  • An example of an arithmetic model that can be constructed by machine learning is an arithmetic model that includes a neural network (so-called artificial intelligence (AI)).
  • learning the computational model may include learning neural network parameters (eg, at least one of weights and biases).
  • the arithmetic device 31 may use the arithmetic model to execute the robot control process.
  • the action of executing the robot control process may include the action of executing the robot control process using the arithmetic model.
  • the computing device 31 may be equipped with a computing model that has been constructed by off-line machine learning using teacher data.
  • the calculation model installed in the calculation device 31 may be updated on the calculation device 31 by online machine learning.
  • the arithmetic device 31 may be implemented in a device external to the arithmetic device 31 (that is, a device provided outside the control device 3) in addition to or instead of the arithmetic model implemented in the arithmetic device 31. may be used to perform the robot control process.
  • Recording media for recording computer programs executed by the arithmetic unit 31 include CD-ROMs, CD-Rs, CD-RWs, flexible disks, MOs, DVD-ROMs, DVD-RAMs, DVD-Rs, DVD+Rs, and DVDs.
  • optical discs such as RW, DVD+RW and Blu-ray (registered trademark)
  • magnetic media such as magnetic tapes, magneto-optical discs, semiconductor memories such as USB memories, and other arbitrary media that can store programs
  • the recording medium may include a device capable of recording a computer program (for example, a general-purpose device or a dedicated device in which a computer program is implemented in at least one form of software, firmware, etc., in an executable state).
  • each process and function included in the computer program may be realized by a logical processing block realized in the arithmetic device 31 by the arithmetic device 31 (that is, computer) executing the computer program, It may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) provided in the arithmetic unit 31, or a logical processing block and a part of hardware
  • FPGA Field Programmable Gate Array
  • ASIC Application Specific Integrated Circuit
  • FIG. 3 shows an example of logical functional blocks implemented within the arithmetic device 31 to execute robot control processing.
  • a shape data generator 311 a position calculator 312 , and a signal generator 313 are realized in the computing device 31 .
  • the processing performed by each of the shape data generation unit 311, the position calculation unit 312, and the signal generation unit 313 will be described later in detail with reference to FIG.
  • the storage device 32 can store desired data.
  • the storage device 32 may temporarily store computer programs executed by the arithmetic device 31 .
  • the storage device 32 may temporarily store data that is temporarily used by the arithmetic device 31 while the arithmetic device 31 is executing a computer program.
  • the storage device 32 may store data that the control device 3 saves for a long period of time.
  • the storage device 32 may include at least one of RAM (Random Access Memory), ROM (Read Only Memory), hard disk device, magneto-optical disk device, SSD (Solid State Drive), and disk array device. good. That is, the storage device 32 may include non-transitory recording media.
  • the communication device 33 can communicate with each of the robot 1 and the imaging unit 2 via a communication network (not shown). Alternatively, the communication device 33 can communicate with another device different from the robot 1 and the imaging unit 2 in addition to or instead of at least one of the robot 1 and the imaging unit 2 via a communication network (not shown). good too. In this embodiment, the communication device 33 may receive (that is, acquire) the workpiece image data IMG_2D and IMG_3D from the imaging unit 2 . Furthermore, the communication device 33 may transmit (that is, output) a robot control signal to the robot 1 .
  • the input device 34 is a device that receives input of information to the control device 3 from the outside of the control device 3 .
  • the input device 34 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by the user of the control device 3 .
  • the input device 34 may include a recording medium reading device capable of reading information recorded as data on a recording medium that can be externally attached to the control device 3 .
  • Information can be input to the control device 3 as data from a device external to the control device 3 via the communication device 33 .
  • the communication device 33 may function as an input device that receives input of information to the control device 3 from outside the control device 3 .
  • the output device 35 is a device that outputs information to the outside of the control device 3 .
  • the output device 35 may output information as an image.
  • the output device 35 may include a display device (so-called display) capable of displaying images.
  • the output device 35 may output information as audio.
  • the output device 35 may include an audio device capable of outputting audio (so-called speaker).
  • the output device 35 may output information on paper. That is, the output device 35 may include a printing device (so-called printer) capable of printing desired information on paper.
  • the output device 35 may output information as data to a recording medium that can be externally attached to the control device 3 .
  • Information can be output as data from the control device 3 to a device external to the control device 3 via the communication device 33 .
  • the communication device 33 may function as an output device that outputs information to the outside of the control device 3 .
  • FIG. 4 is a flowchart showing the flow of robot control processing.
  • the control device 3 uses the communication device 33 to acquire the workpiece image data IMG_2D from the imaging device 21 (step S1). Specifically, the imaging device 21 images the work W at a predetermined 2D imaging rate. For example, the imaging device 21 may image the work W at a 2D imaging rate that images the work W several tens to several hundred times (for example, 500 times) per second. As a result, the imaging device 21 generates work image data IMG_2D at a cycle corresponding to a predetermined 2D imaging rate. For example, the imaging device 21 may generate several tens to several hundreds (for example, 500) pieces of work image data IMG_2D per second.
  • the control device 3 acquires the work image data IMG_2D each time the imaging device 21 generates the work image data IMG_2D. That is, the control device 3 may acquire several tens to several hundred (for example, 500) workpiece image data IMG_2D per second.
  • the control device 3 uses the communication device 33 to acquire work image data IMG_3D from the imaging device 22 (step S1). Specifically, the imaging device 22 images the work W at a predetermined 3D imaging rate.
  • the 3D imaging rate is the same as the 2D imaging rate. However, the 3D imaging rate may differ from the 2D imaging rate.
  • the imaging device 22 may image the work W at a 3D imaging rate that images the work W several tens to several hundred times (for example, 500 times) per second. As a result, the imaging device 22 generates work image data IMG_3D at a cycle corresponding to a predetermined 3D imaging rate.
  • the imaging device 22 may generate several tens to several hundreds (for example, 500) pieces of work image data IMG_3D per second.
  • the control device 3 acquires the work image data IMG_3D each time the imaging device 22 generates the work image data IMG_3D. That is, the control device 3 may acquire several tens to several hundreds (for example, 500) pieces of workpiece image data IMG_3D per second.
  • the shape data generation unit 311 generates work shape data WSD representing the three-dimensional shape of the work W based on the acquired work image data IMG_3D (step S2).
  • the image represented by the workpiece image data IMG_3D includes the workpiece W on which the projection pattern is projected.
  • the three-dimensional shape of the work W on which the projection pattern is projected is reflected in the projection pattern reflected in the image indicated by the work image data IMG_3D.
  • the shape of the projection pattern reflected in the image represented by the work image data IMG_3D reflects the three-dimensional shape of the work W on which the projection pattern is projected.
  • the shape data generator 311 generates the work shape data WSD based on the projection pattern appearing in the image indicated by the work image data IMG_3D. For example, the shape data generation unit 311 calculates the parallax by associating each part (for example, each pixel) between the images indicated by the two image data included in the work image data IMG_3D. Specifically, in this association, the shape data generation unit 311 generates projection patterns reflected in the images represented by the two image data (that is, projection patterns reflected in the respective images). Each part) is associated to calculate the parallax.
  • the shape data generator 311 generates workpiece shape data WSD (that is, calculates the three-dimensional shape of the workpiece W) by a well-known method based on the principle of triangulation using the calculated parallax. It is better to associate each part of images in which projection patterns are reflected (in other words, each part of projected patterns in which they are reflected) than to associate each part of images in which projection patterns are not reflected. is calculated with higher accuracy. Therefore, the accuracy of the work shape data WSD to be generated (that is, the accuracy of calculation of the three-dimensional shape of the work W) becomes higher.
  • the work shape data WSD may be any data as long as it can indicate the three-dimensional shape of the work W.
  • An example of the workpiece shape data WSD is depth image data.
  • Depth image data is an image in which each pixel of the depth image indicated by the depth image data is associated with depth information in addition to luminance information.
  • Depth information is information that indicates the distance (that is, depth) between each part of an object reflected in each pixel and the imaging device 22 .
  • the depth image data may be an image in which the luminance information of each pixel indicates the depth of each part of the object (the distance between each part of the object and the imaging device 22).
  • the shape data generator 311 determines the distance between each part of the object appearing in the image indicated by the work image data IMG_3D and the imaging device 22 based on the projection pattern appearing in the image indicated by the work image data IMG_3D. and associate the calculated distance with each pixel of the work image data IMG_3D as depth information to generate a depth image.
  • Another example of the workpiece shape data WSD is point cloud data.
  • the point cloud data is data representing a set of points in a three-dimensional space corresponding to each part of an object appearing in the image represented by the workpiece image data IMG_3D.
  • the shape data generator 311 may generate point cloud data based on the depth image data and camera parameters of the imaging device 22 .
  • the position calculation unit 312 calculates the position of the work W based on the work image data IMG_2D acquired in step S1 and the work shape data WSD generated in step S2 (step S3).
  • the position calculator 312 may calculate the position of the feature point of the work W.
  • FIG. Examples of feature points of the work W include at least one of the center of the work W, the center of gravity of the work W, the vertex of the work W, the center of the surface of the work W, and the center of gravity of the surface of the work W.
  • the position calculator 312 calculates the position of the workpiece W in the global coordinate system.
  • the global coordinate system is a coordinate system that serves as a reference for the robot system SYS.
  • the global coordinate system is the coordinate system used to control the robot 1 .
  • the robot controller 14 may control the robot arm 12 so that the end effector 13 is positioned at a desired position within the global coordinate system.
  • a global coordinate system is a coordinate system defined by an X-axis (GL), a Y-axis (GL), and a Z-axis (GL) that are orthogonal to each other.
  • the X-axis (GL) may be an axis along the horizontal plane.
  • the Y-axis (GL) may be an axis along the horizontal plane.
  • the Z-axis (GL) may be an axis perpendicular to the horizontal plane.
  • the Z-axis (GL) may be an axis extending along the direction of gravity.
  • the X-axis, Y-axis and Z-axis shown in FIG. 2 may be the X-axis (GL), Y-axis (GL) and Z-axis (GL), respectively.
  • the origin of the global coordinate system does not have to be the origin of the X-axis (GL), Y-axis (GL) and Z-axis (GL) shown in FIG.
  • the origin of the global coordinate system may be an arbitrary position on the base 11 in FIG. (eg, the center of the contact surface or the center of gravity).
  • the position calculation unit 312 calculates the position Tx of the work W in the X-axis direction (GL) parallel to the X-axis (GL) and the Y-axis direction (GL ) and the position Tz of the work W in the Z-axis direction (GL) parallel to the Z-axis (GL).
  • the position calculation unit 312 calculates, in addition to or instead of at least one of the positions Tx, Ty, and Tz, the amount of rotation Rx of the work W around the X axis (GL), the Y axis ( GL)
  • At least one of the rotation amount Ry of the workpiece W around and the rotation amount Rz of the workpiece W around the Z-axis (GL) may be calculated.
  • the amount of rotation Rx of the work W around the X-axis (GL), the amount of rotation Ry of the work W around the Y-axis (GL), and the amount of rotation Rz of the work W around the Z-axis (GL) are respectively the X-axis (GL) It may be regarded as equivalent to the parameter representing the attitude of the surrounding work W, the parameter representing the attitude of the work W around the Y-axis (GL), and the parameter representing the attitude of the work W around the Z-axis (GL).
  • the position calculation unit 312 calculates the position of the work W as the posture of the work W around the X-axis (GL), the posture of the work W around the Y-axis (GL), and the posture of the work W around the Z-axis (GL). can be considered to be calculated.
  • the amount of rotation Rx of the work W around the X-axis (GL), the amount of rotation Ry of the work W around the Y-axis (GL), and the amount of rotation Rz of the work W around the Z-axis (GL) respectively correspond to the X-axis ( GL) may be regarded as indicating the position of the work W in the direction of rotation about the Y-axis (GL), the position of the work W in the direction of rotation about the Y-axis (GL), and the position of the work W in the direction of rotation about the Z-axis (GL).
  • the amount of rotation Rx of the work W around the X-axis (GL), the amount of rotation Ry of the work W around the Y-axis (GL), and the amount of rotation Rz of the work W around the Z-axis (GL) are all equal to the amount of rotation Rx of the work W can be regarded as a parameter representing the position of
  • the amount of rotation Rx of the work W around the X-axis (GL), the amount of rotation Ry of the work W around the Y-axis (GL), and the amount of rotation Ry of the work W around the Z-axis (GL) are respectively the position Rx of the workpiece W in the direction of rotation about the X-axis (GL), the position Ry of the workpiece W in the direction of rotation about the Y-axis (GL), and the direction of rotation about the Z-axis (GL)
  • the position Rz of the workpiece W is called.
  • the signal generator 313 generates a robot control signal based on the position of the workpiece W calculated in step S3 (step S4).
  • the signal generator 313 may generate a robot control signal so that the robot 1 performs a predetermined process on the workpiece W.
  • the signal generation unit 313 may generate the robot control signal so that the positional relationship between the robot 1 and the workpiece W becomes a desired positional relationship.
  • the signal generator 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the positional relationship between the robot 1 and the workpiece W becomes a desired positional relationship.
  • the signal generator 313 generates a robot control signal so that the end effector 13 performs a predetermined process on the work W when the positional relationship between the robot 1 and the work W reaches a desired positional relationship.
  • the signal generator 313 is a robot for controlling the operation of the end effector 13 so as to perform a predetermined process on the work W when the positional relationship between the robot 1 and the work W reaches a desired positional relationship.
  • a control signal may be generated.
  • the robot control signal for controlling the operation of the end effector 13 can also be called an end effector control signal.
  • each of FIGS. 5(a) to 5(d) is for grasping a work W placed on a placing device 4 such as an AGV (Automatic Guided Vehicle) moving on the support surface S.
  • a placing device 4 such as an AGV (Automatic Guided Vehicle) moving on the support surface S.
  • the signal generator 313 performs robot control for controlling the operation of the robot arm 12 so that the end effector 13 moves toward the space directly above the moving workpiece W.
  • a signal may be generated.
  • the signal generator 313 is configured so that the end effector 13 positioned directly above the moving work W can grip the work W while continuing to be positioned directly above the work W.
  • a robot control signal may be generated for controlling the operation of the robot arm 12 so as to approach the workpiece W until the workpiece W is reached.
  • the signal generator 313 controls the robot arm so that the end effector 13 positioned at a position where it can grip the moving workpiece W follows the moving workpiece W and grips the workpiece W.
  • Robotic control signals may be generated to control the operation of 12 and end effector 13 .
  • the signal generation unit 313 causes the robot arm 12 and the end effector 13 to operate so that the end effector 13 that grips the work W leaves the moving work W while gripping the work W.
  • a robot control signal may be generated for controlling.
  • the robot 1 may place the gripped work W on the placement device 4 .
  • the robot 1 may perform placement processing for placing the work W on the placement device 4 .
  • the control device 3 performs the same operation as when calculating the position of the work W to determine the position of the mounting device 4 (for example, the portion of the mounting device 4 where the work W is to be mounted). position), and the robot control signal may be generated based on the calculated position of the mounting device 4 .
  • the imaging device 21 may generate workpiece image data IMG_2D in which the mounting device 4 is captured by capturing an image of the mounting device 4 .
  • the work image data IMG_2D generated by the imaging device 21 is an image in which an object different from the work W is captured or in which the work W is not captured. may be image data indicating Further, the imaging device 22 may generate workpiece image data IMG_3D in which the mounting device 4 is captured by capturing an image of the mounting device 4 .
  • the work image data IMG_3D is image data representing an image in which an object different from the work W is captured or in which the work W is not captured.
  • the control device 3 calculates the position of the mounting device 4 based on workpiece image data IMG_2D and IMG_3D generated by imaging the mounting device 4, and controls the robot based on the calculated position of the mounting device 4. 1 may generate robot control signals to perform the placement process.
  • FIGS. 6(a) to 6(d) shows a certain period during the placement process for placing the work W on the placement device 4 that is moving on the support surface S.
  • FIG. 4 is a side view showing the positional relationship between the robot 1 and the workpiece W at the point in time; In this case, as shown in FIG.
  • the signal generator 313 determines that the end effector 13 gripping the workpiece W moves while gripping the workpiece W based on the calculated position of the placing device 4 .
  • a robot control signal may be generated for controlling the operation of the robot arm 12 so as to move to the space directly above the placement device 4 .
  • the signal generation unit 313 determines that the end effector 13 that is positioned directly above the moving mounting device 4 and grips the work W is located on the basis of the calculated position of the mounting device 4.
  • the robot arm 12 is operated to approach the mounting device 4 until the work W can be placed on the mounting device 4 while continuing to be positioned directly above the mounting device 4 and gripping the work W.
  • a robot control signal may be generated for controlling. As shown in FIG.
  • the signal generator 313 determines that the end effector 13 located at a position where the workpiece W can be placed on the mounting device 4 based on the calculated position of the mounting device 4 is A robot for controlling the operations of the robot arm 12 and the end effector 13 so as to place the work W on the mounting device 4 while following the moving mounting device 4 (that is, release the gripped work W).
  • a control signal may be generated.
  • the signal generation unit 313 determines that the end effector 13, which has finished arranging the workpiece W on the mounting device 4, from the mounting device 4 based on the calculated position of the mounting device 4.
  • a robotic control signal may be generated to control movement of the robotic arm 12 apart.
  • the target object whose position is calculated by the control device 3 is not limited to the work W, and any object different from the work W (
  • the examples shown in FIGS. 6(a) to 6(d) may include a mounting device 4).
  • the imaging device 21 may capture an image of the target object and generate work image data IMG_2D representing an image including the target object.
  • the work image data IMG_2D representing an image including the target object may be simply referred to as image data IMG_2D.
  • the imaging device 22 may capture an image of the target object and generate work image data IMG_3D representing an image including the target object.
  • the work image data IMG_3D representing an image including the target object may be simply referred to as image data IMG_3D.
  • the shape data generation unit 311 of the control device 3 may generate work shape data WSD representing the three-dimensional shape of the target object based on the work image data IMG_3D.
  • the workpiece shape data WSD representing the three-dimensional shape of the target object may be simply referred to as shape data WSD.
  • the position calculator 312 of the control device 3 may calculate the position of the target object based on the work image data IMG_2D and the work shape data WSD.
  • the signal generator 313 of the control device 3 may generate the robot control signal based on the position of the target object.
  • the signal generator 313 may generate the robot control signal so that the end effector 13 approaches the target object.
  • the signal generator 313 may generate the robot control signal so that the end effector 13 follows the moving target object.
  • the signal generator 313 may generate the robot control signal so that the end effector 13 moves away from the target object.
  • the target object whose position is calculated by the control device 3 can also be said to be the target object whose processing is performed by the end effector 13 .
  • the signal generation unit 313 uses the communication device 33 to output the robot control signal generated in step S4 to the robot 1 (in particular, the robot control device 14).
  • the robot controller 14 controls the motion of the robot 1 (for example, the motion of at least one of the robot arm 12 and the end effector 13) based on the robot control signal.
  • control device 3 repeats a series of processes from step S1 to step S4 until it is determined that the robot control process is finished (step S5).
  • the control device 3 continues to acquire workpiece image data IMG_2D and IMG_3D from the imaging devices 21 and 22, respectively, even while the operation of the robot 1 is being controlled based on the robot control signal.
  • each of the imaging devices 21 and 22 can detect the difference between the target object (for example, the workpiece W) and the imaging devices 21 and 22.
  • the target object for example, the work W
  • the target object for example, the work W
  • the target object may be imaged during the relative movement.
  • each of the imaging devices 21 and 22 captures an image of the target object (eg, work W) while the target object (eg, work W) is stationary while the imaging devices 21 and 22 are moving.
  • each of the imaging devices 21 and 22 captures an image of the target object (eg, work W) while the target object (eg, work W) is moving while the imaging devices 21 and 22 are stationary.
  • each of the imaging devices 21 and 22 may image the target object (eg, work W) while the target object (eg, work W) is moving and the imaging devices 21 and 22 are moving. .
  • the control device 3 controls the relative movement between the target object (eg, workpiece W) and the imaging devices 21 and 22 (that is, when at least one of the imaging devices 21 and 22 and the target object (eg, workpiece W) moves). 4, the robot control process shown in FIG. 4 may continue. As a result, even while the operation of the robot 1 is being controlled based on the robot control signal, the control device 3 can determine the position of the target object (for example, the work W) based on the newly acquired work image data IMG_2D and IMG_3D. can be newly calculated (that is, updated). However, each of the imaging devices 21 and 22 may image the target object (eg, work W) while the target object (eg, work W) is stationary and the imaging devices 21 and 22 are stationary. . The control device 3 may perform the robot control processing shown in FIG. 4 while the imaging devices 21 and 22 and the target object (for example, work W) are stationary.
  • the position calculator 312 does not have to calculate the position of the target object (for example, the work W) in the global coordinate system in step S3.
  • the position calculation unit 312 determines that the target object (for example, The position of the work W) may be calculated.
  • the signal generation unit 313 may generate a robot control signal in step S4 based on the position of the target object (for example, workpiece W) in a coordinate system different from the global coordinate system calculated in step S3. good.
  • step S3 of FIG. 4 Details of processing for calculating the position of the target object in step S3 of FIG. 4 will be described.
  • the position calculation unit 312 may calculate the position of any target object by performing the same operation as the processing for calculating the position of the workpiece W. That is, the following description of the processing for calculating the position of the work W can be used as an explanation of the processing for calculating the position of the target object by replacing the term "work W" with the term "target object". It is possible.
  • the position calculation unit 312 performs matching processing using the workpiece image data IMG_2D and the workpiece shape data WSD, and tracking processing using the workpiece image data IMG_2D and the workpiece shape data WSD. Calculate the position.
  • processing for calculating the position of the work W by performing matching processing and tracking processing will be described with reference to FIG.
  • FIG. 7 is a block diagram showing logical processing blocks for calculating the position of the workpiece W by performing matching processing and tracking processing.
  • the position calculation section 312 includes a 2D matching section 3121, a 3D matching section 3122, a 2D tracking section 3123, a 3D tracking section 3124, and a coordinate conversion section 3125.
  • the 2D matching unit 3121 performs 2D matching processing, which is an example of matching processing.
  • the 3D matching unit 3122 performs 3D matching processing, which is an example of matching processing.
  • the 2D tracking unit 3123 performs 2D tracking processing, which is an example of tracking processing.
  • the 3D tracking unit 3124 performs 3D tracking processing, which is an example of tracking processing.
  • the coordinate conversion unit 3125 performs position calculation processing for calculating the position of the workpiece W in the global coordinate system based on the results of the matching processing and tracking processing. Therefore, 2D matching processing, 3D matching processing, 2D tracking processing, 3D tracking processing, and position calculation processing will be described in order below.
  • the 2D matching unit 3121 performs matching processing using work image data IMG_2D as 2D matching processing. Matching processing using work image data IMG_2D is matching processing using work image data IMG_2D and reference image data IMG_2M.
  • the reference image data IMG_2M is image data representing a two-dimensional image of the work W that serves as a reference.
  • the reference image data IMG_2M is, for example, a plurality of images generated by virtually imaging a three-dimensional model of the workpiece W (for example, a CAD model created by CAD (Computer Aided Design)) from a plurality of different photographing directions. It may be image data representing a two-dimensional image.
  • the reference image data IMG_2M may be image data representing a two-dimensional image of the actual work W captured in advance.
  • the reference image data IMG_2M may be image data representing a plurality of two-dimensional images respectively generated by imaging the actual workpiece W from a plurality of different photographing directions.
  • the actual workpiece W that is imaged in advance may be a reference or a non-defective workpiece W.
  • the 2D matching unit 3121 may perform matching processing on the workpiece image data IMG_2D using the workpiece W appearing in the two-dimensional image indicated by the reference image data IMG_2M as a template.
  • the matching process itself may be the same as the existing matching process. For this reason, the detailed description of the matching process is omitted, but the outline thereof will be described below.
  • the 2D matching unit 3121 determines that a feature point (for example, at least one of a feature point and an edge) in the entire work W appearing in the two-dimensional image indicated by the reference image data IMG_2M corresponds to the image indicated by the work image data IMG_2D.
  • the work W reflected in the two-dimensional image indicated by the reference image data IMG_2M is translated, enlarged, reduced, and / Or it may be rotated.
  • the 2D matching unit 3121 matches the feature points of the entire work W appearing in the two-dimensional image indicated by the reference image data IMG_2M with the feature points of the entire work W appearing in the image indicated by the work image data IMG_2D.
  • the coordinate system of the reference image data IMG_2M for example, the coordinate system of the CAD model
  • the 2D imaging coordinate system based on the imaging device 21 that has imaged the work W. You may change the positional relationship.
  • the 2D matching unit 3121 can identify the positional relationship between the coordinate system of the reference image data IMG_2M and the 2D imaging coordinate system. After that, based on the positional relationship between the coordinate system of the reference image data IMG_2M and the 2D imaging coordinate system, the 2D matching unit 3121 shifts the position of the workpiece W in the coordinate system of the reference image data IMG_2M to the position of the workpiece W in the 2D imaging coordinate system. A position may be calculated. Note that the 2D matching unit 3121 determines that the feature part of the part of the work W reflected in the two-dimensional image indicated by the reference image data IMG_2M is the part of the work W reflected in the image indicated by the work image data IMG_2D.
  • the coordinate system of the reference image data IMG_2M (for example, the coordinate system of the CAD model) and the 2D image capturing coordinate system based on the image capturing device 21 that captured the image of the workpiece W so as to approach (typically match) the feature location. You may change the positional relationship with.
  • the matching process using the workpiece image data IMG_2D for example, at least one of SIFT (Scale-Invariant Feature Transform) and SURF (Speed-Upped Robust Feature) known methods may be used.
  • the method of calculating the position of the work W is not limited to the matching process using the work image data IMG_2D described above, and may be any other known method of calculating the position of the work W using the work image data IMG_2D. good.
  • the method of calculating the position of the work W may be a known method of calculating the position of the work W based on the work image data IMG_2D without using the reference image data IMG_2M. and the workpiece image data IMG_2D, a known method of calculating the position of the workpiece W may be used.
  • the method of calculating the position of the work W may be a method of calculating the position of the work W based on the work image data IMG_2D by machine learning or deep learning.
  • a prediction model is constructed that outputs the position of the work W when the work image data IMG_2D is input. can be calculated.
  • This prediction model may be stored in the 2D matching unit 3121 .
  • the 2D matching unit 3121 may read out this prediction model stored in the storage device 32 Coordinate system defined.
  • At least one of the X-axis (2D), Y-axis (2D), and Z-axis (2D) is aligned with the optical axis AX21 (in FIG. 2 see). It should be noted that the optical axis AX21 can also be said to be the optical axis of the imaging device 21 .
  • the 2D matching unit 3121 determines the position of the work W in the 2D imaging coordinate system by determining the position Tx (2D) of the work W in the X-axis direction (2D) parallel to the X-axis (2D), the Y At least one of the position Ty (2D) of the work W in the axial direction (2D) and the position Tz (2D) of the work W in the Z-axis direction (2D) parallel to the Z-axis (2D) may be calculated. .
  • the 2D matching unit 3121 in addition to or instead of at least one of positions Tx (2D), Ty (2D), and Tz (2D), as the position of the work W in the 2D imaging coordinate system, At least one of the amount of rotation Rx (2D) of the work W, the amount of rotation Ry (2D) of the work W around the Y axis (2D), and the amount of rotation Rz (2D) of the work W around the Z axis (2D) is can be calculated.
  • the 2D matching unit 3121 determines the position of the work W as the posture of the work W around the X-axis (2D), the posture of the work W around the Y-axis (2D), and the posture of the work W around the Z-axis (2D). can be considered to be calculated.
  • the amount of rotation Rx (2D) of the workpiece W around the X axis (2D) and the amount of rotation Rx (2D) of the work W around the X axis (2D), and The amount of rotation Ry (2D) of the workpiece W around and the amount of rotation Rz (2D) of the workpiece W around the Z axis (2D) are respectively calculated from the position Rx (2D) of the workpiece W in the direction of rotation around the X axis (2D).
  • the 2D matching unit 3121 calculates the position Tx (2D), the position Ty (2D), the position Tz (2D), the position Rx (2D), the position Ry (2D) and the position Rz (2D). At least one position having relatively high accuracy as compared with the position of the workpiece W to be measured may be calculated. On the other hand, the 2D matching unit 3121 selects the position Tx (2D), the position Ty (2D), the position Tz (2D), the position Rx (2D), the position Ry (2D), and the position Rz (2D). At least one other position with relatively low accuracy compared to the position of the workpiece W calculated by 3122 may not be calculated.
  • the 2D matching unit 3121 performs matching processing using work image data IMG_2D and reference image data IMG_2M that do not have position information in the Z-axis direction (2D).
  • the position Tx (2D), the position Ty (2D), and the position Rz (2D) can be calculated with relatively high accuracy without using positional information in the Z-axis direction (2D).
  • the accuracies of the position Tx (2D), the position Ty (2D), and the position Rz (2D) calculated by the 2D matching unit 3121 are respectively the positions Tx (3D), which are calculated by the 3D matching unit 3122, and the positions Ty (3D) and position Rz(3D) accuracy may be higher.
  • the 2D matching unit 3121 may calculate the position Tx (2D), the position Ty (2D) and the position Rz (2D).
  • the 2D matching unit 3121 may or may not calculate the position Tz (2D), the position Rx (2D), and the position Ry (2D). In the following description, an example will be described in which the 2D matching unit 3121 calculates at least the position Tx (2D), the position Ty (2D), and the position Rz (2D).
  • the control device 3 uses the position Tx (2D), the position Ty (2D), and the position Rz (2D) with relatively high accuracy calculated by the 2D matching unit 3121, in the global coordinate system Calculate the position of the workpiece W at .
  • the control device 3 can be calculated with relatively high accuracy.
  • the 2D matching unit 3121 outputs information on the calculated position of the work W to the coordinate conversion unit 3125 each time the position of the work W is calculated.
  • the 2D matching unit 3121 can match all 6 DOF (Degree Of Freedom) positions (that is, position Tx (2D), position Ty (2D), position Tz (2D), position Rx (2D), position Ry (2D ) and position Rz (2D)), and information on a part of the calculated 6DOF positions (for example, the above-described relatively high-precision position Tx (2D), position Ty (2D) and position Rz ( 2D)) may be output to the coordinate transformation unit 3125 .
  • 6 DOF Degree Of Freedom
  • the 2D matching unit 3121 may calculate a part of the 6DOF position and output information about the calculated part of the 6DOF position to the coordinate conversion unit 3125 .
  • the 2D matching unit 3121 calculates the position Tx (2D), the position Ty (2D), and the position Rz (2D) with relatively high accuracy described above, and calculates the position Tx (2D), the position Ty (2D), and the position Information about Rz(2D) may be output to the coordinate transformation unit 3125 .
  • the 2D matching unit 3121 may calculate all the 6DOF positions and output information about all the calculated 6DOF positions to the coordinate conversion unit 3125 .
  • the coordinate transformation unit 3125 may calculate the 6DOF position of the workpiece W using all the 6DOF positions output by the 2D matching unit 3121 .
  • the coordinate transformation unit 3125 converts some of the 6DOF positions output by the 2D matching unit 3121 (for example, the above-described relatively high-precision position Tx (2D), position Ty (2D), and position Rz (2D)) may be selected, and the 6DOF position of the workpiece W may be calculated using part of the selected 6DOF position.
  • the 2D matching section 3121 may calculate the position of the work W at a predetermined 2D matching cycle. In other words, the 2D matching section 3121 may calculate the position of the workpiece W each time the time corresponding to the 2D matching period elapses. Conversely, it may be considered that the time corresponding to the 2D matching period is required from the time when the 2D matching unit 3121 starts the 2D matching process to the time when the 2D matching process is completed. In this case, the 2D matching section 3121 may output information on the position of the work W to the coordinate transformation section 3125 at a predetermined 2D matching cycle.
  • the 2D matching cycle may be longer than the 2D imaging cycle corresponding to the 2D imaging rate at which the imaging device 21 images the workpiece W.
  • the 2D matching cycle may be a cycle in which the 2D matching unit 3121 can perform the 2D matching process ten-odd times to several tens times (for example, 15 times to 30 times) per second.
  • the imaging device 21 generates new workpiece image data IMG_2D from the time the 2D matching unit 3121 starts the 2D matching process to the time it completes.
  • the 2D matching unit 3121 keeps the work image newly generated by the imaging device 21 even when the imaging device 21 generates new work image data IMG_2D until the already started 2D matching process is completed.
  • a new 2D matching process using the data IMG_2D may not be started.
  • the 2D matching unit 3121 may newly start the 2D matching process using the latest work image data IMG_2D at that time.
  • the timing for generating the work image data IMG_2D shown in FIG. 9 may be the timing for the imaging device 21 to image the work W to generate the work image data IMG_2D.
  • the 2D matching period does not have to be longer than the 2D imaging period.
  • the 2D matching period may be shorter than the 2D imaging period.
  • the 2D matching period may be the same as the 2D imaging period.
  • the 3D matching section 3122 performs matching processing using the workpiece shape data WSD as 3D matching processing.
  • the matching process using the work shape data WSD is the matching process using the work shape data WSD and the reference shape data WMD.
  • the reference shape data WMD is data indicating the three-dimensional shape of the workpiece W that serves as a reference.
  • the reference shape data WMD may be, for example, data representing a three-dimensional model (for example, a CAD model) of the workpiece W.
  • the data representing the three-dimensional shape of the workpiece W may be depth image data, point cloud data, or other data representing the three-dimensional shape of the work W. good too.
  • the reference shape data WMD may be data representing the three-dimensional shape of the work W obtained by measuring the shape of the actual work W in advance.
  • the reference shape data WMD is generated in advance by the shape data generation unit 311 based on the work image data IMG_3D generated by imaging the work W on which the projection pattern from the projection device 23 is projected by the imaging device 22. It may be generated or generated in advance using a well-known shape measuring device different from the robot system SYS.
  • the actual work W that is imaged or measured in advance for generating the reference shape data WSD may be a reference or a non-defective work W.
  • the 3D matching unit 3122 may perform matching processing on the workpiece shape data WSD using the workpiece W indicated by the reference shape data WMD as a template.
  • the matching process itself may be the same as the existing matching process. For this reason, the detailed description of the matching process is omitted, but the outline thereof will be described below.
  • the 3D matching unit 3122 performs the The workpiece W indicated by the reference shape data WMD may be translated, enlarged, reduced and/or rotated. That is, the 3D matching unit 3122 performs The positional relationship between the coordinate system of the reference shape data WMD (for example, the coordinate system of the CAD model) and the 3D image capturing coordinate system based on the image capturing device 22 that captured the workpiece W may be changed.
  • the 3D matching unit 3122 can identify the positional relationship between the coordinate system of the reference shape data WMD and the 3D imaging coordinate system. After that, based on the positional relationship between the coordinate system of the reference shape data WMD and the 3D imaging coordinate system, the 3D matching unit 3122 shifts the position of the work W in the coordinate system of the reference shape data WMD from the position of the work W in the 3D imaging coordinate system. A position may be calculated. Note that the 3D matching unit 3122 is arranged so that the feature location in the part of the work W indicated by the work shape data WSD approaches (typically matches) the feature location in the part of the work W indicated by the reference shape data WMD.
  • the positional relationship between the coordinate system of the reference shape data WMD (for example, the coordinate system of the CAD model) and the 3D imaging coordinate system based on the imaging device 22 that has imaged the work W may be changed.
  • a matching process using the workpiece shape data WSD for example, at least one of RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), ICP (Iterative Closest Point), and DSO (Direct Sparse Odomtry) method may be used.
  • the method of calculating the position of the work W is not limited to the matching process using the work shape data WSD described above, and may be any other known method of calculating the position of the work W using the work shape data WSD. good.
  • the method of calculating the position of the work W may be a well-known method of calculating the position of the work W based on the work shape data WSD without using the reference shape data WMD. and the work shape data WSD may be used to calculate the position of the work W.
  • the method of calculating the position of the work W may be a method of calculating the position of the work W based on the work shape data WSD by machine learning or deep learning.
  • a prediction model that outputs the position of the work W when the work shape data WSD is input is constructed in advance. may be calculated.
  • This prediction model may be stored in the 3D matching section 3122 . .
  • the 3D matching unit 3122 may read this prediction model stored in the storage device 32.
  • the 3D matching unit 3122 performs 3D matching processing using the above-described point cloud data as work shape data WSD. As a result, the calculation accuracy of the position of the work W is higher than when the depth image data is used as the work shape data WSD.
  • the 3D matching unit 3122 uses point cloud data (for example, point cloud data representing a CAD model) representing a three-dimensional shape that serves as a reference for the workpiece W as the reference shape data WMD.
  • point cloud data for example, point cloud data representing a CAD model
  • the 3D matching unit 3122 may perform 3D matching processing using the depth image data described above as the workpiece shape data WSD.
  • the 3D matching unit 3122 may use depth image data (for example, depth image data representing a CAD model) representing the three-dimensional shape of the workpiece W as the reference shape data WMD.
  • the 3D imaging coordinate system is a coordinate system defined by mutually orthogonal X-axis (3D), Y-axis (3D) and Z-axis (3D). At least one of the X-axis (3D), Y-axis (3D), and Z-axis (3D) is the optical axis AX22 (particularly, the terminal optical element such as an objective lens) of the imaging device 22 (Fig. 2 see). It should be noted that the optical axis AX22 can also be said to be the optical axis of the imaging device 22 .
  • the optical axis AX22 is the optical axis of an optical system provided in one of the two monocular cameras. good too.
  • the optical axis AX22 may be the optical axis of either one of the two monocular cameras.
  • the Z-axis (3D) is an axis along the optical axis of the optical system provided in the imaging device 22
  • the 3D matching unit 3122 determines the position of the work W in the 3D imaging coordinate system by determining the position Tx (3D) of the work W in the X-axis direction (3D) parallel to the X-axis (3D), the Y At least one of the position Ty (3D) of the work W in the axial direction (3D) and the position Tz (3D) of the work W in the Z-axis direction (3D) parallel to the Z-axis (3D) may be calculated. .
  • the 3D matching unit 3122 in addition to or instead of at least one of the positions Tx (3D), Ty (3D) and Tz (3D), as the position of the workpiece W in the 3D imaging coordinate system, At least one of the amount of rotation Rx (3D) of the work W, the amount of rotation Ry (3D) of the work W around the Y-axis (3D), and the amount of rotation Rz (3D) of the work W around the Z-axis (3D) is can be calculated.
  • the amount of rotation Rx (3D) of the work W around the X-axis (3D), the amount of rotation Ry (3D) of the work W around the Y-axis (3D), and the amount of rotation Rz (3D) of the work W around the Z-axis (3D) are equivalent to a parameter representing the posture of the work W around the X-axis (3D), a parameter representing the posture of the work W around the Y-axis (3D), and a parameter representing the posture of the work W around the Z-axis (3D).
  • the 3D matching unit 3122 determines the position of the work W as the posture of the work W around the X-axis (3D), the posture of the work W around the Y-axis (3D), and the posture of the work W around the Z-axis (3D). can be considered to be calculated.
  • the amount of rotation Rx (3D) of the workpiece W around the X axis (3D) and the amount of rotation Rx (3D) of the workpiece W around the X axis (3D), and The amount of rotation Ry (3D) of the workpiece W around and the amount of rotation Rz (3D) of the workpiece W around the Z axis (3D) are respectively calculated from the position Rx (3D) of the workpiece W in the direction of rotation around the X axis (3D).
  • the 3D matching unit 3122 calculates the position Tx (3D), the position Ty (3D), the position Tz (3D), the position Rx (3D), the position Ry (3D) and the position Rz (3D). At least one position having relatively high accuracy as compared with the position of the workpiece W to be measured may be calculated. On the other hand, the 3D matching unit 3122 selects the position Tx (3D), the position Ty (3D), the position Tz (3D), the position Rx (3D), the position Ry (3D), and the position Rz (3D). At least one other position whose accuracy is relatively low compared to the position of the workpiece W calculated by 3121 may not be calculated.
  • the 3D matching unit 3122 performs a matching process using workpiece shape data WSD and reference shape data WMD having position information in the Z-axis direction (3D).
  • the position Tz (3D), the position Rx (3D), and the position Ry (3D) can be calculated with relatively high accuracy by using positional information in the Z-axis direction (3D).
  • the accuracies of the position Tz(3D), the position Rx(3D), and the position Ry(3D) calculated by the 3D matching unit 3122 are the positions Tz(2D), the position Rx(2D) calculated by the 2D matching unit 3121, respectively. ) and position Ry(2D).
  • the 3D matching unit 3122 may calculate the position Tz (3D), the position Rx (3D) and the position Ry (3D). On the other hand, the 3D matching unit 3122 may or may not calculate the position Tx (3D), the position Ty (3D), and the position Rz (3D). In the following description, an example will be described in which the 3D matching unit 3122 calculates at least the position Tz (3D), the position Rx (3D), and the position Ry (3D).
  • the control device 3 uses the position Tz (3D), the position Rx (3D), and the position Ry (3D) with relatively high accuracy calculated by the 3D matching unit 3122 in the global coordinate system.
  • the position of the work W at is calculated.
  • the control device 3 can be calculated with relatively high accuracy.
  • the 3D matching unit 3122 outputs information regarding the calculated position of the work W to the coordinate conversion unit 3125 each time the position of the work W is calculated.
  • the 3D matching unit 3121 detects all 6DOF positions (that is, position Tx (3D), position Ty (3D), position Tz (3D), position Rx (3D), position Ry (3D) and position Rz ( 3D)), and information on some of the calculated 6DOF positions (for example, the above-described relatively high-precision position Tz (3D), position Rx (3D), and position Ry (3D)), You may output to the coordinate transformation part 3125.
  • the 3D matching unit 3122 may calculate part of the 6DOF position and output information about the calculated part of the 6DOF position to the coordinate conversion unit 3125 .
  • the 3D matching unit 3122 calculates the position Tz (3D), the position Rx (3D) and the position Ry (3D) with relatively high accuracy described above, and calculates the position Tz (3D), the position Rx (3D) and the position Information about Ry (3D) may be output to the coordinate transformation unit 3125 .
  • the 3D matching unit 3122 may calculate all the 6DOF positions and output information about all the calculated 6DOF positions to the coordinate conversion unit 3125 .
  • the coordinate transformation unit 3125 may calculate the 6DOF position of the workpiece W using all the 6DOF positions output by the 3D matching unit 3122 .
  • the coordinate transformation unit 3125 selects a portion of the 6DOF positions output by the 3D matching unit 3122, and converts a portion of the selected 6DOF positions (for example, the above-described relatively accurate position Tz (3D),
  • the 6DOF position of the workpiece W may be calculated using the position Rx (3D) and the position Ry (3D)).
  • the 3D matching section 3122 may calculate the position of the work W at a predetermined 3D matching cycle. In other words, the 3D matching section 3122 may calculate the position of the workpiece W each time the time corresponding to the 3D matching period elapses. Conversely, it may be considered that the 3D matching unit 3122 requires a period of time corresponding to the 3D matching period from the start of the 3D tracking process to the completion of the 3D tracking process. In this case, the 3D matching section 3122 may output information about the position of the workpiece W to the coordinate transformation section 3125 at a predetermined 3D matching cycle.
  • the 3D matching cycle may be longer than the 3D imaging cycle corresponding to the 3D imaging rate at which the imaging device 22 images the work W.
  • the 3D matching cycle may be a cycle in which the 3D matching unit 3122 can perform the 3D matching process several times to several tens of times (for example, 1 time to 30 times) per second.
  • the imaging device 22 generates new workpiece image data IMG_3D from the time the 3D matching unit 3122 starts the 3D matching process to the time the 3D matching process is completed.
  • the 3D matching unit 3122 even if the imaging device 22 generates new work image data IMG_3D, until the already started 3D matching process is completed, the work image newly generated by the imaging device 22 A new 3D matching process using the data IMG_3D does not have to be started. After completing the started 3D matching process, the 3D matching unit 3122 may newly start the 3D matching process using the latest work image data IMG_3D at that time.
  • the timing of generating the workpiece image data IMG_3D shown in FIG. 11 can also be said to be the timing of imaging the workpiece W by the imaging device 22 in order to generate the workpiece image data IMG_3D.
  • the 3D matching period does not have to be longer than the 3D imaging period.
  • the 3D matching period may be shorter than the 3D imaging period.
  • the 3D matching period may be the same as the 3D imaging period.
  • the 3D matching period may be the same as the 2D matching period.
  • the length of the 3D matching period may be the same as the length of the 2D matching period. That is, the time from when the 3D matching unit 3122 starts to completes the 3D matching process may be the same as the time from when the 2D matching unit 3121 starts to completes the 2D matching process.
  • the 2D matching unit 3121 starts 2D matching processing after the imaging device 21 images the work W
  • the 3D matching unit 3122 starts 3D matching processing after the imaging device 22 images the work W.
  • the time from when the imaging device 21 images the workpiece W until the 2D matching unit 3121 completes the 2D matching process (that is, calculates the position of the workpiece W) is the 3D time from when the imaging device 22 images the workpiece W. It may be the same as the time until the matching unit 3122 completes the 3D matching process (that is, calculates the position of the workpiece W).
  • the 3D matching period does not have to be the same as the 2D matching period.
  • the length of the 3D matching period may be longer than the length of the 2D matching period.
  • the length of the 3D matching period may be shorter than the length of the 2D matching period.
  • the 3D matching period and the 2D matching period may temporally overlap as shown in the second and third timing charts from the top of FIG. . That is, the start time of the 3D matching cycle and the start time of the 2D matching cycle may be the same, and the end time of the 3D matching cycle and the end time of the 2D matching cycle may be the same.
  • the time when the 3D matching unit 3122 starts the 3D matching process and the time when the 2D matching unit 3121 starts the 2D matching process are the same, and the time when the 3D matching unit 3122 completes the 3D matching process.
  • the time at which the 2D matching unit 3121 completes the 2D matching process may be the same.
  • the start time of the 3D matching cycle and the start time of the 2D matching cycle may be shifted by a predetermined time.
  • the end time of the 3D matching cycle and the end time of the 2D matching cycle may be shifted by a predetermined time.
  • the time when the 3D matching section 3122 starts the 3D matching process and the time when the 2D matching section 3121 starts the 2D matching process may be shifted by a predetermined time.
  • the time at which the 3D matching unit 3122 completes the 3D matching process and the time at which the 2D matching unit 3121 completes the 2D matching process may be shifted by a predetermined time.
  • the data sizes of the work shape data WSD and the reference shape data WMD used for the 3D matching process are usually larger than the data sizes of the work image data IMG_2D and the reference image data IMG_2M used for the 2D matching process. Therefore, the 3D matching cycle, which is the time required for the 3D matching unit 3122 to complete the 3D matching process, is longer than the 2D matching cycle, which is the time required for the 2D matching unit 3121 to complete the 2D matching process. may become. Therefore, the 3D matching unit 3122 may perform processing for shortening the 3D matching period such that the 3D matching period is the same as the 2D matching period or the 3D matching period is shorter than the 2D matching period. .
  • the 3D matching unit 3122 uses the results of the 2D matching process (for example, the above-described position Tx (2D), position Ty (2D), position Tz (2D), position Rx (2D), position Ry (2D) and position (at least one of Rz(2D)), a part of the workpiece shape data WSD may be selected as 3D matching target data for 3D matching processing.
  • the positional relationship between the 2D imaging coordinate system and the 3D imaging coordinate system depends on the positional relationship between the imaging devices 21 and 22 . Therefore, the control device 3 stores coordinate system information indicating the positional relationship between the 2D imaging coordinate system and the 3D imaging coordinate system, which is generated based on the positional relationship between the imaging devices 21 and 22, in the storage device 32. You can store it.
  • the coordinate system information transforms a position within one of the 2D imaging coordinate system and the 3D imaging coordinate system into a position within the other coordinate system of the 2D imaging coordinate system and the 3D imaging coordinate system.
  • the control device 3 converts the position in either the 2D imaging coordinate system or the 3D imaging coordinate system to the 2D imaging coordinate system or the 3D imaging coordinate system based on the coordinate system information (particularly, the transformation matrix). Either of the coordinate systems can be transformed to a position in the other coordinate system.
  • the 3D matching unit 3122 converts the position Tx (2D), the position Ty (2D), the position Tz (2D), the position Rx (2D), the position Ry (2D) and the position Rz (2D) into the position Tx (3D) , position Ty (3D), position Tz (3D), position Rx (3D), position Ry (3D) and position Rz (3D). That is, as shown in FIG. 12, the 3D matching section 3122 may estimate the area where the work W is estimated to exist within the 3D imaging coordinate system based on the result of the 2D matching processing. After that, the 3D matching section 3122 may select, as 3D matching target data, a partial data portion corresponding to an area in which the work W is estimated to exist from the work shape data WSD. After that, the 3D matching section 3122 may perform 3D matching processing on the 3D matching target data which is a part of the workpiece shape data WSD.
  • the work shape data WSD may include not only a data portion indicating the shape of the work W, but also a data portion indicating the shape of an object different from the work W.
  • the imaging field of view of the imaging device 22 that generates the workpiece image data IMG_3D used to generate the workpiece shape data WSD may include not only the workpiece W but also an object different from the workpiece W.
  • the work shape data WSD is generated from the work image data IMG_3D generated by imaging the work W placed on the mounting device 4 by the imaging device 22.
  • a data portion indicating the shape of the mounting device 4 is included in addition to the data portion indicating the shape.
  • the 3D matching target data is typically data obtained by removing a data portion indicating the shape of an object different from the work W (that is, an object different from the target object) from the work shape data WSD.
  • the 3D matching target data may typically be regarded as equivalent to data obtained by selectively extracting data portions indicating the shape of the work W from the work shape data WSD.
  • the time required to complete the 3D matching process performed on the 3D matching target data is less than the time required to complete the This is because the time required to complete the 3D matching process depends on the size of the data on which the 3D matching process is performed.
  • the 3D matching cycle becomes shorter than when the 3D matching process is performed on the entire work shape data WSD.
  • the operation of selecting the 3D matching target data, which is a part of the work shape data WSD is equivalent to the operation of not selecting (or deleting) the data portion other than the 3D matching target data in the work shape data WSD. may be regarded as Therefore, based on the result of the 2D matching process, the 3D matching unit 3122 selects a part of the work shape data WSD corresponding to an area in which the work W is estimated not to exist (for example, as described above).
  • 3D matching target data, which is a part of the workpiece shape data WSD may be generated by deleting the data portion indicating the shape of an object different from the workpiece W).
  • the 3D matching unit 3122 may select 3D matching target data based on a user's instruction specifying a part of the work shape data WSD.
  • the control device 3 uses a GUI ( Graphical User Interface) may be displayed.
  • the control device 3 uses the output device 35 including a display device to display a plurality of points indicated by point cloud data, which is an example of the work shape data WSD, and point corresponding to the work W among the plurality of points.
  • a user-operable GUI may be displayed to specify the .
  • the control device 3 uses the output device 35 including a display device to display the depth image indicated by the depth image data, which is an example of the work shape data WSD, and designate the pixels corresponding to the work W in the depth image.
  • a GUI that can be operated by the user may be displayed.
  • the 3D matching unit 3122 may perform the 3D matching process on the 3D matching target data, which is a part of the work shape data WSD designated by the user. As a result, the 3D matching cycle becomes shorter than when the 3D matching process is performed on the entire work shape data WSD.
  • the 2D tracking unit 3123 performs 2D tracking processing by capturing two images of the workpiece W by the imaging device 21 at different times t1 and t2. Tracking processing is performed using the workpiece image data IMG_2D#t1 and IMG_2D#t2.
  • Tracking processing is performed using the workpiece image data IMG_2D#t1 and IMG_2D#t2.
  • the tracking process using the two workpiece image data IMG_2D#t1 and IMG_2D#t2 is, as shown in FIG. and at least one of edges) in the workpiece image data IMG_2D#t2. Note that the tracking process itself may be the same as the existing tracking process. Therefore, detailed description of the tracking process is omitted.
  • the 2D tracking unit 3123 may calculate the amount of change in the position of at least one feature location within the 2D imaging coordinate system between times t1 and t2 by performing tracking processing. After that, the 2D tracking unit 3123 determines the amount of change in the position of the workpiece W within the 2D imaging coordinate system between times t1 and t2 based on the amount of change in the position of at least one characteristic location within the 2D imaging coordinate system. may be calculated.
  • the method of calculating the amount of change in the position of the workpiece W is not limited to the tracking process using the two workpiece image data IMG_2D#t1 and IMG_2D#t2.
  • Other well-known methods of calculating the amount of change in the position of the work W using the method may be used.
  • the method of calculating the amount of change in the position of the work W is a method of calculating the amount of change in the position of the work W based on the two work image data IMG_2D#t1 and IMG_2D#t2 by machine learning or deep learning.
  • a prediction model is constructed that outputs the amount of change in the position of the work W when two work image data IMG_2D#t1 and IMG_2D#t2 are input.
  • the amount of change in the position of the workpiece W may be calculated by inputting the image data IMG_2D#t1 and IMG_2D#t2.
  • the 2D tracking unit 3123 may include this prediction model.
  • the 2D tracking unit 3123 calculates the change amount ⁇ Tx(2D) of the position Tx(2D) of the work W in the X-axis direction (2D) parallel to the X-axis (2D) as the change amount of the position of the work W in the 2D imaging coordinate system. , the amount of change ⁇ Ty (2D) of the position Ty (2D) of the workpiece W in the Y-axis direction (2D) parallel to the Y-axis (2D) and the position of the workpiece W in the Z-axis direction (2D) parallel to the Z-axis (2D) At least one of the amount of change ⁇ Tz(2D) of the position Tz(2D) may be calculated.
  • the 2D tracking unit 3123 in addition to or instead of at least one of the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), and the amount of change ⁇ Tz (2D) as the position of the workpiece W in the 2D imaging coordinate system, (2D) the amount of change ⁇ Rx (2D) in the position Rx (2D) of the work W in the direction of rotation around the Y axis (2D), the amount of change ⁇ Ry (2D) in the position Ry (2D) of the work W in the direction of rotation around the Y axis (2D), and At least one of the amount of change ⁇ Rz(2D) of the position Rz(2D) of the workpiece W in the direction of rotation about the Z-axis (2D) may be calculated.
  • the 2D tracking unit 3123 calculates the amount of change in the position of the work W as the amount of change in the posture of the work W around the X-axis (2D), the amount of change in the posture of the work W around the Y-axis (2D), and the amount of change in the posture of the work W around the Z-axis ( 2D)
  • the amount of change in the posture of the surrounding workpiece W may be calculated.
  • the 2D tracking unit 3123 calculates the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (2D), the amount of change ⁇ Rx (2D), the amount of change ⁇ Ry (2D), and the amount of change ⁇ Rz (2D). At least one amount of change may be calculated with relatively high accuracy compared to the amount of change in the position of the workpiece W calculated by the 3D tracking unit 3124 .
  • the 2D tracking unit 3123 calculates the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (2D), the amount of change ⁇ Rx (2D), the amount of change ⁇ Ry (2D), and the amount of change ⁇ Rz (2D).
  • at least one other amount of change whose accuracy is relatively low compared to the amount of change in the position of the workpiece W calculated by the 3D tracking unit 3124 may not be calculated.
  • the 2D tracking unit 3123 performs tracking processing using work image data IMG_2D that does not have position information in the Z-axis direction (2D).
  • the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), and the amount of change ⁇ Rz (2D) can be calculated with relatively high accuracy without using positional information in the Z-axis direction (2D).
  • the accuracies of the change amount ⁇ Tx(2D), the change amount ⁇ Ty(2D), and the change amount ⁇ Rz(2D) calculated by the 2D tracking unit 3123 are respectively the change amounts ⁇ Tx(3D ), the accuracy of the variation ⁇ Ty(3D) and the variation ⁇ Rz(3D).
  • the amount of change ⁇ Tz (2D), the amount of change ⁇ Rx (2D), and the amount of change ⁇ Ry (2D) with high accuracy without using positional information in the Z-axis direction (2D).
  • the accuracies of the change amount ⁇ Tz(2D), the change amount ⁇ Rx(2D), and the change amount ⁇ Ry(2D) calculated by the 2D tracking unit 3123 are respectively the change amounts ⁇ Tz(3D ), the accuracy of variation ⁇ Rx(3D) and variation ⁇ Ry(3D) may be higher.
  • the 2D tracking unit 3123 may calculate the amount of change ⁇ Tx(2D), the amount of change ⁇ Ty(2D), and the amount of change ⁇ Rz(2D). On the other hand, the 2D tracking unit 3123 may or may not calculate the amount of change ⁇ Tz(2D), the amount of change ⁇ Rx(2D), and the amount of change ⁇ Ry(2D). In the following description, an example will be described in which the 2D tracking unit 3123 calculates at least the amount of change ⁇ Tx(2D), the amount of change ⁇ Ty(2D), and the amount of change ⁇ Rz(2D).
  • the control device 3 uses the relatively high-precision change amount ⁇ Tx (2D), change amount ⁇ Ty (2D), and change amount ⁇ Rz (2D) calculated by the 2D tracking unit 3123 to perform global A position of the workpiece W within the coordinate system is calculated.
  • the control device 3 can can be calculated with relatively high accuracy.
  • the 2D tracking unit 3123 outputs information about the calculated amount of change in the position of the work W to the coordinate conversion unit 3125 each time the amount of change in the position of the work W is calculated.
  • the 2D tracking unit 3123 detects all the amount of change in the position of 6 DOF (that is, the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (2D), the amount of change ⁇ Rx (2D), the amount of change ⁇ Ry(2D) and the amount of change ⁇ Rz(2D)) may be calculated, and information about a part of the calculated amount of change in the position of 6 DOF may be output to the coordinate transformation section 3125 .
  • the 2D tracking unit 3123 may calculate part of the 6DOF position change amount and output information about the calculated part of the 6DOF position change amount to the coordinate conversion unit 3125 .
  • the 2D tracking unit 3123 calculates the amount of change ⁇ Tx(2D), the amount of change ⁇ Ty(2D), and the amount of change ⁇ Rz(2D) with relatively high accuracy, and calculates the amount of change ⁇ Tx(2D), the amount of change ⁇ Ty (2D) and information on the amount of change ⁇ Rz(2D) may be output to the coordinate transformation unit 3125 .
  • the 2D tracking unit 3123 may calculate all the amount of change in the 6 DOF position and output information about all the calculated amount of change in the 6 DOF position to the coordinate conversion unit 3125 .
  • the coordinate transformation unit 3125 may calculate the amount of change in the 6DOF position of the workpiece W using all the amount of change in the 6DOF position output by the 2D tracking unit 3123 .
  • the coordinate transformation unit 3125 selects a portion of the 6DOF position change amount output by the 2D tracking unit 3123, and uses the selected portion of the 6DOF position change amount to calculate the 6DOF position change amount of the workpiece W. can be calculated.
  • the 2D tracking unit 3123 may calculate the amount of change in the position of the work W at a predetermined 2D tracking cycle. In other words, the 2D tracking unit 3123 may calculate the amount of change in the position of the work W each time the time corresponding to the 2D tracking period has passed. Conversely, it may be considered that a time corresponding to the 2D tracking period is required from the time when the 2D tracking unit 3123 starts the 2D tracking process to the time when the 2D tracking process is completed. In this case, the 2D tracking section 3123 may output information about the amount of change in the position of the workpiece W to the coordinate transformation section 3125 at a predetermined 2D tracking cycle.
  • the 2D tracking cycle may be the same as the 2D imaging cycle corresponding to the 2D imaging rate at which the imaging device 21 images the workpiece W.
  • the 2D tracking cycle may be a cycle in which the 2D tracking unit 3123 can perform the 2D tracking process several tens to several hundred times (for example, 500 times) per second.
  • the 2D tracking unit 3123 detects the workpiece image data IMG_2D#t generated at time t and t+1 at the same time as or after the workpiece image data IMG_2D#t+1 is generated at time t+1. and 2D tracking processing using IMG_2D#t+1 may be performed.
  • the 2D tracking unit 3123 detects the workpiece image data IMG_2D#t+1 generated at times t+1 and t+2 at the same time as or after the workpiece image data IMG_2D#t+2 is generated at time t+2. and 2D tracking processing using IMG_2D#t+2 may be performed. As a result, the amount of change in the position of the workpiece W between times t+1 and t+2 is calculated.
  • the 2D tracking cycle does not have to be the same as the 2D imaging cycle.
  • the 2D tracking period may be shorter than the 2D imaging period.
  • the 2D tracking period may be longer than the 2D imaging period.
  • the 2D tracking unit 3123 does not need to calculate the amount of change in the position of the workpiece W within the 2D imaging coordinate system between times t1 and t2. For example, the 2D tracking unit 3123 calculates the positions of the workpiece W within the 2D imaging coordinate system at times t1 and t2 by performing the matching process described above on the two workpiece image data IMG_2D#t1 and IMG_2D#t2. You may Since the position of the workpiece W is calculated at different times (for example, times t1 and t2), the processing performed by the 2D tracking unit 3123 can be said to be tracking processing.
  • the 2D tracking unit 3123 performs the matching processing by the above-described 2D matching unit 3121, and uses fewer characteristic locations than the characteristic locations of the workpiece W appearing in the image indicated by the workpiece image data IMG_2D, to match the two workpieces. Matching processing may be performed for each of the image data IMG_2D#t1 and IMG_2D#t2. In this case, the 2D tracking unit 3123 can shorten the calculation time required for the matching process, so that the tracking process can be performed in a shorter period than the matching process by the 2D matching unit 3121 .
  • the 2D tracking period is shorter than the 2D matching period in which the 2D matching unit 3121 described above performs the 2D matching process.
  • the calculation device 31 may perform calculation focusing on the data portion related to the characteristic locations of the work W in the work image data IMG_2D and the work shape data WSD.
  • the arithmetic unit 31 needs to perform an arithmetic operation to search for the work W from the entire work image data IMG_2D and work shape data WSD.
  • the amount of calculation required for tracking processing is less than the amount of calculation required for matching processing. Therefore, the 2D tracking period is usually shorter than the 2D matching period.
  • the 2D tracking unit 3123 may perform the 2D tracking process multiple times from the start to the end of one 2D matching cycle. In other words, the 2D tracking unit 3123 may perform the 2D tracking process multiple times during the period from when the 2D matching unit 3121 starts the 2D matching process to when the 2D matching process is completed.
  • the 2D tracking period does not have to be shorter than the 2D matching period.
  • the 2D tracking period may be the same as the 2D matching period.
  • the 2D tracking period may be longer than the 2D matching period.
  • the timing of generating the workpiece image data IMG_2D shown in FIG. 14 can also be said to be the timing of imaging the workpiece W by the imaging device 21 in order to generate the workpiece image data IMG_2D.
  • the 3D tracking unit 3124 performs tracking processing using two workpiece shape data WSD#s1 and WSD#s2 as 3D tracking processing.
  • the two workpiece shape data WSD#s1 and WSD#s2 are respectively generated from the two workpiece image data IMG_3D#s1 and IMG_3D#s2 generated by imaging the workpiece W by the imaging device 22 at different times s1 and s2. be done.
  • Tracking processing using two workpiece shape data WSD#s1 and WSD#s2 is, as shown in FIG. (at least one) in the workpiece shape data WSD#s2. Note that the tracking process itself may be the same as the existing tracking process.
  • the 3D tracking unit 3124 may calculate the amount of change in the position of at least one feature location within the 3D imaging coordinate system between times s1 and s2 by performing tracking processing. After that, the 3D tracking unit 3124 determines the amount of change in the position of the workpiece W within the 3D imaging coordinate system between times s1 and s2 based on the amount of change in the position of at least one feature location within the 3D imaging coordinate system. may be calculated.
  • Tracking processing using the two workpiece shape data WSD#s1 and WSD#s2 includes, for example, RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), ICP (Iterative Closest Point), and DSO (Direct At least one known method of sparse odometry may be used.
  • RANSAC Random Sample Consensus
  • SIFT Scale-Invariant Feature Transform
  • ICP Iterative Closest Point
  • DSO Direct At least one known method of sparse odometry may be used.
  • the method of calculating the amount of change in the position of the workpiece W is not limited to the tracking process using the two workpiece shape data WSD#s1 and WSD#s2.
  • Other well-known methods of calculating the amount of change in the position of the work W using the position of the work W may be used.
  • the method of calculating the amount of change in the position of the work W is a method of calculating the amount of change in the position of the work W based on two pieces of work shape data WSD#s1 and WSD#s2 by machine learning or deep learning.
  • a prediction model is constructed in advance such that when two workpiece shape data WSD#s1 and WSD#s2 are input, the amount of change in the position of the workpiece W is output.
  • the amount of change in the position of the workpiece W may be calculated by inputting the workpiece shape data WSD#s1 and WSD#s2.
  • This prediction model may be stored in the 3D tracking unit 3124 . Note that the 3D tracking unit 3124 may read this prediction model stored in the storage device 32 .
  • the 3D tracking unit 3124 calculates the change amount ⁇ Tx(3D) of the position Tx(3D) of the work W in the X-axis direction (3D) parallel to the X-axis (3D) as the change amount of the position of the work W in the 3D imaging coordinate system. , the change amount ⁇ Ty (3D) of the position Ty (3D) of the workpiece W in the Y-axis direction (3D) parallel to the Y-axis (3D) and the position of the workpiece W in the Z-axis direction (3D) parallel to the Z-axis (3D) At least one of the amount of change ⁇ Tz(3D) of the position Tz(3D) may be calculated.
  • the 3D tracking unit 3124 uses the X-axis as the position of the work W in the 3D imaging coordinate system in addition to or instead of at least one of the amount of change ⁇ Tx (3D), the amount of change ⁇ Ty (3D), and the amount of change ⁇ Tz (3D).
  • the 3D tracking unit 3124 calculates the amount of change in the position of the work W as the amount of change in the posture of the work W around the X axis (3D), the amount of change in the posture of the work W around the Y axis (3D), and the amount of change in the posture of the work W around the Z axis ( 3D)
  • the amount of change in the posture of the surrounding workpiece W may be calculated.
  • the 3D tracking unit 3124 performs the following: At least one amount of change having relatively high accuracy as compared with the amount of change in the position of the workpiece W calculated by the 2D tracking unit 3123 may be calculated. On the other hand, the 3D tracking unit 3124 calculates the amount of change ⁇ Tx (3D), the amount of change ⁇ Ty (3D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and the amount of change ⁇ Rz (3D). Of these, at least one other amount of change whose accuracy is relatively low compared to the amount of change in the position of the work W calculated by the 2D tracking unit 3123 may not be calculated.
  • the 3D tracking unit 3124 performs tracking processing using workpiece shape data WSD having position information in the Z-axis direction (3D).
  • the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), and the amount of change ⁇ Ry (3D) can be calculated with relatively high accuracy by using positional information in the Z-axis direction (3D).
  • the accuracy of the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), and the amount of change ⁇ Ry (3D) calculated by the 3D tracking unit 3124 is respectively the amount of change ⁇ Tz (2D ), the accuracy of variation ⁇ Rx(2D) and variation ⁇ Ry(2D) may be higher.
  • the 3D tracking unit 3124 may calculate the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), and the amount of change ⁇ Ry (3D). On the other hand, the 3D tracking unit 3124 may or may not calculate the amount of change ⁇ Tx (3D), the amount of change ⁇ Ty (3D), and the amount of change ⁇ Rz (3D). In the following description, an example will be described in which the 3D tracking unit 3124 calculates at least the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), and the amount of change ⁇ Ry (3D).
  • the control device 3 uses the relatively high-precision change amount ⁇ Tz (3D), change amount ⁇ Rx (3D), and change amount ⁇ Ry (3D) calculated by the 3D tracking unit 3124 to perform global A position of the workpiece W within the coordinate system is calculated.
  • the control device 3 can operate within the global coordinate system can be calculated with relatively high accuracy.
  • the 3D tracking unit 3124 outputs information about the calculated amount of change in the position of the work W to the coordinate conversion unit 3125 each time the amount of change in the position of the work W is calculated.
  • the 3D tracking unit 3124 detects all of the 6DOF variations (that is, variation ⁇ Tx (3D), variation ⁇ Ty (3D), variation ⁇ Tz (3D), variation ⁇ Rx (3D), variation ⁇ Ry ( 3D) and the amount of change ⁇ Rz(3D)), and information on a part of the calculated amount of change of 6DOF may be output to the coordinate transformation section 3125 .
  • the 3D tracking unit 3124 may calculate part of the 6DOF change amount and output information about the calculated part of the 6DOF change amount to the coordinate conversion unit 3125 .
  • the 3D tracking unit 3124 calculates the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), and the amount of change ⁇ Ry (3D) with relatively high accuracy, and calculates the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D) and information on the amount of change ⁇ Ry (3D) may be output to the coordinate transformation unit 3125 .
  • the 3D tracking unit 3124 may calculate all the 6DOF change amounts and output information about all the calculated 6DOF change amounts to the coordinate transformation unit 3125 .
  • the coordinate transformation unit 3125 may calculate the amount of change in 6DOF of the work W using all the amounts of change in 6DOF output by the 3D tracking unit 3124 .
  • the coordinate conversion unit 3125 may select a portion of the 6DOF variation output from the 3D tracking unit 3124 and calculate the 6DOF variation of the workpiece W using the selected portion of the 6DOF variation. good.
  • the 3D tracking unit 3124 may calculate the amount of change in the position of the work W at a predetermined 3D tracking cycle. In other words, the 3D tracking unit 3124 may calculate the amount of change in the position of the work W each time the time corresponding to the 3D tracking period elapses. Conversely, it may be considered that the time corresponding to the 3D tracking period is required from the time when the 3D tracking unit 3124 starts the 3D tracking process to the time when the 3D tracking process is completed. In this case, the 3D tracking unit 3124 may output information regarding the amount of change in the position of the workpiece W to the coordinate transformation unit 3125 at a predetermined 3D tracking cycle.
  • the 3D tracking cycle may be the same as the 3D imaging cycle corresponding to the 3D imaging rate at which the imaging device 22 images the workpiece W.
  • the 3D tracking cycle may be a cycle in which the 3D tracking unit 3124 can perform the 3D tracking process several tens to several hundred times (eg, 500 times) per second.
  • the 3D tracking unit 3124 detects the work image data IMG_3D#s generated at times s and s+1 simultaneously with or after the work image data IMG_3D#s+1 is generated at time s+1. and IMG_3D#s+1, 3D tracking processing may be performed using workpiece shape data WSD#s and WSD#s+1 respectively generated.
  • the 3D tracking unit 3124 detects work image data IMG_3D#s+1 generated at times s+1 and s+2 at the same time or after work image data IMG_3D#s+2 is generated at time s+2. and IMG_3D#s+2, 3D tracking processing may be performed using workpiece shape data WSD#s+1 and WSD#s+2 respectively generated. As a result, the amount of change in the position of the workpiece W between times s+1 and s+2 is calculated.
  • the timing of generating the work image data IMG_3D shown in FIG. 16 can also be said to be the timing of the imaging device 22 imaging the work W in order to generate the work image data IMG_3D.
  • the 3D tracking cycle does not have to be the same as the 3D imaging cycle.
  • the 3D tracking period may be shorter than the 3D imaging period.
  • the 3D tracking period may be longer than the 3D imaging period.
  • the 3D tracking unit 3124 does not need to calculate the amount of change in the position of the workpiece W within the 3D imaging coordinate system between times s1 and s2. For example, the 3D tracking unit 3124 performs the above-described 3D matching processing on each of the two workpiece shape data WSD#s1 and WSD#s2, thereby determining the position of the workpiece W within the 3D imaging coordinate system at times s1 and s2. can be calculated. Since the position of the workpiece W is calculated at different times (for example, times s1 and s2), the processing performed by the 3D tracking unit 3124 can be called tracking processing.
  • the 3D tracking unit 3124 uses fewer feature points than the feature points of the work W indicated by the work shape data WSD, which are used in the matching process by the 3D matching unit 3122 described above, to obtain the two work shape data WSD#s1 and Three matching processes may be performed for each WSD#s2.
  • the 3D tracking unit 3124 can shorten the calculation time required for the matching process, so that the tracking process can be performed in a shorter period than the matching process by the 3D matching unit 3122 .
  • the 3D tracking period is shorter than the 3D matching period in which the 3D matching section 3122 described above performs the 3D matching process. This is because, as described above, the amount of calculation required for tracking processing is less than the amount of calculation required for matching processing. Therefore, the 3D tracking unit 3124 may perform the 3D tracking process multiple times from the start to the end of one 3D matching cycle. The 3D tracking unit 3124 may perform the 3D tracking process multiple times after the 3D matching unit 3122 starts the 3D matching process until the 3D matching process is completed.
  • the 3D tracking period does not have to be shorter than the 3D matching period.
  • the 3D tracking period may be the same as the 3D matching period.
  • the 3D tracking period may be longer than the 3D matching period.
  • the 3D tracking period may be the same as the 2D tracking period.
  • the length of the 3D tracking period may be the same as the length of the 2D tracking period. That is, the time from when the 3D tracking unit 3124 starts to completes the 3D tracking process may be the same as the time from when the 2D tracking unit 3123 starts to completes the 2D tracking process.
  • the 2D tracking unit 3123 starts 2D tracking processing after the imaging device 21 images the work W
  • the 3D tracking unit 3124 starts 3D tracking processing after the imaging device 22 images the work W.
  • the time from when the imaging device 21 images the workpiece W until the 2D tracking unit 3123 completes the 2D tracking process (that is, calculates the amount of change in the position of the workpiece W) is It may be the same as the time from imaging until the 3D tracking unit 3124 completes the 3D tracking process (that is, calculates the amount of change in the position of the work W).
  • the 3D tracking period may not be the same as the 2D tracking period.
  • the length of the 3D tracking period may be longer than the length of the 2D tracking period.
  • the length of the 3D tracking period may be shorter than the length of the 2D tracking period.
  • the 3D tracking cycle and the 2D tracking cycle may temporally overlap as shown in the second and fourth timing charts from the top of FIG. . That is, the start time of the 3D tracking cycle and the start time of the 2D tracking cycle may be the same, and the completion time of the 3D tracking cycle and the end time of the 2D tracking cycle may be the same.
  • the time when the 3D tracking unit 3124 starts the 3D tracking process and the time when the 2D tracking unit 3123 starts the 2D tracking process are the same, and the time when the 3D tracking unit 3124 completes the 3D tracking process.
  • the time at which the 2D tracking unit 3123 completes the 2D tracking process may be the same.
  • the times t1 and t2 at which the imaging device 21 images the workpiece W in order to generate the two workpiece image data IMG_2D#t1 and IMG_2D#t2 used for the 2D tracking process are respectively used for the 3D tracking process.
  • the 3D tracking period and the 2D tracking period do not have to overlap in terms of time.
  • the start time of the 3D tracking cycle and the start time of the 2D tracking cycle may be shifted by a predetermined time.
  • the end time of the 3D tracking cycle and the end time of the 2D tracking cycle may be shifted by a predetermined time.
  • the time when the 3D tracking unit 3124 starts the 3D tracking process and the time when the 2D tracking unit 3123 starts the 2D tracking process may be shifted by a predetermined time.
  • the time when the 3D tracking unit 3124 completes the 3D tracking process and the time when the 2D tracking unit 3123 completes the 2D tracking process may be shifted by a predetermined time.
  • the time t1 at which the imaging device 21 images the workpiece W to generate two pieces of workpiece image data IMG_2D#t1 used for 2D tracking processing is used to generate workpiece image data IMG_3D#s1 used for 3D tracking processing.
  • the time s1 at which the device 22 images the workpiece W may be different.
  • the time t2 at which the imaging device 21 images the work W to generate two pieces of work image data IMG_2D#t2 used for 2D tracking processing is used to generate work image data IMG_3D#s2 used for 3D tracking processing. It may be different from the time s2 at which the device 22 images the work W.
  • the 2D matching period may be the same as the 3D matching period or longer than the 2D tracking period.
  • the 3D matching period may be the same as the 2D matching period or longer than the 3D tracking period.
  • the 2D tracking period may be the same as the 3D tracking period or shorter than the 2D matching period.
  • the 3D tracking period may be the same as the 2D tracking period or shorter than the 3D matching period.
  • the shorter one of the 2D matching period and the 3D matching period is the longer one of the 2D tracking period and the 3D tracking period.
  • the relationship that the period is longer than the period of the other may be established. This is because, as described above, the amount of calculation required for tracking processing is less than the amount of calculation required for matching processing.
  • the data size of the workpiece shape data WSD used for 3D tracking processing is usually larger than the data size of workpiece image data IMG_2D used for 2D tracking processing.
  • the work shape data WSD is point cloud data
  • the data size of the work shape data WSD is likely to be larger than the data size of the work image data IMG_2D. Therefore, the 3D tracking cycle, which is the time required for the 3D tracking unit 3124 to complete the 3D tracking process, is longer than the 2D tracking cycle, which is the time required for the 2D tracking unit 3123 to complete the 2D tracking process. may become. Therefore, the 3D tracking unit 3124 may perform 3D tracking processing such that the 3D tracking cycle is the same as the 2D tracking cycle, or the 3D tracking cycle is shorter than the 2D tracking cycle.
  • the 3D tracking unit 3124 may perform 3D tracking processing using the depth image data described above as the workpiece shape data WSD.
  • the amount of calculation required for performing the tracking process is reduced.
  • the amount of computation required to perform the tracking process is reduced as the 3D tracking period is the same as the 2D tracking period. This is because the data size of depth image data in which depth information is associated with each pixel of a two-dimensional image does not differ greatly from the data size of work image data IMG_2D, which is a two-dimensional image.
  • the 3D tracking unit 3124 may perform 3D tracking processing using the point cloud data described above as the workpiece shape data WSD.
  • the coordinate transformation unit 3125 performs global A position of the workpiece W in the coordinate system is calculated. That is, the coordinate transformation unit 3125 calculates the position Tx, the position Ty, the position Tz, the position Rx, the position Ry, and the position Rz as the positions of the workpiece W in the global coordinate system.
  • the coordinate conversion unit 3125 corrects the result of the 2D matching process based on the result of the 2D tracking process, and corrects the result of the 3D matching process based on the result of the 3D tracking process. correct the result of Specifically, as shown in FIG. 7, the coordinate transformation unit 3125 acquires information about the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), and the amount of change ⁇ Rz (2D) as a result of the 2D tracking process. .
  • the coordinate transformation unit 3125 acquires information on the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), and the amount of change ⁇ Ry (3D) as a result of the 3D tracking process.
  • the coordinate transformation unit 3125 acquires information on the position Tx (2D), the position Ty (2D), and the position Rz (2D) as a result of the 2D matching process.
  • the coordinate transformation unit 3125 acquires information about the position Tz (3D), the position Rx (3D), and the position Ry (3D) as a result of the 3D matching process.
  • the coordinate conversion unit 3125 corrects the position Tx(2D) based on the amount of change ⁇ Tx(2D), thereby correcting the position Tx′(2D) of the work W in the X-axis direction (2D) of the 2D imaging coordinate system. may be calculated.
  • the coordinate conversion unit 3125 calculates the position Ty' (2D) of the work W in the Y-axis direction (2D) of the 2D imaging coordinate system by correcting the position Ty (2D) based on the amount of change ⁇ Ty (2D).
  • the coordinate conversion unit 3125 calculates the position Tz′ (3D) of the workpiece W in the Z-axis direction (3D) of the 3D imaging coordinate system by correcting the position Tz (3D) based on the amount of change ⁇ Tz (3D).
  • the coordinate transformation unit 3125 corrects the position Rx(3D) based on the amount of change ⁇ Rx(3D), so that the position Rx′(3D) of the workpiece W in the rotation direction around the X-axis (3D) of the 3D imaging coordinate system. may be calculated.
  • the coordinate conversion unit 3125 corrects the position Ry (3D) based on the amount of change ⁇ Ry (3D), thereby changing the position Ry′ (3D) of the workpiece W in the rotation direction around the Y-axis (3D) of the 3D imaging coordinate system. may be calculated.
  • the coordinate transformation unit 3125 corrects the position Rz(2D) based on the amount of change ⁇ Rz(2D), so that the position Rz′(2D) of the workpiece W in the rotation direction around the Z-axis (2D) of the 2D imaging coordinate system. may be calculated.
  • the process of correcting the result of the 2D matching process based on the result of the 2D tracking process may include the process of adding the result of the 2D tracking process to the result of the 2D matching process.
  • the process of correcting the result of the 3D matching process based on the result of the 3D tracking process may include the process of adding the result of the 3D tracking process to the result of the 3D matching process.
  • the coordinate transformation unit 3125 may calculate the position Tx'(2D) by adding the change amount ⁇ Tx(2D) to the position Tx(2D).
  • the coordinate transformation unit 3125 may calculate the position Ty'(2D) by adding the change amount ⁇ Ty(2D) to the position Ty(2D).
  • the coordinate transformation unit 3125 may calculate the position Tz'(3D) by adding the amount of change ⁇ Tz(3D) to the position Tz(3D).
  • the coordinate transformation unit 3125 may calculate the position Rx′(3D) by adding the change amount ⁇ Rx(3D) to the position Rx(3D).
  • the coordinate transformation unit 3125 may calculate the position Ry' (3D) by adding the change amount ⁇ Ry (3D) to the position Ry (3D).
  • the coordinate transformation unit 3125 may calculate the position Rz'(2D) by adding the amount of change ⁇ Rz(2D) to the position Rz(2D).
  • the 2D tracking period which is the period in which the variation ⁇ Tx(2D) is generated, is shorter than the 2D matching period, which is the period in which the position Tx(2D) is generated.
  • the 2D tracking unit 3123 calculates the amount of change ⁇ Tx(2D) a plurality of times after the new position Tx(2D) is generated and before the next position Tx(2D) is generated. In this case, each time the change amount ⁇ Tx(2D) is newly calculated, the coordinate conversion unit 3125 performs , the position Tx′ (2D) may be calculated.
  • the coordinate conversion unit 3125 converts the position Tx(2D) to the change amount calculated after the position Tx(2D) is calculated.
  • the position Tx'(2D) may be calculated by adding the sum of ⁇ Tx(2D).
  • the coordinate conversion unit 3125 converts the position Tx′(2D) at that time (that is, the position to which the calculated amount of change ⁇ Tx(2D) has already been added).
  • a new position Tx′(2D) may be calculated by adding the newly calculated amount of change ⁇ Tx(2D) to Tx(2D)). Therefore, each time the amount of change ⁇ Tx(2D) is calculated, the total sum of the amount of change ⁇ Tx(2D) added to the position Tx(2D) increases.
  • the position Tx'(2D) is updated more frequently than the comparative example in which the position Tx(2D) is used as it is as the position Tx'(2D).
  • the update frequency of the position Tx'(2D) depends on the 2D matching period, which is the period for calculating the position Tx(2D). do.
  • the position Tx'(2D) is calculated by correcting the position Tx(2D) based on the change amount ⁇ Tx(2D)
  • the update frequency of the position Tx'(2D) is the change amount ⁇ Tx(2D).
  • the position calculator 312 can calculate the position of the workpiece W within the global coordinate system in a shorter cycle. That is, the position calculator 312 can calculate the position of the workpiece W within the global coordinate system more frequently (in other words, at a higher speed).
  • the amount of change ⁇ Tx(2D) calculated by the 2D tracking process may contain an error corresponding to the amount of deviation from the actual position of the work W.
  • the error included in the sum total of the amount of change ⁇ Tx(2D) which continues to increase each time the amount of change ⁇ Tx(2D) is calculated, also increases.
  • the deviation between the position Tx'(2D) of the work W calculated by the control device 3 and the actual position of the work W increases each time the amount of change ⁇ Tx(2D) is added.
  • the accuracy of the position calculated by the matching process is higher than the accuracy of the position calculated by the tracking process.
  • the coordinate conversion unit 3125 calculates the amount of change ⁇ Tx(2D) added to the position Tx(2D) to calculate the position Tx′(2D). You may reset the summation. In this case, the coordinate conversion unit 3125 calculates the total sum of the amounts of change ⁇ Tx(2D) newly calculated after the position Tx(2D) is newly calculated for the newly calculated position Tx(2D). may be added to calculate the position Tx' (2D). As a result, the position calculator 312 can accurately calculate the position Tx'(2D).
  • the coordinate conversion unit 3125 calculates the position Tx(2D) to which the total sum of the amount of change Tx(2D) is added at the newly calculated position Tx(2D). ' (2D) is corrected.
  • FIG. 17 shows an example in which the position Tx(2D)#1 is newly calculated at time t10, and then the position Tx(2D)#2 is newly calculated at time t20 after the 2D matching period has elapsed from time t10. is shown.
  • the position Tx(2D)#1 has been newly calculated, so the amount of change ⁇ Tx(2D) to be added to the position Tx(2D)#1 has not yet been calculated. Therefore, from time t10 to time t11, position Tx(2D)#1 is used as position Tx'(2D).
  • the coordinate conversion unit 3125 adds the change amount ⁇ Tx(2D)#11 to the position Tx(2D)#1.
  • position Tx(2D)#1+change amount ⁇ Tx(2D)#11 is used as position Tx'(2D).
  • the coordinate transformation unit 3125 converts the amount of change ⁇ Tx(2D)#11 to ⁇ Tx(2D)# at the position Tx(2D)#1. Add 12.
  • the coordinate transformation unit 3125 adds the newly calculated ⁇ Tx(2D)#12 to the position Tx(2D)#1 to which the variation ⁇ Tx(2D)#11 has already been added.
  • position Tx(2D)#1+change amount .DELTA.Tx(2D)#11+change amount .DELTA.Tx(2D)#12 is used as position Tx'(2D).
  • the coordinate transformation unit 3125 converts the amount of change ⁇ Tx(2D)#11 to ⁇ Tx(2D)# at the position Tx(2D)#1. Add 13. Alternatively, the coordinate transformation unit 3125 adds the newly calculated ⁇ Tx(2D)#13 to the position Tx(2D)#1 to which the variation ⁇ Tx(2D)#11 to ⁇ Tx(2D)#12 have already been added. to add.
  • position Tx(2D)#1+change amount .DELTA.Tx(2D)#11+change amount .DELTA.Tx(2D)#12+change amount .DELTA.Tx(2D)#13 is equal to position Tx'(2D) used as
  • position Tx(2D)#2 is newly calculated, the amount of change ⁇ Tx(2D) to be added to the position Tx(2D) to calculate the position Tx′(2D) is The sum is reset to zero. Therefore, from time t20 to time t21, position Tx(2D)#2 is used as position Tx'(2D).
  • the coordinate conversion section 3125 adds the amount of change ⁇ Tx(2D)#21 to the position Tx(2D)#2.
  • position Tx(2D)#2+change amount ⁇ Tx(2D)#21 is used as position Tx'(2D).
  • the coordinate conversion unit 3125 converts the change amount ⁇ Tx(2D)#21 to ⁇ Tx(2D)# at the position Tx(2D)#2. Add 22.
  • the coordinate transformation unit 3125 adds the newly calculated ⁇ Tx(2D)#22 to the position Tx(2D)#2 to which the change amount ⁇ Tx(2D)#21 has already been added.
  • position Tx(2D)#2+change amount ⁇ Tx(2D)#21+change amount ⁇ Tx(2D)#22 is used as position Tx'(2D).
  • the coordinate conversion unit 3125 calculates the position Ty' (2D), the position Tz' (3D), the position Rx, as in the case of calculating the position Tx' (2D). ' (3D), position Ry' (3D) and position Rz' (2D) are calculated. That is, each time the change amount ⁇ Ty(2D) is newly calculated, the coordinate conversion unit 3125 converts the position Ty(2D) to the change amount ⁇ Ty(2D) calculated after the position Ty(2D) is calculated. ) may be added to calculate the position Ty′ (2D).
  • the coordinate conversion unit 3125 converts the position Ty′(2D) at that time (that is, the position to which the calculated amount of change ⁇ Ty(2D) has already been added).
  • the new position Ty'(2D) may be calculated by adding the newly calculated amount of change ⁇ Ty(2D) to Ty(2D)).
  • the coordinate conversion unit 3125 calculates the ⁇ change amount Ty(2D) to be added to the position Ty(2D) to calculate the position Ty'(2D).
  • a position Ty' (2D) may be calculated.
  • the coordinate conversion unit 3125 converts the amount of change ⁇ Tz(3D) calculated after the position Tz(3D) was calculated to the position Tz(3D).
  • the position Tz' (3D) may be calculated by adding the sums.
  • the coordinate conversion unit 3125 converts the position Tz′(3D) at that time (that is, the position to which the calculated change amount ⁇ Tz(3D) has already been added).
  • a new position Tz'(3D) may be calculated by adding the newly calculated amount of change ⁇ Tz(3D) to Tz(3D)).
  • the coordinate conversion unit 3125 calculates the change amount ⁇ Tz(3D) to be added to the position Tz(3D) to calculate the position Tz′(3D).
  • the position Tz'(3D) may be calculated.
  • the coordinate conversion unit 3125 converts the amount of change ⁇ Rx(3D) calculated after the position Rx(3D) is calculated to the position Rx(3D).
  • the position Rx' (3D) may be calculated by adding the sums.
  • the coordinate conversion unit 3125 converts the position Rx′(3D) at that time (that is, the position to which the calculated amount of change ⁇ Rx(3D) has already been added).
  • a new position Rx′(3D) may be calculated by adding the newly calculated amount of change ⁇ Rx(3D) to Rx(3D)).
  • the coordinate conversion unit 3125 calculates the amount of change ⁇ Rx(3D) to be added to the position Rx(3D) to calculate the position Rx′(3D).
  • the position Rx' (3D) may be calculated.
  • the coordinate conversion unit 3125 converts the change amount ⁇ Ry(3D) calculated after the position Ry(3D) is calculated to the position Ry(3D).
  • the position Ry' (3D) may be calculated by adding the sums.
  • the coordinate conversion unit 3125 converts the position Ry′(3D) at that time (that is, the position to which the calculated amount of change ⁇ Ry(3D) has already been added).
  • a new position Ry'(3D) may be calculated by adding the newly calculated amount of change ⁇ Ry(3D) to Ry(3D)).
  • the coordinate conversion unit 3125 calculates the amount of change ⁇ Ry(3D) to be added to the position Ry(3D) to calculate the position Ry′(3D).
  • the position Ry' (3D) may be calculated.
  • the coordinate conversion unit 3125 converts the change amount ⁇ Rz(2D) calculated after the position Rz(2D) is calculated to the position Rz(2D).
  • the position Rz' (2D) may be calculated by adding the sums.
  • the coordinate conversion unit 3125 converts the position Rz′(2D) at that time (that is, the position to which the calculated amount of change ⁇ Rz(2D) has already been added).
  • a new position Rz′(2D) may be calculated by adding the newly calculated amount of change ⁇ Rz(2D) to Rz(2D)).
  • the coordinate transformation unit 3125 calculates the amount of change ⁇ Rz(2D) to be added to the position Rz(2D) to calculate the position Rz′(2D). By resetting the total sum and successively adding the amount of change ⁇ Rz(2D) newly calculated after the position Rz(2D) was newly calculated to the newly calculated position Rz(2D), the position Rz'(2D) may be calculated.
  • the coordinate transformation unit 3125 transforms the position Tx' (2D), the position Ty' (2D), the position Tz' (3D), the position Rx' (3D), the position Ry' (3D), and the position Rz' (2D). Based on this, the position Tx, position Ty, position Tz, position Rx, position Ry, and position Rz of the workpiece W in the global coordinate system are calculated. Specifically, first, the coordinate transformation unit 3125 converts the position Tx' (2D), the position Ty' (2D), the position Tz' (3D), the position Rx' (3D), the position Ry' (3D) and the position Rz ' (2D) to a position in a common coordinate system, which is either a 2D imaging coordinate system or a 3D imaging coordinate system. However, any coordinate system different from the 2D imaging coordinate system and the 3D imaging coordinate system may be used as the common coordinate system.
  • the coordinate transformation unit 3125 does not have to transform the position Tx' (2D), the position Ty' (2D), and the position Rz' (2D).
  • the coordinate transformation unit 3125 transforms the position Tz' (3D), the position Rx' (3D), and the position Ry' (3D) into the position Tz' (2D) in the Z-axis direction (2D) of the 2D imaging coordinate system, Transform into a position Rx' (2D) in the direction of rotation about the X-axis (2D) of the 2D imaging coordinate system and a position Ry' (2D) in the direction of rotation about the Y-axis (2D) of the 2D imaging coordinate system.
  • the coordinate transformation unit 3125 uses the transformation matrix M32 for transforming the position in the 3D imaging coordinate system into the position in the 2D imaging coordinate system to convert the position Tz′ (3D), the position Rx′ (3D) and the position Ry' (3D) may be transformed into position Tz' (2D), position Rx' (2D) and position Ry' (2D).
  • the transformation matrix M32 can be calculated from the positional relationship between the imaging devices 21 and 22 as described above.
  • the accuracy of the position Tx' (2D), the position Ty' (2D), and the position Rz' (2D) is relatively high. Furthermore, as already mentioned above, the accuracy of the position Tz' (3D), the position Rx' (3D) and the position Ry' (3D) is relatively high. As a result, the accuracy of position Tz' (2D), position Rx' (2D) and position Ry' (2D) transformed from position Tz' (3D), position Rx' (3D) and position Ry' (3D) is also Also relatively high.
  • the accuracy of the 6DOF (Degree of Freedom) position of the workpiece W within the 2D imaging coordinate system (within the common coordinate system) used by the coordinate transformation unit 3125 to calculate the position of the workpiece W within the global coordinate system is , relatively high.
  • the control device 3 can calculate the position of the workpiece W within the global coordinate system with high accuracy using the 6DOF position of the workpiece W within the highly accurate 2D imaging coordinate system as described above. .
  • the control device 3 controls the position Tx (2D) and the amount of change ⁇ Tx (2D), the position Ty (2D) and the amount of change ⁇ Ty (2D), the position Tz (3D) and the amount of change ⁇ Tz (3D), the position Rx (3D) and amount of change ⁇ Rx (3D), position Ry (3D) and amount of change ⁇ Ry (3D), and position Rz (2D) and amount of change ⁇ Rz (2D) within the global coordinate system position Tx (2D) and amount of change ⁇ Tx (2D), position Ty (2D) and amount of change ⁇ Ty (2D), position Tz (3D) and amount of change ⁇ Tz (3D), position Rx ( 3D) and amount of change ⁇ Rx (3D), position Ry (3D) and amount of change ⁇ Ry (3D), and position R
  • position Tz' (3D), position Rx' (3D), and position Ry' (3D) are already positions in the 3D imaging coordinate system. is shown. Therefore, the coordinate transformation unit 3125 does not have to transform the position Tz' (3D), the position Rx' (3D) and the position Ry' (3D).
  • the coordinate transformation unit 3125 transforms the position Tx' (2D), the position Ty' (2D), and the position Rz' (2D) into the position Tx' ( 3D), the position Ty′ (3D) in the Y-axis direction (3D) of the 3D imaging coordinate system, and the position Rz′ (3D) in the rotation direction around the Z-axis (3D) of the 3D imaging coordinate system.
  • the coordinate transformation unit 3125 uses the transformation matrix M23 for transforming the position in the 2D imaging coordinate system into the position in the 3D imaging coordinate system to convert the position Tx′ (2D), the position Ty′ (2D) and the position Rz' (2D) may be transformed into position Tx' (3D), position Ty' (3D) and position Rz' (3D).
  • the transformation matrix M23 can be calculated from the positional relationship between the imaging devices 21 and 22 as described above.
  • the transformation matrix M23 may be an inverse matrix of the transformation matrix M32 described above.
  • the accuracy of the position Tz' (3D), the position Rx' (3D) and the position Ry' (3D) is relatively high. Furthermore, as already mentioned above, the position Tx' (2D), the position Ty' (2D) and the position Rz' (2D) are relatively accurate. As a result, the accuracy of the position Tx' (3D), the position Ty' (3D) and the position Rz' (3D) transformed from the position Tx' (2D), the position Ty' (2D) and the position Rz' (2D) is also Also relatively high. Therefore, the accuracy of the 6DOF position of the work W within the 3D imaging coordinate system used by the coordinate transformation unit 3125 to calculate the position of the work W within the global coordinate system is relatively high.
  • the control device 3 can highly accurately calculate the position of the workpiece W within the global coordinate system using the 6DOF position of the workpiece W within the highly accurate 3D imaging coordinate system as described above. .
  • the control device 3 controls the position Tx (2D) and the amount of change ⁇ Tx (2D), the position Ty (2D) and the amount of change ⁇ Ty (2D), the position Tz (3D) and the amount of change ⁇ Tz (3D), the position Rx (3D) and amount of change ⁇ Rx (3D), position Ry (3D) and amount of change ⁇ Ry (3D), and position Rz (2D) and amount of change ⁇ Rz (2D) within the global coordinate system position Tx (2D) and amount of change ⁇ Tx (2D), position Ty (2D) and amount of change ⁇ Ty (2D), position Tz (3D) and amount of change ⁇ Tz (3D), position Rx ( 3D) and amount of change ⁇ Rx (3D), position Ry (3D) and amount of change ⁇ Ry (3D), and global
  • the imaging devices 21 and 22 may be arranged so that the origin of the 2D imaging coordinate system is as close as possible to the origin of the 3D imaging coordinate system.
  • the imaging devices 21 and 22 may be arranged such that the X-axis (2D) of the 2D imaging coordinate system is as close as possible to the X-axis (3D) of the 3D imaging coordinate system.
  • the imaging devices 21 and 22 may be arranged such that the Y-axis (2D) of the 2D imaging coordinate system is as close as possible to the Y-axis (3D) of the 3D imaging coordinate system.
  • the imaging devices 21 and 22 may be arranged such that the Z axis (2D) of the 2D imaging coordinate system is as close as possible to the Z axis (3D) of the 3D imaging coordinate system.
  • the position Tz' (2D), the position Rx '(2D) and position Ry' (2D) are more likely to remain relatively accurate.
  • the position Tx' (3D), the position Ty '(3D) and position Rz'(3D) are more likely to remain relatively accurate.
  • the coordinate transformation unit 3125 performs position Tx' (2D), position Ty' (2D), position Tz' (3D), position Rx' (3D), position Ry' (3D) and position Rz After calculating '(2D), position Tx' (2D), position Ty' (2D), position Tz' (3D), position Rx' (3D), position Ry' (3D) and position Rz' (2D) to a position in the common coordinate system.
  • the coordinate transformation unit 3125 transforms the position Tx (2D), the position Ty (2D), the position Tz (3D), the position Rx (3D), the position Ry (3D), and the position Rz (2D) into You can convert to position.
  • the coordinate transformation unit 3125 converts the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and the amount of change ⁇ Rz (2D). may be transformed into a variation in the common coordinate system.
  • the coordinate transformation unit 3125 transforms the position Tx(2D), the position Ty(2D) and the position Rz(2D) into the change amount ⁇ Tx(2D), the change amount ⁇ Ty(2D) and the change amount ⁇ Rz(2D). You don't have to.
  • the coordinate transformation unit 3125 transforms the position Tz(3D), the position Rx(3D), and the position Ry(3D) into the position Tz(2D_conv), the position Rx(2D_conv ) and position Ry(2D_conv).
  • the coordinate conversion unit 3125 converts the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), and the amount of change ⁇ Ry (3D) into the amount of change ⁇ Tz (2D_conv) within the 2D imaging coordinate system, which is the common coordinate system, It may be converted into the amount of change ⁇ Rx(2D_conv) and the amount of change ⁇ Ry(2D_conv).
  • the coordinate transformation unit 3125 may perform the transformation described above using a transformation matrix M32 for transforming a position within the 3D imaging coordinate system into a position within the 2D imaging coordinate system. After that, the coordinate transformation unit 3125 converts the position Tx (2D), the position Ty (2D), the position Tz (2D_conv), the position Rx (2D_conv), the position Ry (2D_conv), the position Rz (2D), the change amount ⁇ Tx (2D ), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (2D_conv), the amount of change ⁇ Rx (2D_conv), the amount of change ⁇ Ry (2D_conv), and the amount of change ⁇ Rz (2D), the position Tx′ (2D), the position Ty '(2D), position Tz' (2D), position Rx' (2D), position Ry' (2D) and position Rz' (2D) may be calculated.
  • a transformation matrix M32 for transforming a position within the 3D imaging coordinate system into a position
  • the coordinate transformation unit 3125 may calculate the position Tx'(2D) by correcting the position Tx(2D) based on the amount of change ⁇ Tx(2D).
  • the coordinate transformation unit 3125 may calculate the position Ty'(2D) by correcting the position Ty(2D) based on the amount of change ⁇ Ty(2D).
  • the coordinate conversion unit 3125 may calculate the position Tz'(2D) by correcting the position Tz(2D_conv) based on the amount of change ⁇ Tz(2D_conv).
  • the coordinate conversion unit 3125 may calculate the position Rx'(2D) by correcting the position Rx(2D_conv) based on the amount of change ⁇ Rx(2D_conv).
  • the coordinate conversion unit 3125 may calculate the position Ry'(2D) by correcting the position Ry(2D_conv) based on the amount of change ⁇ Ry(2D_conv).
  • the coordinate transformation unit 3125 may calculate the position Rz'(2D) by correcting the position Rz(2D) based on the amount of change ⁇ Rz(2D).
  • position Tz (3D), position Rx (3D), and position Ry (3D) already indicate positions within the 3D imaging coordinate system.
  • the amount of change ⁇ Tz(3D), the amount of change ⁇ Rx(3D), and the amount of change ⁇ Ry(3D) already indicate the amount of change within the 3D imaging coordinate system. Therefore, the coordinate transformation unit 3125 transforms the position Tz (3D), the position Rx (3D), and the position Ry (3D) into the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), and the amount of change ⁇ Ry (3D). You don't have to.
  • the coordinate conversion unit 3125 converts the position Tx (2D), the position Ty (2D), and the position Rz (2D) into the position Tx (3D_conv), the position Ty (3D_conv) within the 3D imaging coordinate system, which is the common coordinate system. ) and position Rz(3D_conv).
  • the coordinate conversion unit 3125 converts the amount of change ⁇ Tx(2D), the amount of change ⁇ Ty(2D), and the amount of change ⁇ Rz(2D) into the amount of change ⁇ Tx(3D_conv) within the 3D imaging coordinate system, which is the common coordinate system, It may be converted into the amount of change ⁇ Ty(3D_conv) and the amount of change ⁇ Rz(3D_conv).
  • the coordinate transformation unit 3125 may perform the transformation described above using a transformation matrix M32 for transforming a position within the 2D imaging coordinate system into a position within the 3D imaging coordinate system.
  • the coordinate transformation unit 3125 converts the position Tx (3D_conv), the position Ty (3D_conv), the position Tz (3D), the position Rx (3D), the position Ry (3D) and the position Rz (3D_conv) into the change amount ⁇ Tx (3D_conv ), the amount of change ⁇ Ty (3D_conv), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and the amount of change ⁇ Rz (3D_conv), the position Tx′ (3D), the position Ty '(3D), position Tz' (3D), position Rx' (3D), position Ry' (3D) and position Rz' (3D) may be calculated.
  • the coordinate conversion unit 3125 may calculate the position Tx'(3D) by correcting the position Tx(3D_conv) based on the amount of change ⁇ Tx(3D_conv).
  • the coordinate conversion unit 3125 may calculate the position Ty'(3D) by correcting the position Ty(3D_conv) based on the amount of change ⁇ Ty(3D_conv).
  • the coordinate transformation unit 3125 may calculate the position Tz'(3D) by correcting the position Tz(3D) based on the amount of change ⁇ Tz(3D).
  • the coordinate transformation unit 3125 may calculate the position Rx′(3D) by correcting the position Rx(3D) based on the change amount ⁇ Rx(3D).
  • the coordinate transformation unit 3125 may calculate the position Ry'(3D) by correcting the position Ry(3D) based on the amount of change ⁇ Ry(3D).
  • the coordinate conversion unit 3125 may calculate the position Rz'(3D) by correcting the position Rz(3D_conv) based on the amount of change ⁇ Rz(3D_conv).
  • the coordinate conversion unit 3125 converts the 6DOF of the work W within the global coordinate system based on the 6DOF position of the work W within the 2D imaging coordinate system or the 6DOF position of the work W within the 3D imaging coordinate system. Calculate the position. That is, the coordinate conversion unit 3125 converts the position Tx, position Ty, position Tz, position Rx, position Ry and position Rz are calculated.
  • the coordinate transformation unit 3125 uses a transformation matrix M for transforming a position within the 2D imaging coordinate system or the 3D imaging coordinate system to a position within the global coordinate system to convert the 6DOF of the work W within the 2D imaging coordinate system. or the 6DOF position of the work W within the 3D imaging coordinate system may be transformed into the 6DOF position of the work W within the global coordinate system.
  • Transformation matrix M may include, for example, a product of transformation matrices that reflect changes in the position coordinates of imaging device 21 or 22 due to rotation of link 121 about the axis defined by each joint 122 of robot arm 12. .
  • This transformation matrix may be a so-called rotation matrix, a rotation matrix containing a translation component, or a matrix based on Euler angles.
  • an existing transformation method may be used, so a detailed description thereof will be omitted.
  • control device 3 controls the position Tx (2D), the position Ty (2D ) and position Rz (2D), and relatively high-precision change amount ⁇ Tx (2D), change amount ⁇ Ty (2D), and change amount ⁇ Rz (2D) among the change amounts of 6DOF calculated by the 2D tracking unit 3123.
  • the position of the workpiece W in the global coordinate system is calculated using the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), and the amount of change ⁇ Ry (3D), which have relatively high accuracy. For this reason, the difference in the accuracy of the 6DOF positions calculated by the 2D matching unit 3121 and the 3D matching unit 3122 is not considered, and the accuracy of the 6DOF change amount calculated by the 2D tracking unit 3123 and the 3D tracking unit 3124 is not considered.
  • the control device 3 can calculate the position of the workpiece W in the global coordinate system with high accuracy as compared with the case where the position of the workpiece W in the global coordinate system is calculated without considering the difference.
  • control device 3 corrects the result of the 2D matching process based on the result of the 2D tracking process, and corrects the result of the 3D matching process based on the result of the 3D tracking process, so that the workpiece in the global coordinate system
  • the position of W is calculated. Therefore, as described above, compared to the case where the position of the workpiece W in the global coordinate system is calculated based on the results of the 2D matching processing and the 3D matching processing without using the results of the 2D tracking processing and the 3D tracking processing.
  • the control device 3 can calculate the position of the workpiece W in the global coordinate system at a shorter period (that is, at a higher frequency). That is, the position calculator 312 can calculate the position of the workpiece W in the global coordinate system at a higher speed.
  • control device 3 can generate (that is, update) control signals and output them to the robot 1 at a higher frequency. It can also be said that the control device 3 can generate (update) the control signal at a higher speed and output it to the robot 1 .
  • control device 3 may generate a control signal each time the position of the workpiece W is calculated.
  • the control device 3 can generate the control signal in the same cycle as the cycle in which the position of the workpiece W is calculated (for example, the 2D tracking cycle or the 3D tracking cycle described above).
  • the control device 3 can output the control signal to the robot 1 in the same period as the period in which the position of the workpiece W is calculated (for example, the 2D tracking period or the 3D tracking period described above).
  • the control device 3 may calculate the workpiece W
  • the control signal is generated and output in the same period as the 2D matching period or 3D matching period (typically, a period shorter than the 2D tracking period or 3D tracking period), which is the period in which the position of is calculated. . Therefore, in this embodiment, the control device 3 can generate (that is, update) the control signal more frequently (faster) and output it to the robot 1 as compared with this case. Note that the control device 3 may generate and output the control signal in a period different from the period in which the position of the workpiece W is calculated (for example, in a period different from the above-described 2D tracking period or 3D tracking period). .
  • Such an effect is especially beneficial when at least one of the robot 1 and the work W is moving. This is because the lower the frequency of calculation of the position of the work W under the condition that at least one of the robot 1 and the work W is moving, the more the position of the work W calculated by the control device 3 and the actual position of the work W will be different. This is because the period during which the degree of divergence is large becomes longer.
  • the control device 3 calculates the position of the workpiece W in the global coordinate system more frequently (faster). and the degree of divergence becomes relatively small. Therefore, the control device 3 can calculate the position of the work W with high accuracy by calculating the position of the work W at a high frequency.
  • control device 3 appropriately generates (that is, updates) a control signal for controlling the robot 1 so as to perform desired processing on the workpiece W whose position is calculated with high accuracy and at high frequency. and output to the robot 1.
  • the possibility that the motion of the robot 1 will be different from the intended motion is reduced.
  • the robot 1 grabs a moving workpiece W as shown in FIG. (ie, generate and output robot control signals). Therefore, even if the movement of the robot 1 deviates from the desired movement for following the moving workpiece, the movement of the robot 1 is corrected at high frequency (high speed) in accordance with the calculation cycle of the position of the workpiece W. be done. As an example, even if the movement of the moving work W is irregular, the movement of the robot 1 is corrected at high frequency (high speed) according to the irregular movement of the work W. As another example, even if the work W moves at high speed, the movement of the robot 1 is corrected at high frequency (high speed) in accordance with the high-speed movement of the work.
  • the robot 1 can follow the moving workpiece W while reducing unnecessary movements.
  • the robot 1 can grip the moving workpiece W while reducing unnecessary movements.
  • the robot 1 can move with respect to a target object such as a work W while reducing wasteful movements.
  • the robot 1 since the position of the target object such as the work W is calculated with high frequency, the possibility of the target object such as the work W falling out of the imaging field of the imaging unit 2 is reduced. Therefore, the robot 1 can move accurately and at high speed with respect to the target object such as the work W (for example, accurately and at high speed follow the target object such as the work W) without losing sight of the target object such as the work W.
  • the end effector 13 can perform desired processing on the target object such as the work W.
  • the robot 1 approaches the target object such as the work W accurately and at high speed, and the end effector 13 performs desired processing on the target object such as the work W. It can be performed.
  • the control device 3 uses the workpiece intermediately calculated in the process of calculating the position of the workpiece W within the global coordinate system.
  • the first method considers the difference in the accuracy of the 6DOF position and variation of W, and the method corrects the results of the 2D matching process and the 3D matching process based on the results of the 2D tracking process and the 3D tracking process, respectively.
  • the second method is adopted. Therefore, the control device 3 can calculate the position of the workpiece W with much higher accuracy than when only one of the first and second methods is employed. However, the control device 3 may employ only one of the first and second methods. Even in this case, the control device 3 can calculate the position of the workpiece W with higher accuracy than when neither of the first and second methods are employed.
  • the control device 3 considers that the amount of change ⁇ Tx(2D) calculated by the 2D tracking process includes an error, and when the position Tx(2D) is newly calculated, may reset the sum of the amount of change Tx(2D) that is added to the position Tx(2D) to calculate the position Tx'(2D). As a result, the control device 3 can calculate the position Tx′(2D) with higher accuracy than when the sum of the amount of change Tx(2D) added to the position Tx(2D) is not reset. .
  • control device 3 calculates the position Ty' (2D), the position Tz' (3D), the position Rx' (3D), the position Ry' (3D) and the position Rz' (2D) with higher accuracy. can do. As a result, the control device 3 can calculate the position of the work W with much higher accuracy.
  • the 2D matching period may temporally overlap the 3D matching period, or the 2D tracking period may temporally overlap the 3D tracking period. good too.
  • the time when the imaging device 21 images the work W for the 2D matching process may be the same as the time when the imaging device 22 images the work W for the 3D matching process.
  • the times t1 and t2 at which the imaging device 21 images the workpiece W for the 3D tracking process may be the same as the times s1 and s2 at which the imaging device 22 images the workpiece W for the 3D tracking process.
  • position Tx' (2D), position Ty' (2D) and position Rz' (2D) calculated from the results of 2D matching processing and 2D tracking processing, and calculated from the results of 3D matching processing and 3D tracking processing
  • the control device 3 determines the position Tx' (2D), the position Ty' (2D), and the position Rz' (2D) of the work W at one time, and the position Tz' (3D) of the work W at one time, Based on the position Rx' (3D) and the position Ry' (3D), the position of the work W at one time in the global coordinate system can be calculated with much higher accuracy.
  • the controller 3 may determine the workpiece at one time.
  • Position Tx' (2D), position Ty' (2D) and position Rz' (2D) of W, and position Tz' (3D) and position Rx' (3D) of workpiece W at another time different from one time and the position Ry' (3D) there is the possibility to calculate the position of the workpiece W at one time or the other.
  • the position of the work W at one time may differ from the position of the work W at another time.
  • the control device 3 receives the work image data from the imaging device 21.
  • Work image data IMG_3D can be obtained from the imaging device 22 at the same time that IMG_2D is obtained. Therefore, the imaging device 21 for generating the work image data IMG_2D and the imaging device 22 for generating the work image data IMG_3D are not prepared separately (for example, a single unit for generating the work image data IMG_2D and IMG_3D).
  • the acquisition frequency of workpiece image data IMG_2D and IMG_3D is higher than in the case where one imaging device) is prepared.
  • the control device 3 appropriately generates (that is, updates) the control signal for controlling the robot 1 so as to perform desired processing on the workpiece W whose position is frequently calculated. and output to the robot 1.
  • FIG. 18 is a block diagram showing the configuration of the control device 3a in the first modified example.
  • the control device 3a in the first modified example differs from the control device 3 described above in that the arithmetic device 31 includes a preprocessing unit 314a as a logical processing block.
  • the preprocessing unit 314a performs preprocessing before matching processing and tracking processing are performed.
  • Other features of the control device 3 a may be identical to other features of the control device 3 .
  • the pre-processing unit 314a may perform pre-processing on the workpiece shape data WSD generated by the shape data generation unit 311.
  • the 3D matching unit 3122 may perform the 3D matching process using the preprocessed work shape data WSD.
  • the 3D tracking unit 3124 may perform 3D tracking processing using two pieces of work shape data WSD that have been preprocessed respectively.
  • the pre-processing performed on the work shape data WSD may include data deletion processing for deleting a part of the data portion of the work shape data WSD.
  • the work shape data WSD may include not only a data portion indicating the shape of the work W, but also a data portion indicating the shape of an object different from the work W (that is, an object different from the target object). have a nature.
  • the work shape data WSD is generated from the work image data IMG_3D generated by the imaging device 22 imaging the work W placed on the mounting device 4
  • the work shape data WSD is generated by the mounting device It may contain a data portion that indicates the shape of 4. Therefore, as shown in FIG. 19, the preprocessing unit 314a may delete the data portion indicating the shape of the mounting device 4 from the workpiece shape data WSD.
  • the mounting surface of the mounting device 4 on which the workpiece W is mounted may be flat.
  • the preprocessing unit 314a may perform plane removal processing, which is an example of data deletion processing, on the workpiece shape data WSD.
  • plane removal process include at least one of plane removal process using RANSAC (Random Sample Consensus) and plane removal process using the least squares method.
  • Such data deletion processing reduces the data size of the workpiece shape data WSD.
  • the amount of calculation required for the 3D matching process using the workpiece shape data WSD and the amount of calculation required for the 3D tracking process using the workpiece shape data WSD are reduced. Therefore, the 3D matching cycle, which is the time required for the 3D matching unit 3122 to complete the 3D matching process, and the 3D tracking cycle, which is the time required for the 3D tracking unit 3124 to complete the 3D tracking process, can be shortened.
  • the position calculation unit 312 mistakenly regards the data portion indicating the shape of an object different from the work W (that is, an object different from the target object) as the data portion indicating the shape of the work W. less likely to be recognized. This is because the data portion indicating the shape of the object different from the workpiece W, which may become noise in the tracking processing and matching processing described above, is deleted. Therefore, the position calculator 312 can appropriately recognize the data portion indicating the shape of the workpiece W.
  • FIG. For example, when the workpiece shape data WSD is point cloud data, the position calculator 312 can recognize the point cloud regarding the workpiece W appropriately.
  • the position calculation unit 312 can recognize depth information about the work W appropriately. As a result, the position detection unit 312 can calculate the position of the workpiece W with higher accuracy than when the data deletion process is not performed.
  • the pre-processing unit 314a extracts an object different from the work W from the work shape data WSD based on a user's instruction to designate a data portion indicating a shape of an object different from the work W in the work shape data WSD. may be deleted.
  • the control device 3 uses an output device 35 including a display device to designate a part of the work shape data WSD as a data portion indicating the shape of an object different from the work W.
  • a user-operable GUI may be displayed for this purpose.
  • the control device 3 uses the output device 35 including a display device to display a plurality of points indicated by the point cloud data, which is an example of the work shape data WSD, and select an object different from the work W among the plurality of points.
  • a user-operable GUI may be displayed to designate a point indicating the shape of the .
  • the control device 3 uses an output device 35 including a display device to display a depth image indicated by depth image data, which is an example of the work shape data WSD, and to display the shape of an object different from the work W in the depth image.
  • a user-operable GUI may be displayed to specify the pixel indicating the .
  • the preprocessing unit 314a may perform preprocessing on at least one of the workpiece image data IMG_2D and IMG_3D.
  • the 2D matching unit 3121 may perform 2D matching processing using preprocessed workpiece image data IMG_2D.
  • the 2D tracking unit 3123 may perform 2D tracking processing using two preprocessed workpiece image data IMG_2D.
  • the shape data generation unit 311 may generate the work shape data WSD using the preprocessed work image data IMG_3D.
  • the 3D matching unit 3122 may perform 3D matching processing using work shape data WSD generated from preprocessed work image data IMG_3D.
  • the 3D tracking unit 3124 may perform 3D tracking processing using two pieces of work shape data WSD generated from two pieces of work image data IMG_3D that have been preprocessed.
  • the pre-processing performed on at least one of the workpiece image data IMG_2D and IMG_3D may include gamma correction processing.
  • the gamma correction process adjusts the contrast of the work image data IMG_2D (or work image data IMG_3D) to emphasize the edge of the work W appearing in the work image data IMG_2D (or work image data IMG_3D). may include processing.
  • At least one of the imaging devices 21 and 22 may be capable of automatically adjusting exposure.
  • the imaging unit 2 may be provided with a photometric device 24a capable of measuring the brightness (that is, luminance) of the workpiece W. .
  • At least one of the imaging devices 21 and 22 may automatically adjust the exposure based on the measurement results of the photometric device 24a.
  • at least one of the imaging devices 21 and 22, in addition to or in place of the measurement result of the photometric device 24a obtains the luminance values (for example, multiple The exposure may be automatically adjusted based on the average luminance value of the pixels).
  • At least one of the imaging devices 21 and 22 may automatically adjust the exposure so that the luminance value of the work image data IMG_2D (or work image data IMG_3D) falls within a desired range.
  • the contrast of the work image data IMG_2D (or the work image data IMG_3D) is increased by the gamma correction process so as to emphasize the edge of the work W reflected in the work image data IMG_2D (or the work image data IMG_3D). can be properly adjusted.
  • the pre-processing performed on at least one of the work image data IMG_2D and IMG_3D may include high dynamic range synthesis processing.
  • a plurality of work image data IMG_2D (or a plurality of work image data IMG_3D) generated by imaging the work W under a plurality of imaging environments with different exposures are synthesized.
  • processing for generating work image data IMG_2D (or work image data IMG_3D) having a wide dynamic range with little blown-out highlights or blocked-up shadows may be included.
  • the pre-processing performed on at least one of the work image data IMG_2D and IMG_3D may include noise removal processing. As shown in FIG. 22, the noise removal process removes noise contained in the work image data IMG_2D (or work image data IMG_3D) to obtain work image data IMG_2D (or work image data IMG_3D) with less noise. ) may be generated.
  • the pre-processing unit 314a may perform known filter processing on at least one of the work image data IMG_2D and IMG_3D as the noise removal processing. For example, the preprocessing unit 314a may perform at least one filtering process of an averaging filter, a median filter, and an expansion filter on at least one of the work image data IMG_2D and IMG_3D.
  • the preprocessing unit 314a may perform known filter processing on at least one of the work image data IMG_2D and IMG_3D in addition to the noise removal processing.
  • the preprocessing unit 314a may perform at least one of a smoothing filter and an edge enhancement filter on at least one of the work image data IMG_2D and IMG_3D.
  • FIG. 23 is a block diagram showing the configuration of the robot system SYS in the second modified example.
  • robot system SYSb the second modified example of robot system SYS is called "robot system SYSb" to distinguish it from robot system SYS described above.
  • the robot system SYSb in the second modified example may differ from the robot system SYS described above in that it includes an imaging unit 2b instead of the imaging unit 2.
  • Other features of the robot system SYSb may be the same as other features of the robot system SYS.
  • the imaging unit 2b in the second modified example differs from the imaging unit 2 described above in that it includes an illumination device 24b, a filter 25b, and a filter 26b. Other features of the imaging unit 2b may be the same as other features of the imaging unit 2b.
  • the illumination device 24b is a device capable of irradiating the workpiece W with illumination light.
  • the illumination device 24b is a device capable of illuminating the work W with the illumination light by irradiating the work W with the illumination light.
  • the illumination device 24b may illuminate the work W with illumination light containing light components in the second wavelength band (for example, the wavelength band of red light).
  • the projection light with which the projection device 23 irradiates the workpiece W includes a light component in a first wavelength band (for example, a wavelength band of blue light) different from the second wavelength band. You may
  • the projection device 23 is a device that projects a desired projection pattern onto the work W by irradiating the work W with projection light.
  • the projection device 23 may be regarded as illuminating the workpiece W with projection light. That is, the projection device 23 may be regarded as an illumination device that illuminates the workpiece W with projection light. As described above, when the projection device 23 is regarded as an illumination device, the projection light may not be light capable of projecting a desired projection pattern onto the target object.
  • the filter 25b can attenuate the light component in the first wavelength band.
  • the image capturing device 21 captures an image of the work W by receiving the light from the work W with the image sensor through the filter 25b.
  • the filter 25b attenuates the projection light. Therefore, return light (for example, at least one of reflected light and scattered light of the projection light) from the workpiece W irradiated with the projection light (that is, on which the projection pattern is projected) is attenuated by the filter 25b.
  • the image pickup device 21 is affected by the projection light emitted from the projection device 23.
  • the work W illuminated by the illumination light emitted from the illumination device 24b can be properly imaged without receiving the light.
  • the filter 26b can attenuate the optical components in the second wavelength band.
  • the image capturing device 22 captures an image of the work W (in other words, a projection pattern projected onto the work W) by receiving light from the work W with the image sensor through the filter 26b.
  • the filter 26b attenuates the illumination light because the illumination light contains light components in the second wavelength band. Therefore, return light (for example, at least one of reflected light and scattered light of the illumination light) from the workpiece W irradiated with the illumination light from the illumination device 24b is attenuated by the filter 26b.
  • the imaging device 22 can detect the illumination light emitted from the illumination device 24b.
  • the workpiece W illuminated by the projection light emitted from the projection device 23 (in other words, the projection pattern projected onto the workpiece W) can be properly imaged without being affected.
  • the imaging device 21 can image the work W without being affected by the imaging of the work W (projection pattern projected onto the work W) by the imaging device 22 .
  • the imaging device 22 can image the work W (the projection pattern projected onto the work W) without being affected by the imaging of the work W by the imaging device 21 . Therefore, the imaging devices 21 and 22 can image the work W at the same time. That is, even if the time when the imaging device 21 images the work W and the time when the imaging device 22 images the work W (the projection pattern projected onto the work W) are the same, each of the imaging devices 21 and 22 , the workpiece W can be imaged properly.
  • FIG. 24 is a side view showing the appearance of the robot system SYS in the third modified example.
  • robot system SYSc the third modification of the robot system SYS is referred to as "robot system SYSc" to distinguish it from the robot system SYS described above.
  • the robot system SYSc in the third modification differs from the robot system SYS described above in that the positional relationships between the robot arm 12 and the imaging devices 21 and 22 are changed.
  • Other features of the robot system SYSc may be the same as other features of the robot system SYS.
  • the imaging device 22 may be attached at a position farther from the robot arm 12 than the position at which the imaging device 21 is attached.
  • the imaging devices 21 and 22 may be attached to the robot arm 12 such that the distance from the imaging device 22 to the robot arm 12 is longer than the distance from the imaging device 21 to the robot arm 12 .
  • the imaging device 22 may be attached at a position farther from the end effector 13 than the position at which the imaging device 21 is attached.
  • the imaging devices 21 and 22 may be attached to the robot arm 12 such that the distance from the imaging device 22 to the end effector 13 is longer than the distance from the imaging device 21 to the end effector 13 .
  • the imaging devices 21 and 22 may be attached to the robot arm 12 via the support member 15c.
  • the imaging device 21 may be attached to the first portion of the support member 15c.
  • the imaging device 22 may be attached to a second portion of the support member 15c that is farther from the robot arm 12 than the first portion.
  • the imaging device 22 may be attached to a second portion of the support member 15c that is farther from the end effector 13 than the first portion.
  • the imaging device 21 is more likely to be able to capture an image of the work W from the space directly above the work W or the vicinity of the space, compared to the imaging device 22 . This is because when the end effector 13 performs a predetermined process on the work W, there is a high possibility that the end effector 13 will be positioned right above the work W. This is because there is a high possibility that the image capturing device 21 that is used can also capture an image of the work W from the space directly above the work W or the vicinity of the space.
  • the imaging device 21 When the imaging device 21 images the work W from the space directly above the work W or the vicinity of the space, the imaging device 21 detects that the Z axis (2D) of the 2D imaging coordinate system intersects the upper surface of the work W at an angle of There is a high possibility that the workpiece W can be imaged in a state close to 90 degrees. As a result, the control device 3 extracts the position Tx(2D), the position Ty(2D), and the position Rz(2D) from the workpiece image data IMG_2D, and the amount of change ⁇ Tx(2D), the amount of change ⁇ Ty(2D), and The amount of change ⁇ Rz(2D) can be calculated with higher accuracy.
  • the imaging device 21 can generate work image data IMG_2D representing an image in which the work W with little upper surface distortion is reflected. For this reason, compared to the case of using the workpiece image data IMG_2D representing an image in which the workpiece W whose upper surface is highly distorted is reflected, the control device 3 controls the workpiece W under the condition in which the mark is formed on the upper surface of the workpiece W.
  • the marks formed on the upper surface of the workpiece W can be appropriately recognized from the image data IMG_2D.
  • the control device 3 can accurately perform the process using the mark. For example, when the control device 3 performs a process of calculating the position of the work W based on the position of the mark as the process using the mark, the control device 3 can calculate the position of the work W more accurately. can be done.
  • the imaging device 22 is more likely to be able to capture an image of the work W from a space obliquely above the work W. As a result, the imaging device 22 is more likely to be able to image not only the top surface of the work W but also the side surface of the work W (or any surface other than the top surface).
  • the workpiece shape data WSD generated from the workpiece image data IMG_3D indicates not only the three-dimensional shape of the top surface of the workpiece W, but also the three-dimensional shape of the side surface (or any surface other than the top surface) of the workpiece W. more likely. Therefore, it is possible to generate work shape data WSD in which the three-dimensional shape of the work W is more appropriately reflected.
  • the 3D matching unit 3122 and The 3D tracking unit 3124 can easily identify the sides and vertices of the work W as the feature points of the work W (for example, at least one of feature points and edges). As a result, the control device 3 can perform each of the 3D matching process and the 3D tracking process more appropriately.
  • the projection device 23 described above may be provided on the support member 15 c or the robot arm 12 . Also, the projection device 23 described above may be provided at a location different from the support member 15 c or the robot arm 12 .
  • the lighting device 24b described above may be provided on the support member 15c or the robot arm 12.
  • FIG. The illumination device 24b described above may be provided at a location different from the support member 15c or the robot arm 12.
  • the robot arm 12 is attached with an end effector 13 (for example, a hand gripper) that performs at least one of the gripping process, the placement process, and the fitting process.
  • an end effector 13 for example, a hand gripper
  • the end effector 13 is not limited to a device that performs at least one of the gripping process, the placement process, and the fitting process, and may be a device that performs other processes on the target object such as the workpiece W.
  • the robot arm 12 may be attached with a processing device for processing the workpiece W, which is an example of the end effector 13 .
  • the processing device performs at least one of additional processing of adding a new shaped object to the work W, removal processing of removing a part of the work W, welding processing of joining two works W, and cutting processing of cutting the work W. You may do one.
  • the processing device may process the workpiece W using a tool. In this case, a processing device including tools may be attached to the robot arm 12 .
  • the processing apparatus may process the work W by irradiating the work W with an energy beam (for example, light, electromagnetic wave, and charged particle beam).
  • an energy beam for example, light, electromagnetic wave, and charged particle beam
  • a processing device including an irradiation device that irradiates the workpiece W with an energy beam may be attached to the robot arm 12 .
  • a processing device as an example of the end effector 13 may perform a soldering process for soldering components to the workpiece W.
  • the processing device may solder the component to the work W using a soldering iron.
  • a processing device including a soldering iron may be attached to the robot arm 12 .
  • the processing apparatus may solder the component to the work W by irradiating the solder with an energy beam (eg, light, electromagnetic wave, and charged particle beam).
  • an energy beam eg, light, electromagnetic wave, and charged particle beam
  • a processing device including an irradiation device that irradiates the workpiece W with an energy beam may be attached to the robot arm 12 .
  • the robot arm 12 may be attached with a measuring device for measuring the workpiece W, which is an example of the end effector 13 .
  • the measuring device may be capable of measuring the characteristics of the workpiece W. At least one of the shape of the work W, the size of the work W, and the temperature of the work W is given as an example of the characteristics of the work W.
  • the measuring device may measure the workpiece W using a touch probe. In this case, a measuring device including a touch probe may be attached to the robot arm 12 .
  • the measuring device may measure the workpiece W by irradiating the workpiece W with an energy beam (for example, light, electromagnetic wave, and charged particle beam). In this case, a measuring device including an irradiation device that irradiates the workpiece W with an energy beam may be attached to the robot arm 12 .
  • an energy beam for example, light, electromagnetic wave, and charged particle beam
  • the control device 3 When at least one of a processing device and a measuring device as an example of the end effector 13 is attached to the robot arm 12, the control device 3 outputs a control signal for controlling the operation of at least one of the processing device and the measuring device. may be generated.
  • the control device 3 may generate a control signal for controlling rotation of a tool included in the processing device.
  • the control device 3 may generate a control signal for controlling on/off of the energy beam by the irradiation device provided in at least one of the processing device and the measurement device.
  • the imaging unit 2 includes the imaging device 22 and the projection device 23 in order to generate the work image data IMG_3D.
  • the imaging unit 2 does not have to be equipped with the projection device 23 in order to generate the work image data IMG_3D.
  • the imaging device 22 is a stereo camera, and therefore the workpiece shape data representing the shape of the target object such as the workpiece W can be obtained from the two workpiece image data IMG_3D respectively generated by the two imaging elements of the stereo camera. This is because WSD can be generated.
  • the shape data generation unit 311 calculates the parallax by associating each part (for example, each pixel) of the images indicated by the two image data included in the work image data IMG_3D by a known process, Work shape data WSD may be generated by a well-known method based on the principle of triangulation using the calculated parallax.
  • the imaging device 22 does not have to be a stereo camera.
  • the imaging device 22 may be a monocular camera that images a target object such as a workpiece W using a single imaging device.
  • the workpiece W on which the projection pattern is projected is reflected in the image indicated by the workpiece image data IMG_3D.
  • the three-dimensional shape of the work W on which the projection pattern is projected is reflected in the shape of the projection pattern reflected in the image indicated by the work image data IMG_3D. Therefore, even if the imaging device 22 is not a stereo camera, the shape data generator 311 generates the work shape data WSD by a known process based on the projection pattern appearing in the image indicated by the work image data IMG_3D. can be generated.
  • the imaging unit 2 may include either one of the imaging devices 21 and 22 (that is, a single imaging device) and the projection device 23.
  • image data generated by imaging the work W with a single imaging device while the projection device 23 is not projecting the desired projection pattern onto the work W may be used as the work image data IMG_2D. good.
  • image data generated by imaging the work W with a single imaging device while the projection device 23 is projecting a desired projection pattern onto the work W is used as the work image data IMG_3D. good.
  • the position calculator 312 calculates the position of the workpiece W based on the workpiece image data IMG_2D and the workpiece shape data WSD generated from the workpiece image data IMG_3D in step S3 of FIG.
  • the 2D matching unit 3121 performs 2D matching processing to obtain the position Tx (2D), the position Ty (2D), the position Tz (2D), the position Rx At least one of (2D), position Ry (2D) and position Rz (2D) may be calculated.
  • the 3D matching unit 3122 obtains a position Tx (3D), a position Ty (3D), a position Tz (3D), a position Rx (3D), a position At least one of Ry(3D) and position Rz(3D) may be calculated.
  • the 2D tracking unit 3123 performs 2D tracking processing to obtain the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (2D), the amount of change ⁇ Rx (2D), the amount of change ⁇ Ry (2D), and the amount of change At least one of the quantities ⁇ Rz(2D) may be calculated.
  • the 3D tracking unit 3124 performs 3D tracking processing to obtain the amount of change ⁇ Tx (3D), the amount of change ⁇ Ty (3D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and the amount of change At least one of the quantities ⁇ Rz(3D) may be calculated.
  • the imaging unit 2 includes either one of the imaging devices 21 and 22 (that is, a single imaging device)
  • the 2D imaging coordinate system and the 3D imaging coordinate system are the same coordinate system. Therefore, it can be said that the 2D matching unit 3121 and the 3D matching unit 3122 calculate the 6DOF position of the workpiece W in the same coordinate system.
  • the 2D tracking unit 3123 and the 3D tracking unit 3124 calculate the amount of change in the 6DOF position of the work W in the same coordinate system.
  • the coordinate transformation unit 3125 may calculate the position of the workpiece W in the global coordinate system by using the result of the 2D matching process, the result of the 3D matching process, the result of the 2D tracking process, and the 3D tracking process. Note that when the 2D imaging coordinate system and the 3D imaging coordinate system are the same coordinate system, the coordinate transformation unit 3125 does not need to perform coordinate transformation using the transformation matrix M32 or M23 described above.
  • the signal generator 313 may generate a robot control signal based on the calculated position of the workpiece W.
  • the imaging unit 2 may include an imaging device 22 that is a stereo camera, but may not include the imaging device 21 .
  • image data generated by capturing an image of the workpiece W by either one of the two monocular cameras included in the imaging device 22 may be used as the workpiece image data IMG_2D.
  • image data representing two images respectively generated by imaging the workpiece W with both of the two monocular cameras provided in the imaging device 22 may be used as the workpiece image data IMG_3D.
  • the position calculator 312 calculates the position of the workpiece W based on the workpiece image data IMG_2D and the workpiece shape data WSD generated from the workpiece image data IMG_3D in step S3 of FIG.
  • the 2D matching unit 3121 performs 2D matching processing to obtain the position Tx (2D), the position Ty (2D), the position Tz (2D), the position Rx At least one of (2D), position Ry (2D) and position Rz (2D) may be calculated.
  • the 3D matching unit 3122 obtains a position Tx (3D), a position Ty (3D), a position Tz (3D), a position Rx (3D), a position At least one of Ry(3D) and position Rz(3D) may be calculated.
  • the 2D tracking unit 3123 performs 2D tracking processing to obtain the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (2D), the amount of change ⁇ Rx (2D), the amount of change ⁇ Ry (2D), and the amount of change At least one of the quantities ⁇ Rz(2D) may be calculated.
  • the 3D tracking unit 3124 performs 3D tracking processing to obtain the amount of change ⁇ Tx (3D), the amount of change ⁇ Ty (3D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and the amount of change At least one of the quantities ⁇ Rz(3D) may be calculated.
  • the 2D imaging coordinate system and the 3D imaging coordinate system may be the same coordinate system, or the 2D imaging coordinate system and A coordinate system different from the 3D imaging coordinate system may be used.
  • a coordinate system in which an axis extending along the optical axis of one of the two monocular cameras is the Z axis may be used as each of the 2D imaging coordinate system and the 3D imaging coordinate system.
  • the 2D imaging coordinate system and the 3D imaging coordinate system may be the same coordinate system.
  • a coordinate system in which the axis extending along the optical axis of one of the two monocular cameras is the Z axis is used as the 2D imaging coordinate system
  • the A coordinate system in which the axis extending along the optical axis is the Z-axis may be used as each of the 3D imaging coordinate systems.
  • the 2D imaging coordinate system and the 3D imaging coordinate system may be different coordinate systems.
  • the 2D tracking unit 3123 and the 3D tracking unit 3124 calculate the amount of change in the 6DOF position of the work W in the same coordinate system.
  • the coordinate transformation unit 3125 may calculate the position of the workpiece W in the global coordinate system by using the result of the 2D matching process, the result of the 3D matching process, the result of the 2D tracking process, and the 3D tracking process. Note that when the 2D imaging coordinate system and the 3D imaging coordinate system are the same coordinate system, the coordinate transformation unit 3125 does not need to perform coordinate transformation using the transformation matrix M32 or M23 described above.
  • the signal generator 313 may generate a robot control signal based on the calculated position of the workpiece W.
  • control device 3 (particularly, the shape data generator 311) generates the work shape data WSD from the work image data IMG_3D.
  • a device different from the control device 3 may generate the workpiece shape data WSD from the workpiece image data IMG_3D.
  • the imaging unit 2 that generates the work image data IMG_3D may generate the work shape data WSD from the work image data IMG_3D.
  • the control device 3 may acquire the work shape data WSD from a device different from the control device 3 .
  • the control device 3 may perform at least one of the above-described 3D matching processing and 3D tracking processing using workpiece shape data WSD acquired from a device different from the control device 3 .
  • the control device 3 acquires the workpiece shape data WSD from the imaging unit 2 and performs at least one of the above-described 3D matching processing and 3D tracking processing. You may do one.
  • the coordinate transformation unit 3215 uses the transformation matrix M32 to transform the position Tz′ (3D), the position Rx′ (3D), and the position Ry′. (3D) to position Tz' (2D), position Rx' (2D) and position Ry' (2D).
  • the coordinate transformation unit 3215 may calculate the position Tz' (2D), the position Rx' (2D) and the position Ry' (2D) by the following method.
  • the coordinate transformation unit 3215 calculates the position Tz′ (3D), the position Rx′ (3D), and the position Ry′ (3D), and converts the amount of change ⁇ Tx (3D) to the position Tx (3D ) to calculate the position Tx′ (3D), add the amount of change ⁇ Ty (3D) to the position Ty (3D) to calculate the position Ty′ (3D), and calculate the amount of change ⁇ Rz (3D)
  • the position Rz' (3D) may be calculated by adding to the position Rz (3D).
  • (3D) and the method of calculating the position Rz′ (3D) by adding the change amount ⁇ Rz (3D) to the position Rz (3D) are the already explained change amount ⁇ Tz (3D) Since it may be the same as the method of calculating the position Tz' (3D) by adding it to the position Tz (3D), its detailed description is omitted.
  • the coordinate conversion unit 3215 may calculate the 6DOF position in the 3D imaging coordinate system based on the result of the 3D matching process and the result of the 3D tracking process.
  • the coordinate transformation unit 3215 may then calculate a rigid transformation matrix RTM30 from the 6DOF position in the 3D imaging coordinate system using a well-known method for computing rigid transformation matrices.
  • the coordinate transformation unit 3215 may calculate the transformation matrix TM30 representing the position of 6DOF in the 2D imaging coordinate system by multiplying the rigid body transformation matrix RTM30 and the transformation matrix M32.
  • the coordinate transformation unit 3215 may calculate the 6DOF position in the 2D imaging coordinate system from the calculated transformation matrix TM30 using a well-known method.
  • the coordinate conversion unit 3215 converts the calculated position of 6DOF in the 2D imaging coordinate system into the position in the Z-axis direction (2D) of the 2D imaging coordinate system, the position in the rotation direction around the X-axis (2D) of the 2D imaging coordinate system, and the positions in the rotation direction around the Y-axis (2D) of the 2D imaging coordinate system may be extracted as position Tz' (2D), position Rx' (2D) and position Ry' (2D), respectively.
  • the calculated position Tz'(2D) is calculated based on the results of the 3D matching process and the results of the 3D tracking process.
  • Each component of the 6DOF position in the coordinate system is the position that contributed to the calculation. That is, the calculated position Tz' (2D) includes not only the components of the position Tz' (3D), but also the position Tx' (3D), the position Ty' (3D), the position Rx' (3D), the position Ry' ( 3D) and position Rz'(3D) are also positions that contributed to the calculation.
  • the position Rx′ (2D) and the position Ry′ (2D) are calculated using the rigid transformation matrix, the closer the 2D imaging coordinate system is to the 3D imaging coordinate system, the more the calculated position Rx′ ( 2D) and position Ry' (2D) are more accurate.
  • the imaging devices 21 and 22 may be arranged so that the 2D imaging coordinate system is as close as possible to the 3D imaging coordinate system.
  • the imaging devices 21 and 22 may be arranged such that the X-axis (2D) of the 2D imaging coordinate system is as close as possible to the X-axis (3D) of the 3D imaging coordinate system.
  • the imaging devices 21 and 22 may be arranged such that the Y-axis (2D) of the 2D imaging coordinate system is as close as possible to the Y-axis (3D) of the 3D imaging coordinate system.
  • the imaging devices 21 and 22 may be arranged such that the Z axis (2D) of the 2D imaging coordinate system is as close as possible to the Z axis (3D) of the 3D imaging coordinate system.
  • the coordinate transformation unit 3215 uses the transformation matrix M23 to transform the position Tx′ (2D), the position Ty′ (2D) and Position Rz' (2D) is transformed into position Tx' (3D), position Ty' (3D) and position Rz' (3D).
  • the coordinate transformation unit 3215 may calculate the position Tx' (3D), the position Ty' (3D), and the position Rz' (3D) by the following method.
  • the coordinate transformation unit 3215 calculates the position Tx′ (2D), the position Ty′ (2D), and the position Rz′ (2D), and converts the change amount ⁇ Tz (2D) to the position Tz (2D ) to calculate the position Tz′ (2D), add the amount of change ⁇ Rx (2D) to the position Rx (2D) to calculate the position Rx′ (2D), and calculate the amount of change ⁇ Ry (2D) by
  • the position Ry' (2D) may be calculated by adding to the position Ry (2D).
  • (2D) and the method of calculating the position Ry′ (2D) by adding the change amount ⁇ Ry(2D) to the position Ry(2D)
  • the already explained change amount ⁇ Tx(2D) is Since it may be the same as the method of calculating the position Tx'(2D) by adding to the position Tx(2D), the detailed description thereof is omitted.
  • the coordinate transformation unit 3215 may calculate the 6DOF position in the 2D imaging coordinate system based on the result of the 2D matching process and the result of the 2D tracking process.
  • the coordinate transformation unit 3215 may then calculate the rigid transformation matrix RTM20 from the 6DOF position in the 2D imaging coordinate system using a well-known method for computing rigid transformation matrices.
  • the coordinate transformation unit 3215 may calculate the transformation matrix TM20 representing the position of 6DOF in the 3D imaging coordinate system by multiplying the rigid body transformation matrix RTM20 and the transformation matrix M21.
  • the coordinate transformation unit 3215 may calculate the position of 6DOF in the 3D imaging coordinate system from the calculated transformation matrix TM20.
  • the coordinate conversion unit 3215 converts the calculated 6DOF position in the 3D imaging coordinate system from the position in the X-axis direction (3D) of the 3D imaging coordinate system to the Y-axis direction ( 3D) and the position in the direction of rotation about the Z-axis (3D) of the 3D imaging coordinate system may be extracted as position Tx′ (3D), position Ty′ (3D) and position Rz′ (3D), respectively. good.
  • the calculated position Tx' (3D) is calculated based on the results of the 2D matching process and the 2D tracking process.
  • Each component of the 6DOF position in the coordinate system is the position that contributed to the calculation. That is, the calculated position Tx' (3D) includes not only the component of the position Tx' (2D), but also the position Ty' (2D), the position Tz' (2D), the position Rx' (2D), the position Ry' ( 2D) and position Rz′(2D) are also positions that contributed to the calculation.
  • the closer the 2D imaging coordinate system is to the 3D imaging coordinate system the more the position Tx'(2D) contributes to the calculation of the position Tx'(3D).
  • the imaging devices 21 and 22 may be arranged so that the 2D imaging coordinate system is as close as possible to the 3D imaging coordinate system.
  • the imaging devices 21 and 22 may be arranged such that the X-axis (2D) of the 2D imaging coordinate system is as close as possible to the X-axis (3D) of the 3D imaging coordinate system.
  • the imaging devices 21 and 22 may be arranged such that the Y-axis (2D) of the 2D imaging coordinate system is as close as possible to the Y-axis (3D) of the 3D imaging coordinate system.
  • the imaging devices 21 and 22 may be arranged such that the Z axis (2D) of the 2D imaging coordinate system is as close as possible to the Z axis (3D) of the 3D imaging coordinate system.
  • the coordinate transformation unit 3125 transforms the position Tx (2D), the position Ty (2D), the position Tz (3D), the position Rx (3D), the position Ry (3D), and the position Rz (2D) into positions within the common coordinate system.
  • the rigid transformation matrix described above may be used.
  • the coordinate transformation unit 3125 transforms the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and the amount of change ⁇ Rz (2D) into common coordinates.
  • the above-described rigid transformation matrix may be used.
  • the coordinate transformation unit 3215 uses the transformation matrix M32 to transform the position Tz (3D), the position Rx (3D), and the position Ry (3D) is converted into a position Tz (2D_conv), a position Rx (2D_conv), and a position Ry (2D_conv) within a 2D imaging coordinate system, which is a common coordinate system.
  • the coordinate transformation unit 3125 converts the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), and the amount of change ⁇ Ry (3D) into the 2D imaging coordinate system, which is the common coordinate system, using the conversion matrix M32.
  • the coordinate transformation unit 3215 uses a well-known method for calculating a rigid transformation matrix to convert the 6DOF position in the 3D imaging coordinate system calculated by 3D matching processing (that is, , position Tz(3D), position Ty(3D), position Tz(3D), position Rx(3D), position Ry(3D), and position Rz(3D)).
  • the coordinate transformation unit 3215 may calculate the transformation matrix TM31 representing the position of 6DOF in the 2D imaging coordinate system by multiplying the rigid body transformation matrix RTM31 and the transformation matrix M32. After that, the coordinate transformation unit 3215 may calculate the 6DOF position in the 2D imaging coordinate system from the calculated transformation matrix TM31 using a well-known method.
  • the coordinate conversion unit 3215 converts the calculated position of 6DOF in the 2D imaging coordinate system into the position in the Z-axis direction (2D) of the 2D imaging coordinate system, the position in the rotation direction around the X-axis (2D) of the 2D imaging coordinate system, and the positions in the rotation direction around the Y-axis (2D) of the 2D imaging coordinate system may be extracted as position Tz (2D_conv), position Rx (2D_conv), and position Ry (2D_conv), respectively.
  • the coordinate transformation unit 3215 uses a well-known technique for calculating a rigid body transformation matrix to calculate the amount of change (that is, the amount of change ⁇ Tz(3D ), the amount of change ⁇ Ty (3D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and the amount of change ⁇ Rz (3D)), the rigid transformation matrix RTM32 may be calculated. After that, the coordinate transformation unit 3215 may calculate the transformation matrix TM32 by multiplying the rigid body transformation matrix RTM32 and the transformation matrix M32. After that, the coordinate transformation unit 3215 may calculate the amount of change in the 6DOF position in the 2D imaging coordinate system from the calculated transformation matrix TM32.
  • the coordinate conversion unit 3215 calculates the amount of change in the position in the Z-axis direction (2D) of the 2D imaging coordinate system, the X-axis (2D) of the 2D imaging coordinate system, and the amount of change in the position of the 6DOF position in the 2D imaging coordinate system.
  • the amount of change in the position in the direction of rotation around and the amount of change in the direction of rotation in the direction of rotation around the Y-axis (2D) of the 2D imaging coordinate system are represented by the amount of change ⁇ Tz (2D_conv), the amount of change ⁇ Rx (2D_conv), and the amount of change ⁇ Ry, respectively. (2D_conv) may be extracted.
  • the imaging devices 21 and 22 may be arranged so that the 2D imaging coordinate system is as close as possible to the 3D imaging coordinate system.
  • the coordinate conversion unit 3215 uses the conversion matrix M23 to convert the position Tx (2D), the position Ty (2D), and Position Rz(2D) is converted into position Tx(3D_conv), position Ty(3D_conv) and position Rz(3D_conv) within the 3D imaging coordinate system, which is a common coordinate system.
  • the coordinate transformation unit 3125 converts the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), and the amount of change ⁇ Rz (2D) into the 3D imaging coordinate system, which is the common coordinate system, using the conversion matrix M23.
  • the amount of change ⁇ Tx (3D_conv), the amount of change ⁇ Ty (3D_conv), and the amount of change ⁇ Rz (3D_conv) are converted.
  • the coordinate transformation unit 3215 uses a well-known method for calculating a rigid body transformation matrix, and uses a 6DOF position in the 2D imaging coordinate system calculated by 2D matching processing (that is, , position Tz(2D), position Ty(2D), position Tz(2D), position Rx(2D), position Ry(2D), and position Rz(2D)).
  • the coordinate transformation unit 3215 may calculate the transformation matrix TM21 representing the position of 6DOF in the 3D imaging coordinate system by multiplying the rigid body transformation matrix RTM21 and the transformation matrix M23. After that, the coordinate transformation unit 3215 may calculate the position of 6DOF in the 3D imaging coordinate system from the calculated transformation matrix TM21 using a well-known method.
  • the coordinate conversion unit 3215 converts the calculated 6DOF position in the 3D imaging coordinate system to the position in the X-axis direction (3D) of the 3D imaging coordinate system, the position in the Y-axis direction (3D) of the 3D imaging coordinate system, and the 3D imaging
  • the positions in the direction of rotation about the Z-axis (3D) of the coordinate system may be extracted as position Tx (3D_conv), position Ty (3D_conv), and position Rz (3D_conv), respectively.
  • the coordinate transformation unit 3215 uses a well-known technique for calculating a rigid body transformation matrix to calculate the amount of change in the 6DOF position in the 2D imaging coordinate system calculated by the 2D tracking process (that is, the amount of change ⁇ Tz(2D ), the amount of change ⁇ Ty(2D), the amount of change ⁇ Tz(2D), the amount of change ⁇ Rx(2D), the amount of change ⁇ Ry(2D), and the amount of change ⁇ Rz(2D)), the rigid transformation matrix RTM22 may be calculated. After that, the coordinate transformation unit 3215 may calculate the transformation matrix TM22 by multiplying the rigid body transformation matrix RTM22 and the transformation matrix M23.
  • the coordinate transformation unit 3215 may calculate the amount of change in the 6DOF position in the 3D imaging coordinate system from the calculated transformation matrix TM22.
  • the coordinate conversion unit 3215 calculates the amount of change in the position of the 6DOF position in the 3D imaging coordinate system, the amount of change in the X-axis direction (3D) of the 3D imaging coordinate system, the Y-axis direction (3D ) and the positional change in the direction of rotation about the Z-axis (3D) of the 3D imaging coordinate system, respectively, are the amount of change ⁇ Tx (3D_conv), the amount of change ⁇ Ty (3D_conv), and the amount of change ⁇ Rz (3D_conv) can be extracted as Also in this case, for the reasons already explained, as the 2D imaging coordinate system approaches the 3D imaging coordinate system, the calculated position Tx (3D_conv), position Ty (3D_conv), and position Rz (3D_conv), and The accuracy of the amount of change ⁇ Tx(3D_conv), the
  • the control device 3 controls the position Tx' (2D), the position Ty' (2D), the position Tz' (3D), the position Rx' (3D), the position Ry' (3D) and the position Rz' (2D). ), the position of the workpiece W within the global coordinate system is calculated. However, the control device 3 controls position Tx' (2D), position Ty' (2D), position Tz' (3D), position Rx' (3D), position Ry' (3D) and position Rz' (2D).
  • the position of the workpiece W within the global coordinate system may be calculated without using at least one of
  • the control device 3 calculates the position Tx' (3D) based on the position Tx (3D) and the amount of change ⁇ Tx (3D), and based on the calculated position Tx' (3D), in the global coordinate system may be calculated.
  • the control device 3 calculates the position Ty' (3D) based on the position Ty (3D) and the amount of change ⁇ Ty (3D), and moves the workpiece within the global coordinate system based on the calculated position Ty' (3D).
  • the position of W may be calculated.
  • the control device 3 calculates a position Tz'(2D) based on the position Tz(2D) and the amount of change ⁇ Tz(2D), and based on the calculated position Tz'(2D), moves the workpiece within the global coordinate system.
  • the position of W may be calculated.
  • the control device 3 calculates the position Rx'(2D) based on the position Rx(2D) and the amount of change ⁇ Rx(2D), and based on the calculated position Rx'(2D), moves the workpiece within the global coordinate system.
  • the position of W may be calculated.
  • the control device 3 calculates the position Ry' (2D) based on the position Ry (2D) and the change amount ⁇ Ry (2D), and moves the workpiece within the global coordinate system based on the calculated position Ry' (2D).
  • the position of W may be calculated.
  • the control device 3 calculates a position Rz' (3D) based on the position Rz (3D) and the change amount ⁇ Rz (3D), and based on the calculated position Rz' (3D), moves the workpiece within the global coordinate system.
  • the position of W may be calculated.
  • control device 3 controls the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Rz (2D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D ), position Tx (2D), position Ty (2D) and position Rz (2D), and position Tz (3D), position Rx (3D) and position Ry (3D), the workpiece in the global coordinate system
  • the position of W is calculated.
  • control device 3 controls the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D) and the amount of change ⁇ Rz (2D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D) and the amount of change ⁇ Ry (3D), the position global coordinate system without using at least one of Tx(2D), position Ty(2D) and position Rz(2D), and position Tz(3D), position Rx(3D) and position Ry(3D) You may calculate the position of the workpiece
  • control device 3 the 6DOF position calculated by the 2D matching processing unit 3121, the 6DOF position calculated by the 3D matching processing unit 3122, the amount of change in the 6DOF position calculated by the 2D tracking processing unit 3123, and the 3D tracking
  • the position of the workpiece W within the global coordinate system may be calculated based on at least one of the amount of change in the 6DOF position calculated by the processing unit 3124 .
  • the control device 3 corrects the result of the 2D matching process based on the result of the 2D tracking process, and corrects the result of the 3D matching process based on the result of the 3D tracking process, so that the global coordinates
  • the position of the workpiece W in the system is calculated.
  • the control device 3 may calculate the position of the workpiece W in the global coordinate system based on the results of the 2D matching process and the 3D matching process without using the results of the 2D tracking process and the 3D tracking process.
  • the coordinate transformation unit 3125 transforms the position Tx (2D), the position Ty (2D), and the position Rz (2D) calculated by the 2D matching process into the position Tx' (2D), the position Ty' (2D), and the position Rz'.
  • the control device 3 calculates intermediately in the process of calculating the position of the work W within the global coordinate system. There is no change in considering the difference in the accuracy of the 6DOF position of the workpiece W to be processed.
  • the control device 3 can calculate the position of the work W with a correspondingly high degree of accuracy, compared to the case where the difference in accuracy of the 6DOF position of the work W calculated intermediately is not taken into account. Therefore, the robot 1 moves accurately and at high speed with respect to the work W (for example, accurately and at high speed follows (approaches) the work W) without losing sight of the work W (that is, the target object), and ends. Desired processing can be performed on the workpiece W by the effector 13 . When the work W is stationary, the robot 1 can approach the work W accurately and at high speed, and perform desired processing on the work W with the end effector 13 .
  • control device 3 may calculate (that is, update) the position of the workpiece W within the global coordinate system at a cycle corresponding to the 2D matching cycle or the 3D matching cycle. As a result, the control device 3 may output the control signal to the robot 1 at a cycle corresponding to the 2D matching cycle or the 3D matching cycle.
  • the 2D matching unit 3121 calculates the position Tx (2D), the position Ty (2D), the position Tz (2D), the position Rx (2D), the position Ry (2D), and the position Rz (2D). good too.
  • the 3D matching unit 3122 uses the position Tx (2D), the position Ty (2D), the position Tz (2D), the position Rx (2D), the position Ry (2D) and the position Rz (2D) to match the position Tx (3D ), position Ty (3D), position Tz (3D), position Rx (3D), position Ry (3D) and position Rz (3D).
  • the 3D matching unit 3122 uses the position Tx (2D), the position Ty (2D), the position Tz (2D), the position Rx (2D), the position Ry (2D), and the position Rz (2D) to determine the workpiece shape
  • a part of the data WSD is selected as 3D matching target data for 3D matching processing, and by performing 3D matching processing using the selected 3D matching target data, position Tx (3D), position Ty (3D), position Tz(3D), position Rx(3D), position Ry(3D) and position Rz(3D) may be calculated.
  • the 2D tracking unit 3123 performs the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (2D), the amount of change ⁇ Rx (2D), the amount of change ⁇ Ry (2D), and the amount of change ⁇ Rz ( 2D) may be calculated.
  • the 3D tracking unit 3124 calculates the amount of change ⁇ Tx (3D), the amount of change ⁇ Ty (3D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and the amount of change ⁇ Rz (3D). need not be calculated.
  • the 3D tracking unit 3124 calculates the amount of change ⁇ Tx (3D), the amount of change ⁇ Ty (3D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and the amount of change ⁇ Rz (3D). Even if at least one is calculated, it is not necessary to output the calculated amount of change in the position of the workpiece W to the coordinate conversion section 3125 .
  • the 3D tracking unit 3124 does not calculate the amount of change ⁇ Tx (3D), the amount of change ⁇ Ty (3D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and the amount of change ⁇ Rz (3D).
  • the position calculator 312 does not have to include the 3D tracking unit 3124 .
  • the coordinate transformation unit 3125 uses the results of the 2D matching processing, the results of the 3D matching processing, and the results of the 2D tracking processing, while the position of the workpiece W in the global coordinate system is calculated without using the results of the 3D tracking processing. may be calculated.
  • the coordinate transformation unit 3125 may correct the result of the 2D matching process based on the result of the 2D tracking process and correct the result of the 3D matching process based on the result of the 2D tracking process. For example, the coordinate transformation unit 3125 may calculate the position Tx'(2D) by correcting the position Tx(2D) based on the amount of change ⁇ Tx(2D). The coordinate transformation unit 3125 may calculate Ty'(2D) by correcting the position Ty(2D) based on the amount of change ⁇ Ty(2D).
  • the coordinate conversion unit 3125 converts the change amount ⁇ Tz(2D) in the 2D imaging coordinate system into the change amount ⁇ Tz(3D_conv) in the 3D imaging coordinate system, and converts the position Tz(3D) based on the converted change amount ⁇ Tz(3D_conv).
  • the position Tz' (3D) may be calculated by correcting .
  • the coordinate transformation unit 3125 transforms the change amount ⁇ Rx(2D) in the 2D imaging coordinate system into the change amount ⁇ Rx(3D_conv) in the 3D imaging coordinate system, and converts the position Rx(3D) based on the converted change amount ⁇ Rx(3D_conv).
  • the position Rx' (3D) may be calculated by correcting .
  • the coordinate conversion unit 3125 converts the change amount ⁇ Ry(2D) in the 2D imaging coordinate system into the change amount ⁇ Ry(3D_conv) in the 3D imaging coordinate system, and converts the position Ry(3D) based on the converted change amount ⁇ Ry(3D_conv).
  • the position Ry' (3D) may be calculated by correcting .
  • the coordinate transformation unit 3125 may calculate the position Rz'(2D) by correcting the position Rz(2D) based on the amount of change ⁇ Rz(2D). Subsequent processing may be identical to the processing described above.
  • the coordinate transformation unit 3125 may calculate the position of the work W in the global coordinate system based on the work shape data WSD in addition to the results of the 2D matching process, the results of the 3D matching process, and the results of the 2D tracking process. .
  • the coordinate conversion unit 3125 converts the image pickup device 22 to the workpiece W at time s2, which is the later of times s1 and s2, in addition to the result of the 2D matching process, the result of the 3D matching process, and the result of the 2D tracking process.
  • the position of the work W in the global coordinate system may be calculated based on the work shape data WSD generated from the work image data IMG_3D#s2 generated by imaging the .
  • the signal generator 313 Since the signal generator 313 generates the robot control signal based on the position of the workpiece W in the global coordinate system, in this case, the signal generator 313 generates the result of the 2D matching process, the result of the 3D matching process, and the 2D It may be considered that the robot control signal is generated based on the workpiece shape data WSD in addition to the result of the tracking process.
  • the above-described 2D tracking unit 3123 is based on two workpiece image data IMG_2D#t1 and IMG_2D#t2 generated by imaging the workpiece W by the imaging device 21 at different times t1 and t2, and the workpiece shape data WSD. , the amount of change in the position of the workpiece W within the 2D imaging coordinate system between times t1 and t2 may be calculated.
  • the 2D tracking unit 3123 based on the two workpiece image data IMG_2D#t1 and IMG_2D#t2 and the workpiece shape data WSD, the above-described change amount ⁇ Tx(2D), change amount ⁇ Ty(2D), change amount ⁇ Tz( 2D), the amount of change ⁇ Rx(2D), the amount of change ⁇ Ry(2D), and the amount of change ⁇ Rz(2D).
  • the 2D tracking unit 3123 is generated by imaging the workpiece W by the imaging device 22 at time s2 which is the later of the two workpiece image data IMG_2D#t1 and IMG_2D#t2, which are different from each other.
  • the amount of change in the position of the workpiece W may be calculated based on the workpiece shape data WSD generated from the workpiece image data IMG_3D#s2.
  • the coordinate transformation unit 3125 converts the result of the 2D matching process, the result of the 3D matching process, and the result of the 2D tracking process (that is, based on the two workpiece image data IMG_2D#t1 and IMG_2D#t2 and the workpiece shape data WSD).
  • the position of the workpiece W in the global coordinate system may be calculated based on the amount of change calculated by Since the signal generator 313 generates the robot control signal based on the position of the workpiece W in the global coordinate system, in this case, the signal generator 313 generates the result of the 2D matching process, the result of the 3D matching process, and the 2D Even if it is assumed that the robot control signal is generated based on the result of the tracking process (that is, the amount of change calculated based on the two workpiece image data IMG_2D#t1 and IMG_2D#t2 and the workpiece shape data WSD). good.
  • the 3D tracking unit 3124 performs the amount of change ⁇ Tx (3D), the amount of change ⁇ Ty (3D), the amount of change ⁇ Tz (3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and the amount of change ⁇ Rz (3D). ) may be calculated.
  • the 2D tracking unit 3123 calculates the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (2D), the amount of change ⁇ Rx (2D), the amount of change ⁇ Ry (2D), and the amount of change ⁇ Rz (2D). need not be calculated.
  • the 2D tracking unit 3123 calculates the amount of change ⁇ Tx (2D), the amount of change ⁇ Ty (2D), the amount of change ⁇ Tz (2D), the amount of change ⁇ Rx (2D), the amount of change ⁇ Ry (2D), and the amount of change ⁇ Rz (2D). Even if at least one is calculated, it is not necessary to output the calculated amount of change in the position of the workpiece W to the coordinate conversion section 3125 .
  • the 2D tracking unit 3123 does not calculate amount of change ⁇ Tx(2D), amount of change ⁇ Ty(2D), amount of change ⁇ Tz(2D), amount of change ⁇ Rx(2D), amount of change ⁇ Ry(2D) and amount of change ⁇ Rz(2D)
  • the position calculator 312 does not have to include the 2D tracking unit 3123 .
  • the coordinate transformation unit 3125 uses the result of the 2D matching process, the result of the 3D matching process, and the result of the 3D tracking process, while the position of the workpiece W in the global coordinate system is calculated without using the result of the 2D tracking process. may be calculated.
  • the coordinate transformation unit 3125 corrects the result of the 2D matching process based on the result of the 3D tracking process, and corrects the result of the 3D matching process based on the result of the 3D tracking process.
  • the coordinate conversion unit 3125 converts the amount of change ⁇ Tx (3D) in the 3D imaging coordinate system into the amount of change ⁇ Tx (2D_conv) in the 2D imaging coordinate system, and the position Tx ( 2D) may be corrected to calculate the position Tx'(2D).
  • the coordinate conversion unit 3125 converts the change amount ⁇ Ty(3D) in the 3D imaging coordinate system into the change amount ⁇ Ty(2D_conv) in the 2D imaging coordinate system, and converts the position Ty(2D) based on the converted change amount ⁇ Ty(2D_conv). Ty'(2D) may be calculated by correcting .
  • the coordinate transformation unit 3125 may calculate the position Tz'(3D) by correcting the position Tz(3D) based on the amount of change ⁇ Tz(3D).
  • the coordinate transformation unit 3125 may calculate the position Rx'(3D) by correcting the position Rx(3D) based on the amount of change ⁇ Rx(3D).
  • the coordinate transformation unit 3125 may calculate the position Ry'(3D) by correcting the position Ry(3D) based on the amount of change ⁇ Ry(3D).
  • the coordinate transformation unit 3125 transforms the amount of change ⁇ Rz(3D) in the 3D imaging coordinate system into the amount of change ⁇ Rz(2D_conv) in the 2D imaging coordinate system, and converts the amount of change ⁇ Rz(2D_conv) into the position Rz(2D).
  • the position Rz' (2D) may be calculated by correcting . Subsequent processing may be identical to the processing described above.
  • the coordinate transformation unit 3125 may calculate the amount of change ⁇ Tx(2D_conv), the amount of change ⁇ Ty(2D_conv), and the amount of change ⁇ Rz(2D_conv) using the rigid body transformation matrix as described above.
  • the above-described 3D tracking unit 3124 is based on two workpiece image data IMG_2D#t1 and IMG_2D#t2 generated by imaging the workpiece W by the imaging device 21 at different times t1 and t2, and the workpiece shape data WSD. , the amount of change in the position of the workpiece W within the 3D imaging coordinate system between times t1 and t2 may be calculated.
  • the 3D tracking unit 3124 based on the two workpiece image data IMG_2D#t1 and IMG_2D#t2 and the workpiece shape data WSD, changes the amount of change ⁇ Tx (3D), the amount of change ⁇ Ty (3D), the amount of change ⁇ Tz ( 3D), the amount of change ⁇ Rx (3D), the amount of change ⁇ Ry (3D), and the amount of change ⁇ Rz (3D).
  • the 3D tracking unit 3124 is generated by imaging the workpiece W by the imaging device 22 at time s2, which is the later of the two workpiece image data IMG_2D#t1 and IMG_2D#t2, which are different from each other.
  • the amount of change in the position of the workpiece W may be calculated based on the workpiece shape data WSD generated from the workpiece image data IMG_3D#s2.
  • the coordinate transformation unit 3125 converts the result of the 2D matching process, the result of the 3D matching process, and the result of the 3D tracking process (that is, based on the two workpiece image data IMG_2D#t1 and IMG_2D#t2 and the workpiece shape data WSD).
  • the position of the workpiece W in the global coordinate system may be calculated based on the amount of change calculated by Since the signal generator 313 generates the robot control signal based on the position of the workpiece W in the global coordinate system, in this case, the signal generator 313 generates the result of the 2D matching process, the result of the 3D matching process, and the 3D Even if it is assumed that the robot control signal is generated based on the result of the tracking process (that is, the amount of change calculated based on the two workpiece image data IMG_2D#t1 and IMG_2D#t2 and the workpiece shape data WSD). good.
  • the control device 3 may include a robot control device 14.
  • the control device 3 may function as the robot control device 14 .
  • a logical functional block capable of functioning as the robot control device 14 may be realized in the arithmetic device 31 of the control device 3 .
  • the control device 3 may control the operation of the robot 1 (for example, control the driving of the actuator built in the joint 122 of the robot arm 12) based on the robot control signal generated by the control device 3. In this case, the robot 1 does not have to be equipped with the robot controller 14 .
  • the shape data generation unit 311 of the control device 3 generates a work shape from one work image data IMG_3D (for example, one work image data IMG_3D containing two image data when the imaging device 22 is a stereo camera). Data WSD is generated.
  • the shape data generator 311 may generate the work shape data WSD from a plurality of pieces of work image data IMG_3D.
  • the shape data generator 311 may generate the work shape data WSD from a plurality of pieces of work image data IMG_3D obtained by imaging the work W from different directions.
  • the signal generation unit 313 moves the work W in different directions with the position of the work W as a starting point.
  • the image pickup unit 2 that is, the robot arm 11
  • the signal generator 313 generates a robot control signal for operating the robot arm 11 so that the workpiece W can be imaged from different directions starting from the position of the workpiece W, and outputs the signal to the robot controller 14 .
  • the imaging device 22 may generate a plurality of pieces of workpiece image data IMG_3D obtained by imaging the workpiece W from different directions by performing imaging multiple times as the imaging unit 2 moves from the position of the workpiece W as a starting point. .
  • the shape data generator 311 may generate shape data for each of the plurality of workpiece image data IMG_3D through the above process. This shape data can also be said to be work shape data WSD.
  • the shape data generator 311 may generate the workpiece shape data WSD by synthesizing a plurality of generated shape data by existing processing.
  • the shape data generation unit 311 may synthesize a plurality of point cloud data by performing ICP (Iterative Closest Point) processing.
  • ICP Intelligent Closest Point
  • the shape data generation unit 311 performs a process of aligning the positions of the feature locations on the depth image indicated by the respective depth image data to generate a plurality of depth image data.
  • Depth image data may be synthesized.
  • the position calculation unit 312 calculates the position of the work W by executing at least one of the above-described matching processing and tracking processing using the work shape data WSD generated by the synthesizing process by the shape data generation unit 311, and outputs a signal
  • the generation unit 313 may generate a robot control signal based on the position of the work W calculated by the position calculation unit 312 and output it to the robot control device 14 so that the end effector 13 can process the work W. .
  • a control device that generates a control signal for controlling a robot that moves an imaging unit, a calculation unit that generates the control signal; an output unit that outputs the control signal generated by the calculation unit; with
  • the calculation unit is within a first coordinate system defined by a first axis, a second axis, and a third axis that are orthogonal to each other, the first a position of the object in at least one of a first direction parallel to a first axis, a second direction parallel to the second axis, and a third direction parallel to the third axis; the first axis; calculating a first position including at least one of a pose of the object about at least one of the second axis and the third axis; Based on shape data representing a three-dimensional shape of the object and generated from second image data acquired by the imaging unit capturing an image of the object, a fourth axis, a fifth axi
  • the first image data is generated by the image capturing unit capturing an image of the object during relative movement between the image capturing unit and the object, The control device according to appendix 1, wherein the second image data is generated by imaging the object with the imaging unit during the relative movement.
  • the calculation unit is the at least one direction between the different times within the first coordinate system based on two pieces of the first image data obtained by imaging the object by the imaging unit at mutually different times; calculating a first amount of change that is the amount of change in at least one of the position of the object in and the orientation of the object about the at least one axis; Based on two pieces of shape data representing the three-dimensional shape of the object generated from each of the two pieces of second image data obtained by imaging the object by the imaging unit at different times, the second a second amount of change that is the amount of change in at least one of the position of the object in the at least one direction and the orientation of the object about the at least one axis between the different times in the coordinate system; calculate, 5.
  • control device according to any one of appendices 1 to 4, wherein the control signal is generated based on the first position, the first amount of change, the second position, and the second amount of change.
  • the output unit outputs the control signal generated by the calculation unit based on the first position, the first change amount, the second position, and the second change amount in a second period. Control device as described.
  • Appendix 7 The mutually different times at which the imaging unit images the object to obtain the two first image data and the mutually different times at which the imaging unit images the object to obtain the two second image data 7.
  • the control device according to appendix 5 or 6, wherein the times are the same.
  • [Appendix 9] A time from when the object is imaged by the imaging unit until the first position is calculated based on the first image data, and after the object is imaged by the imaging unit based on the second image data The control device according to appendix 8, wherein the time until the second position is calculated is the same.
  • [Appendix 10] a time from when the imaging unit images the object at different times until the first amount of change is calculated based on the two first image data, and when the object is captured at different times by the imaging unit The control device according to appendix 8 or 9, wherein the time from imaging to calculation of the second amount of change based on the two second image data is the same.
  • the calculation unit is calculating the first position in a third period; calculating the first amount of change in a fourth period shorter than the third period; calculating the second position in a fifth period; 11.
  • [Appendix 12] 12.
  • the control device according to appendix 11, wherein the fourth period and the sixth period are the same period.
  • [Appendix 13] 13 13.
  • the output unit outputs the control signal generated by the calculation unit based on the first position, the first amount of change, the second position, and the second amount of change in a second period, 14.
  • the calculation unit is A seventh direction parallel to the seventh axis within a common coordinate system defined by a seventh axis, an eighth axis, and a ninth axis orthogonal to each other based on the first position and the first change amount , an eighth direction parallel to the eighth axis, and a ninth direction parallel to the ninth axis; calculating a third position including at least one of a pose of the object about at least one of nine axes; a position of the object in at least one remaining direction different from the at least one direction and the at least one axis within the common coordinate system based on the second position and the second amount of change; calculating a fourth position comprising at least one of a different pose of said object about at least one remaining axis; 17.
  • Appendix 18 18.
  • the calculation unit is calculating the first position in a third period; calculating the first amount of change in a fourth period shorter than the third period; calculating the second position in a fifth period; calculating the second amount of change in a sixth cycle shorter than the fifth cycle; The first amount of change is newly calculated in the fourth period after the first position is newly calculated as one period of the third period and before the next calculation of the first position.
  • the control device each time, calculating the third position based on the newly calculated first change amount and the newly calculated first position;
  • the second amount of change is newly calculated in the sixth period after the second position is newly calculated as one period of the fifth period until the second position is next calculated.
  • the control device according to appendix 17 or 18, wherein the fourth position is calculated based on the newly calculated second amount of change and the newly calculated second position each time.
  • the seventh axis is the first axis
  • the eighth axis is the second axis
  • the ninth axis is the third axis
  • the calculation unit is At least the position of the object in the at least one direction and the pose of the object about the at least one axis within the common coordinate system by correcting the first position with the first amount of change.
  • the seventh axis is the fourth axis
  • the eighth axis is the fifth axis
  • the ninth axis is the sixth axis
  • a position of the object in the at least one direction and a position of the object about the at least one axis within the first coordinate system calculated by correcting the first position with the first change amount
  • calculating the third position in the common coordinate system based on at least one of the orientation
  • 20 The control device according to any one of appendices 17 to 19, wherein at least one of posture and posture is calculated as the fourth position.
  • the calculation unit is At least the position of the object in the at least one direction and the pose of the object about the at least one axis within the common coordinate system by correcting the first position with the first amount of change.
  • the control device according to any one of appendices 17 to 19, wherein the fourth position is calculated by correcting with at least one change amount.
  • the seventh axis is the fourth axis
  • the eighth axis is the fifth axis
  • the ninth axis is the sixth axis
  • the calculation unit is At least one of the position of the object in the at least one direction and the pose of the object about the at least one axis in the common coordinate system calculated based on the first position is defined as the first correcting at least one of the position of the object in the at least one direction and the orientation of the object about the at least one axis within the common coordinate system, which is calculated based on the amount of change.
  • the third position includes a position of the object in the eighth direction, a position of the object in the ninth direction, and a pose of the object about the seventh axis;
  • the fourth position includes a position of the object in the seventh direction, a pose of the object about the eighth axis, and a pose of the object about the ninth axis;
  • the seventh axis is the first axis, the eighth axis is the second axis, the ninth axis is the third axis, 23.
  • the control device according to any one of appendices 17 to 20 and 22, wherein the first axis is an axis along an optical axis of an optical system of the imaging section.
  • the third position includes a position of the object in the eighth direction, a position of the object in the ninth direction, and a pose of the object about the seventh axis;
  • the fourth position includes a position of the object in the seventh direction, a pose of the object about the eighth axis, and a pose of the object about the ninth axis;
  • the seventh axis is the fourth axis,
  • the eighth axis is the fifth axis,
  • the ninth axis is the sixth axis, 24.
  • the control device according to any one of appendices 17 to 19, 21 and 23, wherein the fourth axis is an axis along the optical axis of the optical system of the imaging unit.
  • the first axis is an axis along the optical axis of the optical system of the imaging unit
  • the fourth axis is an axis along the optical axis of the optical system of the imaging unit
  • the first position includes a position of the object in the second direction, a position of the object in the third direction, and a pose of the object about the first axis
  • a control device according to any one of the preceding paragraphs.
  • the first axis is an axis along the optical axis of the optical system of the imaging unit
  • the fourth axis is an axis along the optical axis of the optical system of the imaging unit
  • the first position includes a position of the object in the second direction, a position of the object in the third direction, and a pose of the object about the first axis
  • the first amount of change includes an amount of change in the position of the object in the second direction, an amount of change in the position of the object in the third direction, and an amount of change in posture of the object about the first axis.
  • the second position includes a position of the object in the fourth direction, a pose of the object about the fifth axis, and a pose of the object about the sixth axis;
  • the second amount of change is an amount of change in position of the object in the fourth direction, an amount of change in posture of the object about the fifth axis, and an amount of change in the posture of the object about the sixth axis.
  • the control device according to any one of appendices 5 to 25.
  • the calculation unit is calculating the first position by performing a matching process using the first image data and reference image data representing a two-dimensional image serving as a reference of the object; 28.
  • the control device according to any one of appendices 1 to 27, wherein the second position is calculated by performing a matching process using the shape data and reference shape data indicating a three-dimensional shape serving as a reference of the object.
  • the calculation unit is calculating the first position by performing a matching process using the first image data and reference image data representing a two-dimensional image serving as a reference of the object; calculating the second position by performing a matching process using the shape data and reference shape data indicating a three-dimensional shape serving as a reference of the object; a data portion indicating a first characteristic portion of the object in one of the first image data and the first characteristic portion of the object in the other first image data of the two first image data; Calculate the first amount of change by performing tracking processing using the data part shown, Tracking using a data portion indicating the second characteristic portion of the object in one of the shape data and a data portion indicating the second characteristic portion of the object in the other shape data of the two shape data 28.
  • the control device according to any one of appendices 5 to 25 and 27, wherein the second amount of change is calculated by performing processing.
  • the calculation unit is Selecting a data portion indicating at least a partial shape of the object from the shape data based on the first position, and performing matching processing using the selected data portion and the reference shape data. 30.
  • the control device according to appendix 28 or 29, wherein the second position is calculated by .
  • the calculation unit is subjecting the shape data to data deletion processing for deleting a data portion indicating a shape of another object different from the object; 31.
  • the control device according to any one of appendices 28 to 30, wherein the second position is calculated by performing a matching process using the shape data subjected to the data deletion process and the reference shape data.
  • Appendix 32 32.
  • Appendix 33 The robot is provided with a processing device for processing the object, 33.
  • [Appendix 34] 34 34.
  • the control device according to appendix 33, wherein the control signal is a signal for controlling the processing device.
  • Appendix 35 35.
  • the control device according to any one of appendices 1 to 34, wherein the imaging unit can automatically adjust exposure based on a result of imaging the object by the imaging unit.
  • the calculation unit is performing image processing on at least one of the first image data and the second image data; The first position can be calculated based on the first image data subjected to the image processing, 36.
  • the control device according to any one of appendices 1 to 35, wherein the second position can be calculated based on the shape data generated from the second image data.
  • the calculation unit is performing image processing on at least one of the two first image data and the two second image data; It is possible to calculate the first amount of change based on the two first image data subjected to the image processing, 30.
  • a control device according to any one of the preceding paragraphs.
  • the imaging unit generates at least one of the second image data and the two second image data by imaging the object onto which pattern light is projected from a light projection device provided on the robot arm. 39. Control device according to any one of the appendices 1 to 38.
  • a control device A control device according to any one of claims 1 to 39; A control system comprising: the imaging unit; [Appendix 41]
  • the imaging unit includes a first imaging device and a second imaging device, a first filter that attenuates a light component in a first wavelength band of light from the object; a second filter that attenuates a light component in a second wavelength band different from the first wavelength band out of the light from the object,
  • the first imaging device captures an image of the object by receiving light from the object through the first filter, 41.
  • the control system according to appendix 40, wherein the second imaging device images the object by receiving light from the object through the second filter.
  • a control device according to any one of claims 1 to 39; the imaging unit; A robot system comprising: the robot; [Appendix 45] A control method for generating a control signal for controlling a robot that moves an imaging unit, Within a first coordinate system defined by a first axis, a second axis, and a third axis that are orthogonal to each other, the first a position of the object in at least one of a first direction parallel to a first axis, a second direction parallel to the second axis, and a third direction parallel to the third axis; the first axis; calculating a first position comprising at least one of a pose of the object about at least one of the second axis and the third axis; Based on the shape data representing the three-dimensional shape of the object generated from the second image data acquired by the imaging unit imaging the object, a fourth axis, a fifth axis, and a third axis orthogonal to each other are generated.
  • Appendix 46 A computer program that causes a computer to execute the control method according to appendix 45.
  • a control device that generates a control signal for controlling a robot that moves an imaging unit, a calculation unit that generates the control signal; an output unit that outputs the control signal generated by the calculation unit; with
  • the calculation unit is within a first coordinate system defined by a first axis, a second axis, and a third axis that are orthogonal to each other, the first a position of the object in at least one of a first direction parallel to a first axis, a second direction parallel to the second axis, and a third direction parallel to the third axis; the first axis; calculating a first position including at least one of a pose of the object about at least one of the second axis and the third axis; acquiring shape data generated by the imaging unit and indicating a three-dimensional shape of the object from the second image data acquired by the imaging unit imaging the object, from the imaging unit; based on the shape data, in a second coordinate system defined by mutually orthogonal fourth, fifth, and sixth axe
  • the first image data is generated by the image capturing unit capturing an image of the object during relative movement between the image capturing unit and the object, 48.
  • the control device according to appendix 47, wherein the second image data is generated by imaging the object with the imaging unit during the relative movement.
  • the control device according to appendix 47 or 48, wherein the output unit outputs the control signal generated by the calculation unit based on the first position and the second position in a first cycle.
  • the time at which the imaging unit images the object to obtain the first image data and the time at which the imaging unit images the object to obtain the second image data are the same.
  • the calculation unit is the at least one direction between the different times within the first coordinate system based on two pieces of the first image data obtained by imaging the object by the imaging unit at mutually different times; calculating a first amount of change that is the amount of change in at least one of the position of the object in and the orientation of the object about the at least one axis; Based on two pieces of shape data representing the three-dimensional shape of the object generated from each of the two pieces of second image data obtained by imaging the object by the imaging unit at different times, the second a second amount of change that is the amount of change in at least one of the position of the object in the at least one direction and the orientation of the object about the at least one axis between the different times in the coordinate system; calculate, 51.
  • control device The control device according to any one of appendices 47 to 50, wherein said control signal is generated based on said first position, said first amount of change, said second position and said second amount of change.
  • the output unit outputs the control signal generated by the calculation unit based on the first position, the first change amount, the second position, and the second change amount in a second period. Control device as described.
  • Appendix 53 The mutually different times at which the imaging unit images the object to obtain the two first image data and the mutually different times at which the imaging unit images the object to obtain the two second image data 53.
  • the control device according to appendix 51 or 52, wherein the times are the same.
  • [Appendix 55] A time from when the object is imaged by the imaging unit until the first position is calculated based on the first image data, and after the object is imaged by the imaging unit based on the second image data 55.
  • the control device according to appendix 54, wherein the time until the second position is calculated is the same.
  • [Appendix 56] a time from when the imaging unit images the object at different times until the first amount of change is calculated based on the two first image data, and when the object is captured at different times by the imaging unit 56.
  • the calculation unit is calculating the first position in a third period; calculating the first amount of change in a fourth period shorter than the third period; calculating the second position in a fifth period; 57.
  • the control device according to any one of appendices 51 to 56, wherein the second amount of change is calculated in a sixth period shorter than the fifth period.
  • the control device according to appendix 57, wherein the fourth period and the sixth period are the same period.
  • the output unit outputs the control signal generated by the calculation unit based on the first position, the first amount of change, the second position, and the second amount of change in a second period, 60.
  • the control device according to any one of appendices 57 to 59, wherein the predetermined period is the same as the fourth period or the sixth period.
  • the control device according to any one of appendices 57 to 60, wherein the third period and the fifth period are the same period.
  • the calculation unit is A seventh direction parallel to the seventh axis within a common coordinate system defined by a seventh axis, an eighth axis, and a ninth axis orthogonal to each other based on the first position and the first change amount , an eighth direction parallel to the eighth axis, and a ninth direction parallel to the ninth axis; calculating a third position including at least one of a pose of the object about at least one of nine axes; a position of the object in at least one remaining direction different from the at least one direction and the at least one axis within the common coordinate system based on the second position and the second amount of change; calculating a fourth position comprising at least one of a different pose of said object about at least one remaining axis; 63.
  • a control device according to any one of clauses 51 to 62, wherein said control signal is generated based on said third position and said fourth position.
  • Appendix 64 64. The control device according to appendix 63, wherein the output unit outputs the control signal generated by the calculation unit based on the third position and the fourth position in a seventh cycle.
  • the calculation unit is calculating the first position in a third period; calculating the first amount of change in a fourth period shorter than the third period; calculating the second position in a fifth period; calculating the second amount of change in a sixth cycle shorter than the fifth cycle; The first amount of change is newly calculated in the fourth period after the first position is newly calculated as one period of the third period and before the next calculation of the first position.
  • the control device each time, calculating the third position based on the newly calculated first change amount and the newly calculated first position;
  • the second amount of change is newly calculated in the sixth period after the second position is newly calculated as one period of the fifth period until the second position is next calculated.
  • 65. The control device according to appendix 63 or 64, wherein the fourth position is calculated based on the newly calculated second change amount and the newly calculated second position each time.
  • the seventh axis is the first axis
  • the eighth axis is the second axis
  • the ninth axis is the third axis
  • the calculation unit is At least the position of the object in the at least one direction and the pose of the object about the at least one axis within the common coordinate system by correcting the first position with the first amount of change.
  • the seventh axis is the fourth axis
  • the eighth axis is the fifth axis
  • the ninth axis is the sixth axis
  • a position of the object in the at least one direction and a position of the object about the at least one axis within the first coordinate system calculated by correcting the first position with the first change amount
  • calculating the third position in the common coordinate system based on at least one of the orientation; position of the object in the remaining at least one direction and of the object about the remaining at least one axis within the common coordinate system by correcting the second position with the second change amount
  • the control device according to any one of appendices 63 to 65, wherein at least one of posture and posture is calculated as the fourth position.
  • the calculation unit is At least the position of the object in the at least one direction and the pose of the object about the at least one axis within the common coordinate system by correcting the first position with the first amount of change.
  • the calculation unit is At least one of the position of the object in the at least one direction and the pose of the object about the at least one axis in the common coordinate system calculated based on the first position is defined as the first correcting at least one of the position of the object in the at least one direction and the orientation of the object about the at least one axis within the common coordinate system, which is calculated based on the amount of change. calculating the third position by position of the object in the remaining at least one direction and of the object about the remaining at least one axis within the common coordinate system by correcting the second position with the second change amount; 66.
  • the control device according to any one of appendices 63 to 65, wherein at least one of posture and posture is calculated as the fourth position.
  • the third position includes a position of the object in the eighth direction, a position of the object in the ninth direction, and a pose of the object about the seventh axis;
  • the fourth position includes a position of the object in the seventh direction, a pose of the object about the eighth axis, and a pose of the object about the ninth axis;
  • the seventh axis is the first axis
  • the eighth axis is the second axis
  • the ninth axis is the third axis, 69.
  • the third position includes a position of the object in the eighth direction, a position of the object in the ninth direction, and a pose of the object about the seventh axis;
  • the fourth position includes a position of the object in the seventh direction, a pose of the object about the eighth axis, and a pose of the object about the ninth axis;
  • the seventh axis is the fourth axis,
  • the eighth axis is the fifth axis,
  • the ninth axis is the sixth axis, 69.
  • the control device according to any one of appendices 63 to 65, 67 and 69, wherein the fourth axis is an axis along the optical axis of the optical system of the imaging unit.
  • the first axis is an axis along the optical axis of the optical system of the imaging unit
  • the fourth axis is an axis along the optical axis of the optical system of the imaging unit
  • the first position includes a position of the object in the second direction, a position of the object in the third direction, and a pose of the object about the first axis; 72.
  • the second position includes a position of the object in the fourth direction, a pose of the object about the fifth axis, and a pose of the object about the sixth axis
  • the first axis is an axis along the optical axis of the optical system of the imaging unit
  • the fourth axis is an axis along the optical axis of the optical system of the imaging unit
  • the first position includes a position of the object in the second direction, a position of the object in the third direction, and a pose of the object about the first axis
  • the first amount of change includes an amount of change in the position of the object in the second direction, an amount of change in the position of the object in the third direction, and an amount of change in posture of the object about the first axis.
  • the second position includes a position of the object in the fourth direction, a pose of the object about the fifth axis, and a pose of the object about the sixth axis;
  • the second amount of change is an amount of change in position of the object in the fourth direction, an amount of change in posture of the object about the fifth axis, and an amount of change in the posture of the object about the sixth axis.
  • 72. The control device of any one of clauses 51-71.
  • the calculation unit is calculating the first position by performing a matching process using the first image data and reference image data representing a two-dimensional image serving as a reference of the object; 74.
  • the control device according to any one of appendices 47 to 73, wherein the second position is calculated by performing a matching process using the shape data and reference shape data indicating a three-dimensional shape serving as a reference of the object.
  • the calculation unit is calculating the first position by performing a matching process using the first image data and reference image data representing a two-dimensional image serving as a reference of the object; calculating the second position by performing a matching process using the shape data and reference shape data indicating a three-dimensional shape serving as a reference of the object; a data portion indicating a first characteristic portion of the object in one of the first image data and the first characteristic portion of the object in the other first image data of the two first image data; Calculate the first amount of change by performing tracking processing using the data part shown, Tracking using a data portion indicating the second characteristic portion of the object in one of the shape data and a data portion indicating the second characteristic portion of the object in the other shape data of the two shape data 74.
  • the control device according to any one of appendices 51 to 71 and 73, wherein the second amount of change is calculated by performing processing.
  • the calculation unit is Selecting a data portion indicating at least a partial shape of the object from the shape data based on the first position, and performing matching processing using the selected data portion and the reference shape data.
  • the control device according to appendix 74 or 75, wherein the second position is calculated by: [Appendix 77]
  • the calculation unit is subjecting the shape data to data deletion processing for deleting a data portion indicating a shape of another object different from the object; 77.
  • the control device according to any one of appendices 74 to 76, wherein the second position is calculated by performing a matching process using the shape data subjected to the data deletion process and the reference shape data.
  • Appendix 78 78.
  • Appendix 79 The robot is provided with a processing device for processing the object, 79.
  • the control device of any one of Clauses 47-78, wherein the robot moves the processing device.
  • the control device of Claim 79, wherein the control signal is a signal for controlling the processing device.
  • Appendix 81 81.
  • the control device according to any one of appendices 47 to 80, wherein at least one of the first and imaging units can automatically adjust exposure based on a result of imaging the object by at least one of the first and imaging units.
  • the calculation unit is performing image processing on at least one of the first image data and the second image data; The first position can be calculated based on the first image data subjected to the image processing, 82.
  • the control device according to any one of appendices 47 to 81, wherein the second position can be calculated based on the shape data generated from the second image data on which the image processing has been performed.
  • the calculation unit is performing image processing on at least one of the two first image data and the two second image data; It is possible to calculate the first amount of change based on the two first image data subjected to the image processing, 76. Any one of claims 51 to 71, 73 and 75, wherein the second amount of change can be calculated based on the second shape data generated from the two pieces of second image data subjected to the image processing.
  • a control device according to any one of the preceding paragraphs.
  • Appendix 84 84.
  • the imaging unit generates at least one of the second image data and the two second image data by imaging the object onto which pattern light is projected from a light projection device provided on the robot arm. 85.
  • a control device according to any one of clauses 47-84.
  • [Appendix 86] a control device according to any one of claims 47 to 85;
  • a control system comprising: the imaging unit; [Appendix 87]
  • the imaging unit includes a first imaging device and a second imaging device, a first filter that attenuates a light component in a first wavelength band of light from the object; a second filter that attenuates a light component in a second wavelength band different from the first wavelength band out of the light from the object,
  • the first imaging device captures an image of the object by receiving light from the object through the first filter, 87.
  • the control system of Claim 86 wherein the second imaging device images the object by receiving light from the object through the second filter.
  • Appendix 88 a first illumination device that illuminates the object with first illumination light containing a light component in the first wavelength band; 88.
  • the control system of clause 87 further comprising: a second illumination device that illuminates the object with a second illumination light that includes a light component in the second wavelength band.
  • Appendix 89 89.
  • the present invention is not limited to the above-described embodiments, and can be modified as appropriate within the scope not contrary to the gist or idea of the invention that can be read from the scope of claims and the entire specification.
  • a control system, a robot system, a control method and a computer program are also included in the technical scope of the present invention.

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Manipulator (AREA)
PCT/JP2021/048572 2021-12-27 2021-12-27 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム Ceased WO2023127021A1 (ja)

Priority Applications (6)

Application Number Priority Date Filing Date Title
KR1020247018276A KR20240130076A (ko) 2021-12-27 2021-12-27 제어 장치, 제어 시스템, 로봇 시스템, 제어 방법 및 컴퓨터 프로그램
PCT/JP2021/048572 WO2023127021A1 (ja) 2021-12-27 2021-12-27 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム
US18/724,200 US20250214249A1 (en) 2021-12-27 2021-12-27 Control apparatus, control system, robot system, control method, and computer program
EP21969905.5A EP4458526A4 (en) 2021-12-27 2021-12-27 CONTROL DEVICE, CONTROL SYSTEM, ROBOT SYSTEM, CONTROL METHOD AND COMPUTER PROGRAM
JP2023570515A JPWO2023127021A1 (https=) 2021-12-27 2021-12-27
CN202180104846.8A CN118369192A (zh) 2021-12-27 2021-12-27 控制装置、控制系统、机器人系统、控制方法及计算机程序

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/048572 WO2023127021A1 (ja) 2021-12-27 2021-12-27 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム

Publications (1)

Publication Number Publication Date
WO2023127021A1 true WO2023127021A1 (ja) 2023-07-06

Family

ID=86998336

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/048572 Ceased WO2023127021A1 (ja) 2021-12-27 2021-12-27 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム

Country Status (6)

Country Link
US (1) US20250214249A1 (https=)
EP (1) EP4458526A4 (https=)
JP (1) JPWO2023127021A1 (https=)
KR (1) KR20240130076A (https=)
CN (1) CN118369192A (https=)
WO (1) WO2023127021A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025069298A1 (ja) * 2023-09-28 2025-04-03 株式会社ニコン 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4516465A4 (en) * 2022-04-28 2026-03-11 Nikon Corp CONTROL DEVICE, CONTROL SYSTEM, ROBOTIC SYSTEM, CONTROL METHOD, AND COMPUTER PROGRAM
CN119403659A (zh) * 2022-06-21 2025-02-07 发那科株式会社 取得工件的位置数据的装置、控制装置、机器人系统、方法以及计算机程序
US12583111B2 (en) * 2023-09-06 2026-03-24 Mitsubishi Electric Research Laboratories, Inc. Eye-on-hand reinforcement learner for dynamic grasping with active pose estimation

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000288974A (ja) * 1999-04-08 2000-10-17 Fanuc Ltd 画像処理機能を持つロボット装置
US20130230235A1 (en) 2010-11-19 2013-09-05 Canon Kabushiki Kaisha Information processing apparatus and information processing method
JP2019119027A (ja) * 2018-01-10 2019-07-22 キヤノン株式会社 ロボットシステムの制御方法、およびロボットシステム
WO2021010181A1 (ja) * 2019-07-17 2021-01-21 日立金属株式会社 検査装置、検査方法、位置決め方法、およびプログラム

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20140009774A (ko) * 2012-07-13 2014-01-23 삼성전자주식회사 3ｄ 이미지 센서 및 이를 포함하는 시스템
JP2016065718A (ja) * 2013-02-08 2016-04-28 三洋電機株式会社 情報取得装置および物体検出装置
US9694498B2 (en) * 2015-03-30 2017-07-04 X Development Llc Imager for detecting visual light and projected patterns
JP2017020873A (ja) * 2015-07-09 2017-01-26 キヤノン株式会社 被計測物の形状を計測する計測装置
US12175741B2 (en) * 2021-06-22 2024-12-24 Intrinsic Innovation Llc Systems and methods for a vision guided end effector

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000288974A (ja) * 1999-04-08 2000-10-17 Fanuc Ltd 画像処理機能を持つロボット装置
US20130230235A1 (en) 2010-11-19 2013-09-05 Canon Kabushiki Kaisha Information processing apparatus and information processing method
JP2019119027A (ja) * 2018-01-10 2019-07-22 キヤノン株式会社 ロボットシステムの制御方法、およびロボットシステム
WO2021010181A1 (ja) * 2019-07-17 2021-01-21 日立金属株式会社 検査装置、検査方法、位置決め方法、およびプログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4458526A4

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025069298A1 (ja) * 2023-09-28 2025-04-03 株式会社ニコン 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム

Also Published As

Publication number Publication date
CN118369192A (zh) 2024-07-19
JPWO2023127021A1 (https=) 2023-07-06
EP4458526A1 (en) 2024-11-06
EP4458526A4 (en) 2025-11-12
US20250214249A1 (en) 2025-07-03
KR20240130076A (ko) 2024-08-28

Similar Documents

Publication Publication Date Title
WO2023127021A1 (ja) 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム
US20180338090A1 (en) Image processing system, image processing device, and image processing program
JP5949242B2 (ja) ロボットシステム、ロボット、ロボット制御装置、ロボット制御方法、およびロボット制御プログラム
CN111604942B (zh) 物体检测装置、控制装置以及物体检测用计算机程序
US10059002B2 (en) Image processing apparatus, image processing method, and non-transitory computer-readable medium
WO2024027647A1 (zh) 机器人控制方法、系统和计算机程序产品
KR20220098699A (ko) 로봇 모션 용 비전 시스템의 자동 핸드-아이 캘리브레이션을 위한 시스템 및 방법
JP6836561B2 (ja) 画像処理装置及び画像処理方法
CN110136208A (zh) 一种机器人视觉伺服系统的联合自动标定方法及装置
JP7323057B2 (ja) 制御装置、制御方法、および、制御プログラム
US20240157568A1 (en) System for welding at least a portion of a piece and related methods
JP2021026599A (ja) 画像処理システム
NL2035240B1 (en) Method and systems for compensating for a deviation in a plurality of workingposes of a manipulator of an autonomous charging device
CN112743537B (zh) 注释装置
JPWO2023127021A5 (https=)
JP7031204B2 (ja) 制御装置、制御装置の制御方法、情報処理プログラム、および記録媒体
CN119784839A (zh) 机械臂位姿确定方法、设备、存储介质及程序产品
WO2023209974A1 (ja) 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム
US20240399588A1 (en) Position and posture measurement system
JP2019060695A (ja) ３次元物体検出装置、ロボット、及びプログラム
JP7401250B2 (ja) 画像処理装置、制御方法およびプログラム
US20260102920A1 (en) Arithmetic apparatus, arithmetic system, robot system, arithmetic method, and computer program
JP2017110975A (ja) 計測装置、システム、計測方法、決定方法及びプログラム
WO2026069666A1 (ja) 制御装置、制御システム、ロボットシステム、制御方法及びコンピュータプログラム
CN120439268A (zh) 用于机械手臂的抓取方法、电子装置及系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21969905

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202180104846.8

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 2023570515

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2021969905

Country of ref document: EP

Effective date: 20240729

WWP Wipo information: published in national office

Ref document number: 18724200

Country of ref document: US