WO2023163219A1 - 情報処理装置、ロボット制御システム及びプログラム - Google Patents

情報処理装置、ロボット制御システム及びプログラム Download PDF

Info

Publication number
WO2023163219A1
WO2023163219A1 PCT/JP2023/007351 JP2023007351W WO2023163219A1 WO 2023163219 A1 WO2023163219 A1 WO 2023163219A1 JP 2023007351 W JP2023007351 W JP 2023007351W WO 2023163219 A1 WO2023163219 A1 WO 2023163219A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
pixel
window
size
information processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2023/007351
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
卓哉 宮本
祥 文野
博昭 宮村
敬之 石田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Kyocera Corp
Original Assignee
Kyocera Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Kyocera Corp filed Critical Kyocera Corp
Priority to CN202380021452.5A priority Critical patent/CN118696207A/zh
Priority to US18/841,599 priority patent/US20250037289A1/en
Priority to JP2024503317A priority patent/JPWO2023163219A1/ja
Priority to EP23760217.2A priority patent/EP4488616A1/en
Publication of WO2023163219A1 publication Critical patent/WO2023163219A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1694—Program controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1612—Program controls characterised by the hand, wrist, grip control
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
    • G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
    • G01B11/2545—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object with one projection direction and several detection directions, e.g. stereo
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional [3D] objects
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/37—Measurements
    • G05B2219/37366—Colour, surface colour
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/37—Measurements
    • G05B2219/37425—Distance, range
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/37—Measurements
    • G05B2219/37567—3-D vision, stereo vision, with two cameras
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/39—Robotics, robotics to robotics hand
    • G05B2219/39543—Recognize object and plan hand shapes in grasping movements
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40053—Pick 3-D object from pile of objects
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40607—Fixed camera to observe workspace, object, workpiece, global
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds

Definitions

  • This disclosure relates to information processing technology.
  • Patent Document 1 describes a technique for identifying an object at a high position from among a pile of objects.
  • an information processing device includes an acquisition unit and a generation unit.
  • the acquisition unit acquires depth information representing the depth within a measurement space in which a plurality of objects exist.
  • the generation unit performs filtering on the depth information according to the object size of the object, and generates high position information representing a high position part of the plurality of objects that is higher than the others in the measurement space. .
  • the robot control system includes an acquisition unit and a generation unit included in the information processing apparatus, a specification unit, a degree acquisition unit, a holding object determination unit, a holding posture determination unit, a robot and a control unit.
  • the specifying unit determines whether the target object is in the photographed image obtained by photographing the measurement space for each of the plurality of target objects.
  • a captured object image is specified based on high position information.
  • the degree obtaining unit obtains, for each of the plurality of target objects, a degree of ease of holding the target object based on the depth information.
  • the holding target determination unit determines a holding target object to be held by the robot from among the plurality of target objects based on the degrees of the plurality of target objects.
  • the holding posture determination unit determines a holding posture of the robot with respect to the object to be held based on the object image.
  • the robot control section controls the robot based on the holding posture determined by the holding posture determination section.
  • the program causes the computer device to acquire depth information representing the depth in a measurement space in which a plurality of objects exist.
  • the program causes the computer device to perform filtering processing according to the object size of the object on the depth information, and to express a high position part of the plurality of objects that is higher than the others in the measurement space. Let it run to generate location information.
  • FIG. 1 is a schematic diagram showing an example of a robot system;
  • FIG. 1 is a schematic diagram showing an example of an information processing device;
  • FIG. It is a schematic diagram showing an example of a control part of an information processor.
  • 4 is a flow chart showing an example of the operation of the information processing device;
  • FIG. 4 is a schematic diagram showing an example of a filter window;
  • FIG. 4 is a schematic diagram showing an example of how a filter window is set on a depth image;
  • FIG. 4 is a schematic diagram for explaining an example of filtering;
  • FIG. 4 is a schematic diagram for explaining an example of filtering;
  • FIG. 4 is a schematic diagram for explaining an example of filtering;
  • FIG. 4 is a schematic diagram for explaining an example of filtering;
  • FIG. 4 is a schematic diagram for explaining an example of filtering;
  • FIG. 4 is a schematic diagram for explaining an example of filtering;
  • FIG. 4 is a schematic diagram for explaining an example of filtering;
  • FIG. 4 is a schematic diagram for explaining an example of filtering;
  • FIG. 4 is a schematic diagram for explaining an example of filtering;
  • FIG. 4 is a schematic diagram for explaining an example of a method of specifying an object image;
  • FIG. 10 is a schematic diagram for explaining an example of a method for determining the degree of ease of holding;
  • FIG. 4 is a schematic diagram showing an example of a filter window;
  • FIG. 4 is a schematic diagram showing an example of a filter window;
  • FIG. 4 is a schematic diagram showing an example of a filter window;
  • FIG. 4 is a schematic diagram showing an example of a filter window;
  • FIG. 10 is a schematic diagram for explaining an example of the operation of the specifying unit;
  • FIG. 1 is a schematic diagram showing an example of the robot system 1.
  • the robot system 1 includes, for example, a robot 2 and a robot control system 3 that controls the robot 2 .
  • the robot control system 3 includes, for example, a sensor device 4 that detects the state in a measurement space 100 that is a space to be measured, a robot control section 5 that controls the robot 2, and an information processing device 6.
  • the information processing device 6 generates information necessary for robot control by the robot control unit 5 based on the detection result of the sensor device 4 .
  • the robot 2 can move the object 10 from the work start table 11 to the work target table 12, for example.
  • Object 10 is also called a work object or workpiece.
  • the robot control unit 5 controls the robot 2 so that the object 10 moves from the work start table 11 to the work target table 12 .
  • the robot 2 includes, for example, an arm 20 and an end effector 21 that holds the object 10 .
  • the robot 2 can move the object 10 from the work start table 11 to the work target table 12 by moving the arm 20 while holding the object 10 with the end effector 21 .
  • the robot controller 5 can control the arm 20 and the end effector 21 .
  • the end effector 21 may hold the object 10 by gripping the object 10, may hold the object 10 by sucking the object 10, or may hold the object 10 by other methods. When gripping the object 10, the end effector 21 may grip the object 10 with two fingers, or may grip the object 10 with two or more fingers.
  • Trays 13 and 14 are placed on the work start table 11 and the work target table 12, respectively.
  • a plurality of objects 10 are randomly stacked on the tray 13 .
  • the robot 2 holds the objects 10 on the tray 13 one by one, moves them to the work target table 12 , and arranges them in the tray 14 .
  • the plurality of objects 10 on the tray 13 may be collectively referred to as an object group 10A.
  • the sensor device 4 is capable of detecting depth and color within the measurement space 100 .
  • the sensor device 4 can acquire depth information 700 (see FIG. 3 described later) representing the depth within the measurement space 100 and color information 710 (see FIG. 3 described later) within the measurement space 100.
  • the sensor device 4 includes, for example, a depth sensor 40 that acquires depth information 700 and a camera 41 that photographs the measurement space 100 and acquires color information 710 within the measurement space 100 .
  • the depth sensor 40 generates depth information 700 using, for example, a stereo method.
  • the depth sensor 40 includes, for example, a stereo camera composed of a plurality of cameras 40a each capturing an image of the measurement space 100.
  • FIG. Each camera 40 a photographs the object group 10 A on the tray 13 from above the tray 13 .
  • the depth sensor 40 generates depth information 700 based on image data obtained by the multiple cameras 40a.
  • Depth sensor 40 generates depth information 700, for example, based on the principle of triangulation.
  • the depth sensor 40 generates depth image data representing a depth image 701 (see FIG. 3 described later) as the depth information 700 .
  • a depth image 701 is an image representing depths to a plurality of measurement points in the measurement space 100 .
  • the measurement points are located on the surface of the object within the measurement space 100 .
  • the plurality of measurement points include measurement points located on the surface of the tray 13 which is an object in the measurement space 100 and measurement points located on the surface of the object group 10A on the tray 13 .
  • the depth image 701 can also be said to be an image representing the depth to the object in the measurement space 100 .
  • the depth represented by the depth information 700 is the depth of the measurement point when viewed from the depth sensor 40 (specifically, the stereo camera).
  • the depth information 700 may be called depth image data 700 .
  • the depth image data 700 includes pixel values of a plurality of pixels forming the depth image 701 .
  • a plurality of pixels forming the depth image 701 respectively correspond to a plurality of measurement points within the measurement space 100 .
  • a pixel value of a certain pixel in the depth image 701 corresponds to a measurement point corresponding to the certain pixel.
  • a pixel value included in the depth image data 700 represents the depth to the corresponding measurement point, in other words, the depth to the object 10 . It can be said that the pixel values included in the depth image data 700 are depth information representing the depth up to one measurement point.
  • a pixel value included in the depth image data 700 is, for example, a value of zero or more.
  • pixel position information representing the position of the pixel and the pixel value of the pixel are associated with each other.
  • the depth sensor 40 may generate the depth information 700 using a method other than the stereo method.
  • a projector method may be used, a stereo method and a projector method may be used together, or a ToF (Time of Flight) method may be used.
  • a ToF (Time of Flight) method may be used.
  • the camera 41 is a color camera, and generates color image data representing a color image 711 (see FIG. 3 described later) showing the measurement space 100 as the color information 710 in the measurement space 100 .
  • the color information 710 may be called color image data 710 .
  • a color image 711 is a photographed image obtained by photographing the measurement space 100 .
  • the color image 711 shows the tray 13 and the object group 10A thereon.
  • pixel position information representing the position of the pixel and the pixel value of the pixel are associated with each other.
  • a plurality of pixels forming the color image 711 correspond to a plurality of measurement points in the measurement space 100, respectively.
  • a pixel value of a certain pixel in the color image 711 corresponds to a measurement point corresponding to the certain pixel.
  • a pixel value included in the color image data 710 can be said to be color information representing the color of one corresponding measurement point.
  • Pixel values included in the color image data 710 include, for example, an R component (red component), a G component (green component), and a B component (blue component).
  • Color image data 710 is also called RGB image data.
  • the pixel position of a pixel included in an image is represented by (m, n).
  • m indicates the position in the row direction (in other words, the horizontal direction)
  • n indicates the position in the column direction (in other words, the vertical direction).
  • an image is composed of a plurality of pixels of N rows and M columns, the leftmost column of the image is the 0th column, the rightmost column of the image is the (M-1)th column, and the first column of the image is Let the top row be the 0th row and the bottom row of the image be the (N-1)th row.
  • m indicates a column number and is an integer that satisfies 0 ⁇ m ⁇ M ⁇ 1.
  • n indicates a row number and is an integer that satisfies 0 ⁇ n ⁇ N-1.
  • the number of pixels in the color image 711 and the number of pixels in the depth image 701 match each other. That is, the number of measurement points corresponding to the pixels forming the color image 711 matches the number of measurement points corresponding to the pixels forming the depth image 701 .
  • the pixel at the pixel position (m, n) of the color image 711 and the pixel at the pixel position (m, n) of the depth image 701 correspond to the same measurement point.
  • the pixel value of the pixel at the pixel position (m, n) included in the color image data 710 and the pixel value of the pixel at the pixel position (m, n) included in the depth image data 700 correspond to the same measurement point.
  • there is A pixel value representing the color of a measurement point included in the color image data 710 and a pixel value representing the depth up to the measurement point included in the depth image data 700 correspond to pixels at the same pixel position.
  • FIG. 2 is a schematic diagram showing an example of the configuration of the information processing device 6.
  • the information processing device 6 includes, for example, a control unit 60, a first interface 61, a second interface 62, and a storage unit 63.
  • the information processing device 6 can also be called an information processing circuit, for example.
  • the information processing device 6 is, for example, a kind of computer device.
  • the first interface 61 is capable of communicating with the sensor device 4.
  • the first interface 61 can also be called an interface circuit, for example.
  • the first interface 61 can be said to be, for example, a communication unit or a communication circuit.
  • the first interface 61 may perform wired communication with the sensor device 4, or may perform wireless communication.
  • the first interface 61 may communicate with the sensor device 4 according to at least one communication standard.
  • At least one communication standard to which the first interface 61 conforms includes, for example, USB (Universal Serial Bus), I2C (Inter-Integrated Circuit), CSI (Clocked Serial Interface), SPI (Serial Peripheral Interface), WiFi and Ethernet.
  • USB Universal Serial Bus
  • I2C Inter-Integrated Circuit
  • CSI Clocked Serial Interface
  • SPI Serial Peripheral Interface
  • WiFi and Ethernet may include at least one of Also, the first interface 61 may communicate with the sensor device 4 through the Internet.
  • the second interface 62 is capable of communicating with a device external to the robot system 1 (simply referred to as an external device).
  • a second interface 62 is capable of communicating with at least one external device.
  • the second interface 62 can also be called an interface circuit, for example. Also, the second interface 62 can be said to be, for example, a communication section or a communication circuit.
  • the second interface 62 may perform wired communication with an external device, or may perform wireless communication.
  • the second interface 62 may communicate with an external device according to at least one communication standard.
  • the at least one communication standard that the second interface 62 complies with may include, for example, at least one of USB, I2C, CSI, SPI, WiFi, and Ethernet.
  • the second interface 62 may also communicate with external devices through the Internet.
  • At least one external device with which the second interface 62 can communicate may include a display device or a computer device.
  • the computer device may be a server such as a cloud server, or may be a mobile device.
  • the mobile device may be a mobile phone such as a smart phone, a tablet terminal, a personal computer, or a wearable device.
  • the wearable device may be a type worn on the arm such as a wristband type or a wristwatch type, a type worn on the head such as a headband type or glasses type, or a type worn on the body such as a clothing type. It may be of a wearable type.
  • the control unit 60 can comprehensively manage the operation of the information processing device 6 by controlling other components of the information processing device 6 .
  • the control unit 60 can also be said to be, for example, a control circuit.
  • Control unit 60 includes at least one processor to provide control and processing power to perform various functions, as described in further detail below.
  • the at least one processor may be implemented as a single integrated circuit (IC) or as multiple communicatively coupled integrated circuit ICs and/or discrete circuits. good.
  • the at least one processor can be implemented according to various known techniques.
  • a processor includes one or more circuits or units configured to perform one or more data computing procedures or processes, such as by executing instructions stored in associated memory.
  • the processor may be firmware (eg, discrete logic components) configured to perform one or more data computing procedures or processes.
  • the processor is one or more processors, controllers, microprocessors, microcontrollers, application specific integrated circuits (ASICs), digital signal processors, programmable logic devices, field programmable gate arrays, or any of these. Any combination of devices or configurations, or other known combinations of devices and configurations, may be included to perform the functions described below.
  • ASICs application specific integrated circuits
  • the control unit 60 may include, for example, a CPU (Central Processing Unit) as a processor.
  • the storage unit 63 may include non-temporary recording media readable by the CPU of the control unit 60, such as ROM (Read Only Memory) and RAM (Random Access Memory).
  • the storage unit 63 stores, for example, a program 630 for controlling the information processing device 6 .
  • Various functions of the control unit 60 are realized by executing the program 630 in the storage unit 63 by the CPU of the control unit 60, for example.
  • control unit 60 may include multiple CPUs.
  • the control unit 60 may also include at least one DSP (Digital Signal Processor). Further, all functions of the control unit 60 or some functions of the control unit 60 may be realized by a hardware circuit that does not require software to realize the functions.
  • the storage unit 63 may also include a non-temporary computer-readable recording medium other than the ROM and RAM.
  • the storage unit 63 may include, for example, a small hard disk drive and an SSD (Solid State Drive).
  • the robot control unit 5 is, for example, a type of computer device, and has the same configuration as the information processing device 6, for example.
  • the robot control unit 5 can communicate with, for example, the second interface 62 of the information processing device 6 .
  • the control unit 60 of the information processing device 6 generates information required for robot control by the robot control unit 5 .
  • Information generated by the control unit 60 is transmitted from the second interface 62 to the robot control unit 5 .
  • the robot control section 5 controls the robot 2 based on information transmitted from the second interface 62 .
  • the robot control unit 5 may be called, for example, a robot control circuit.
  • FIG. 3 is a schematic diagram showing an example of a plurality of functional blocks formed in the control unit 60 when the control unit 60 of the information processing device 6 executes the program 630.
  • the control unit 60 has, for example, an information acquisition unit 600, a conversion unit 601, a generation unit 602, a specification unit 603, a degree acquisition unit 604, a holding target determination unit 605, and a holding posture determination unit 606 as functional blocks. All the functions of the information acquisition unit 600 or part of the functions of the information acquisition unit 600 may be realized by hardware circuits that do not require software to realize the functions. This also applies to the generation unit 602 , the identification unit 603 , the degree acquisition unit 604 , the holding target determination unit 605 and the holding posture determination unit 606 .
  • the information acquisition unit 600 performs acquisition processing for acquiring depth information 700 and color information 710 from the sensor device 4 through the first interface 31 .
  • the information acquisition unit 600 transmits acquisition instruction information for instructing acquisition of the depth information 700 and the color information 710 to the sensor device 4 via the first interface 31, for example.
  • the sensor device 4 controls the depth sensor 40 and the camera 41 to cause the stereo camera of the depth sensor 40 and the camera 41 to perform photography, respectively.
  • the depth sensor 40 generates depth information 700 (in other words, depth image data 700) based on the image data obtained by the stereo camera, and the camera 41 produces color information 710 (in other words, color image data 710). Generate.
  • the sensor device 4 then transmits the depth information 700 and the color information 710 to the information processing device 6 .
  • the depth information 700 and the color information 710 received by the first interface 61 from the sensor device 4 are input to the control section 60 . Accordingly, in the acquisition process, the information acquisition section 600 can acquire the depth information 700 and the color information 710 in response to the transmission of the acquisition instruction information.
  • the depth information 700 and the color information 710 are stored in the storage section 63 through the control section 60 .
  • the conversion unit 601 converts the depth image data 700 acquired by the information acquisition unit 600 into depth image data 705 in which the smaller the depth to the measurement point, the larger the corresponding pixel value.
  • the conversion unit 601 generates depth image data 705 by inverting each pixel value of the depth image data 700 .
  • the maximum value of the range of possible pixel values is called the pixel maximum value.
  • Depth image data 705 is obtained by inverting each pixel value in the depth image data 700 .
  • the depth image data 705 may be called depth information 705 .
  • the generating unit 602 performs filtering according to the size of the object 10 (simply referred to as the object size) on the depth information 705 obtained by the transforming unit 601, so that the plurality of objects 10 on the tray 13 have more depth information than others.
  • a high position portion is, for example, a portion of the object group 10A that is higher than its surroundings. For each object 10, if the object 10 has a portion that is higher than its surroundings, then that portion is included in the high portion.
  • a portion of the object group 10A that is higher than the surroundings can be said to be a portion of the object group 10A that is positioned closer to the depth sensor 40 than the surroundings.
  • a position higher than the surroundings in the object group 10A does not necessarily mean the highest point in the object group 10A.
  • the object 10 including the portion included in the high position portion is called the target object 10.
  • a target object 10 is an object 10 that has a portion that is located higher than the others.
  • the high position portion represented by the high position information 720 includes a portion of at least one target object 10 that is at a higher position than the others. That is, the high portion includes a portion of at least one target object 10 that is higher than its surroundings.
  • a portion of one target object 10 at a position higher than the others may be referred to as an object high position portion.
  • the high position portion indicated by the high position information 720 may be called an overall high position portion.
  • the overall height position portion includes at least one object height position portion of the target object 10 .
  • the object height position portion of one target object 10 constitutes a part of the overall height position portion. Also called
  • the robot system 1 will be described on the assumption that the object high position portions of the plurality of target objects 10 are included in the overall high position portion. In other words, the robot system 1 will be described assuming that a plurality of target objects 10 are present on the tray 13 .
  • the specifying unit 603 specifies an object image in which the target object 10 is captured in the captured image of the measurement space 100 based on the high position information 720.
  • the identifying unit 603 identifies an object image in which the target object 10 is shown in the color image 711 represented by the color image data 710 acquired by the information acquiring unit 600 based on the high position information 720 .
  • the object image specified by the specifying unit 603 may be called a specified object image.
  • the degree acquisition unit 604 obtains the degree of ease of holding the target object 10 for each of the plurality of target objects 10 based on the depth information 705 and the specific object image in which the target object 10 is captured.
  • the degree of ease of holding the target object 10 may be referred to as the degree of ease of holding.
  • the degree of ease of holding means the degree of ease with which the robot 2 can hold the target object 10 .
  • the holding target determination unit 605 determines the holding target object 10 to be held by the robot 2 from among the plurality of target objects 10 based on the ease of holding the plurality of target objects 10 .
  • the holding posture determination unit 606 determines the holding posture of the robot 2 with respect to the holding target object 10 based on the specific object image in which the holding target object 10 is captured. That is, the holding posture determination unit 606 determines in what posture the robot 2 holds the hold target object 10 based on the specific object image in which the hold target object 10 is captured. For example, based on the specific object image, the holding posture determination unit 606 specifies the position and posture within the measurement space 100 of the holding target object 10 reflected therein.
  • the holding posture determination unit 606 determines the holding posture of the robot 2 with respect to the hold target object 10 based on the specified position and posture of the hold target object 10 . For example, when the end effector 21 of the robot 2 grips the object 10 with two fingers, the holding posture determination unit 606 determines from which direction the end effector 21 grips the object 10 with two fingers as the holding posture. decide. The control unit 60 notifies the robot control unit 5 of the holding posture determined by the holding posture determination unit 606 through the second interface 62 .
  • the robot control unit 5 controls the robot 2 based on the notified holding posture. Specifically, the robot control unit 5 controls the arm 20 and the end effector 21 of the robot 2 so that the robot 2 holds the holding target object 10 in the notified holding posture. As a result, the robot 2 can hold an object 10 that has a portion located higher than the others, among the plurality of objects 10 on the tray 13 , and that the robot 2 can easily hold. After that, the robot control unit 5 controls the robot 2 so that the holding target object 10 held by the robot 2 is arranged on the tray 14 .
  • FIG. 4 is a flowchart showing an example of a holding posture determination process for determining the holding posture of the robot 2 with respect to the object 10 based on the depth information 700 and the color information 710.
  • FIG. The control unit 60 repeatedly executes the holding posture determination process.
  • the information acquisition unit 600 first acquires depth information 700 and color information 710 from the sensor device 4 in step s1.
  • the conversion unit 601 performs conversion processing for converting the depth information 700 into the depth information 705.
  • the generation unit 602 performs filter processing on the depth information 705 according to the object size to generate high position information 720 .
  • the specifying unit 603 specifies, for each of the plurality of target objects 10, an object image in which the target object 10 is shown in the color image 711 represented by the color information 710, based on the high position information 720.
  • step s5 the degree obtaining unit 604 obtains the degrees of ease of holding each of the plurality of target objects 10 based on the depth information 705 and a plurality of specific object images in which the plurality of target objects 10 are respectively captured. demand.
  • step s ⁇ b>6 the hold target determination unit 605 determines the hold target object 10 from the plurality of target objects 10 based on the ease of holding of the plurality of target objects 10 .
  • step s7 the holding posture determination unit 606 determines the holding posture of the robot 2 with respect to the holding target object 10 based on the specific object image in which the holding target object 10 is captured.
  • step s7 when the robot 2 holds the object 10 to be held on the tray 13 and carries it to the tray 14 as described above, the holding posture determination processing shown in FIG. A holding posture for the new holding target object 10 is determined. Thereafter, every time the object 10 to be held on the tray 13 is carried to the tray 14, the holding posture determination process is executed.
  • step s7 the holding posture determination unit 606 may determine the holding posture of the robot 2 with respect to the object 10 appearing in the object image, based on the object image specified in step s4.
  • the information processing device 6 may be realized by a cloud server. In this case, the information processing device 6 may communicate with the robot control unit 5 through the Internet. Also, the control unit 60 of the information processing device 6 may function as the robot control unit 5 . In this case, the information processing device 6 functions as the robot control system 3 . Also, the robot control unit 5 may be realized by a cloud server. In this case, the robot 2 may be provided with a communication unit connected to the Internet, and the robot control unit 5 may control the robot 2 through the Internet. Also, the robot control system 3 may not include the sensor device 4 . Also, the robot control system 3 may include a plurality of robot control units 5 . At least part of the holding posture determination process shown in FIG. 4 may be executed by the robot control unit 5 . For example, the process of step s7 may be executed by the robot control section 5. FIG.
  • the sensor device 4 may generate the depth information 705 and transmit it to the information processing device 6 .
  • the conversion unit 601 becomes unnecessary, and the depth information 705 acquired by the information acquisition unit 600 is used in the generation unit 602 and the degree acquisition unit 604 .
  • the depth information 705 representing the depth in the measurement space 100 in which the plurality of objects 10 are present is subjected to filtering according to the object size, and the object group 10A has other objects.
  • Elevation information 720 is generated that represents the elevated portion located at a higher position than . In this way, by using the object size to generate the high position information 720, it is possible to appropriately identify the high position portion located higher than the others in the object group 10A based on the high position information 720.
  • the generation unit 602 performs filtering on a depth image 706 represented by the depth information 705 (in other words, depth image data 705) using at least one type of spatial filter function according to the object size, and generates high position information. 720 is generated.
  • the high position information 720 is, for example, image data representing the filtered depth image 706 .
  • the image represented by the high position information 720 (that is, the filtered depth image 706) may be referred to as a high position image.
  • the high position information 720 may be referred to as high position image data 720 .
  • a high-position image represents a portion of the object group 10A that is positioned higher than others.
  • the high position information 720 includes pixel values of a plurality of pixels forming the high position image.
  • pixel position information representing the position of the pixel and the pixel value of the pixel are associated with each other.
  • a plurality of pixels forming the high-position image correspond to a plurality of measurement points in the measurement space 100, respectively.
  • a pixel value of a certain pixel in the high-position image corresponds to a measurement point corresponding to the certain pixel.
  • a pixel value included in the high position information 720 indicates the degree to which the corresponding measurement point is at a higher position than the others. For example, the pixel value included in the high position information 720 indicates the degree to which the corresponding measurement point is positioned higher than its surroundings. The higher the pixel value included in the high position information 720 is, the higher the corresponding measurement point is.
  • the number of pixels in the high-position image matches, for example, the number of pixels in the color image 711 and the number of pixels in the depth image 706 .
  • the pixel at the pixel position (m, n) of the high-position image and the pixel at the pixel position (m, n) of the color image 711 correspond to the same measurement point.
  • the pixel value of the pixel at the pixel position (m, n) included in the high-position image data 720 and the pixel value of the pixel at the pixel position (m, n) included in the color image data 710 are obtained at the same measurement point. Yes.
  • the pixel at the pixel position (m, n) in the high-position image and the pixel at the pixel position (m, n) in the depth image 706 correspond to the same measurement point.
  • the pixel value of the pixel at the pixel position (m, n) included in the high-position image data 720 and the pixel value of the pixel at the pixel position (m, n) included in the depth image data 705 are obtained at the same measurement point. Yes.
  • pixels at the same pixel positions in a plurality of images are expressed as corresponding to each other.
  • the pixel at pixel position (m, n) in the high-position image, the pixel at pixel position (m, n) in depth image 706, and the pixel at pixel position (m, n) in color image 711 correspond to each other.
  • pixel values of pixels at the same pixel positions in a plurality of images are expressed as mutually corresponding.
  • the pixel value of the pixel at the pixel position (m, n) of the high position image the pixel value of the pixel at the pixel position (m, n) of the depth image 706, and the pixel position (m, n) of the color image 711
  • the pixel values of the pixels correspond to each other.
  • the included pixel values representing the color of the certain measurement point correspond to each other.
  • first partial image (or first region) that is part of an image corresponds to a second partial image (or second region) that is a part of another image
  • first partial image or Alternatively, it means that the pixel positions of the plurality of pixels forming the first region match the pixel positions of the plurality of pixels forming the second partial image (or the second region).
  • first partial image that is part of the high-position image corresponds to a second partial image that is part of the depth image
  • the pixel positions of the plurality of pixels forming the first partial image correspond to the They match the pixel positions of the plurality of pixels forming the second partial image.
  • the pixel of the depth image 706 may be called the first pixel, and the pixel value of the first pixel may be called the first pixel value.
  • the pixel of the high-position image may be called the second pixel, and the pixel value of the second pixel may be called the second pixel value.
  • a pixel of the color image 711 may be called a third pixel, and a pixel value of the third pixel may be called a third pixel value.
  • FIG. 5 is a schematic diagram showing an example of the filter window 200 of the first spatial filter function used in the generation process.
  • FIG. 6 is a schematic diagram showing an example of how the filter window 200 is set for the depth image 706 represented by the depth image data 705. As shown in FIG.
  • the generation unit 602 determines the first pixel of interest from among the plurality of first pixels forming the depth image 706 as the first pixel of interest. Then, the generation unit 602 sets the filter window 200 to the depth image 706 so as to include the first pixel of interest (see FIG. 6). The generation unit 602 performs a predetermined operation on the pixel values of the plurality of first pixels in the filter window 200 to obtain the pixel value of the first pixel of interest (also referred to as the first pixel value of interest) in the high-position image. A unit filtering process is performed to obtain a corresponding second pixel value.
  • the generation unit 602 moves the position of the first pixel of interest in the depth image 706 by one pixel, and performs unit filtering each time the position of the first pixel of interest is moved by one pixel. That is, the generation unit 602 moves the filter window 200 on the depth image 706 in units of one pixel, and performs unit filtering each time the filter window 200 is moved by one pixel.
  • a plurality of second pixel values in high position information 720 are determined that respectively correspond to the plurality of first pixel values.
  • the filter window 200 has, for example, a plus partial window 201 and two minus partial windows 202, as shown in FIG.
  • a plus partial window 201 (also referred to as plus window 201) is a partial filter window in which the pixel value of each pixel in the region is multiplied by a predetermined plus value.
  • a minus partial window 202 (also referred to as minus window 202) is a partial filter window in which the pixel value of each pixel in the region is multiplied by a predetermined minus value.
  • the outer shape of the plus partial window 201 is, for example, a rectangle.
  • the outline of the minus partial window 202 is, for example, a square.
  • the plus partial window 201 and the two minus partial windows 202 are arranged along one direction, for example.
  • a plus partial window 201 is sandwiched between two minus partial windows 202 .
  • the predetermined positive value is for example +1.0 and the predetermined negative value is for example -0.5.
  • the window arrangement direction A direction perpendicular to the window arrangement direction is called a window vertical direction. The same is true for other types of filter windows to be described later.
  • the window alignment direction and window vertical direction of the filter window 200 match the row direction and column direction of the depth image 706 in which the filter window 200 is set, respectively.
  • Each of the window alignment direction size and the window vertical direction size of the filter window 200 is the size of a plurality of pixels.
  • the size of the filter windows 200 in the window arrangement direction and the size in the window vertical direction are the same, and the outer shape of the filter windows 200 is square.
  • the size of the filter window 200 in the window arrangement direction and the size in the window vertical direction may differ from each other. That is, the size of the filter window 200 in the window alignment direction may be larger or smaller than the size of the filter window 200 in the window vertical direction.
  • Each of the window alignment direction size 201a and the window vertical direction size 201b of the plus window 201 is the size of a plurality of pixels.
  • each of the window arrangement direction size 202a and the window vertical direction size 202b of the minus window 202 is the size of a plurality of pixels.
  • the generation unit 602 sets the filter window 200 in the depth image 706 so that the first pixel of interest is positioned at the center of the plus window 201, for example. Then, in the unit filtering process, the generation unit 602 multiplies each pixel value of a plurality of first pixels including the target first pixel in the plus window 201 by a predetermined plus value.
  • the generation unit 602 obtains the total sum of the plurality of first pixel values multiplied by the predetermined positive value as the positive total sum value.
  • a positive total value indicates a positive value.
  • the generation unit 602 multiplies the pixel value of each first pixel in the minus window 202 by a negative value.
  • the generating unit 602 obtains the total sum of the plurality of first pixel values multiplied by the predetermined negative value as the negative total sum value.
  • a negative total value indicates a negative value.
  • the generation unit 602 obtains a negative total sum value for each of the two negative partial windows 202 .
  • the generation unit 602 obtains the total sum of the positive total value and the two negative total values as a calculation value.
  • the generation unit 602 sets the calculated value as the second pixel value corresponding to the first pixel value of interest in the high-position image data 720 .
  • the generation unit 602 sets the second pixel value corresponding to the first pixel value of interest to zero.
  • the second pixel value of the high position image becomes a value equal to or greater than zero. Note that even if the calculated value is a negative value, the calculated value may be used as the second pixel value as it is.
  • the filter window 200 is set in the depth image 706 so as to include, for example, the first pixel of interest and a plurality of surrounding first pixels. Then, a predetermined operation is performed on the pixel values of the first pixel of interest and a plurality of surrounding first pixels to obtain a second pixel value corresponding to the first pixel of interest.
  • the first spatial filter function is a function representing such processing.
  • the position of the first pixel of interest moves, for example, along the raster scan direction. That is, the filter window 200 moves on the depth image 706 along, for example, the raster scan direction.
  • the first pixel at the pixel position (0, 0) of the depth image 706 is set as the first pixel of interest.
  • the first pixel of interest is moved by one pixel along the row direction, and the first pixel at the pixel position (1, 0) is set as the first pixel of interest.
  • unit filtering is executed each time the first pixel of interest moves by one pixel along the row direction.
  • the first pixel at the pixel position (M, 0) is set as the first pixel of interest
  • the first pixel at the pixel position (M, 1) is set as the first pixel of interest.
  • the first pixel of interest is moved by one pixel along the row direction, and the first pixel at the pixel position (M, 1) is set as the first pixel of interest. is the first pixel of interest.
  • the unit filtering process is executed with the first pixel at the pixel position (M, N) as the first pixel of interest, a plurality of are obtained, and the generation process ends.
  • the size of the plus window 201 is, for example, a size corresponding to the object size.
  • the window alignment direction size 201a and the window vertical direction size 201b of the plus window 201 are each sized according to the object size.
  • the window alignment direction size 201a and the window vertical direction size 201b are sizes corresponding to the size of the object 10 in a specific direction.
  • the window arrangement direction size 201a may be a size corresponding to the width direction size of the object 10, may be a size corresponding to the depth direction size of the object 10, or may be a size corresponding to the height direction size of the object 10.
  • the window vertical size 201b may be a size corresponding to the width direction size of the object 10, a size corresponding to the depth direction size of the object 10, or a height direction size of the object 10.
  • the size in the width direction may be read as the size in the horizontal direction or the size in the longitudinal direction
  • the size in the depth direction may be read as the size in the vertical direction or the size in the width direction
  • the size in the height direction may be read as the size in the thickness direction. can be read as
  • the window alignment direction size 201a and the window vertical direction size 201b correspond to the size of the object 10 on the captured image of the camera 40a or camera 41 of the sensor device 4, for example. That is, the size of the object 10 on the captured image of the camera 40a or the camera 41 of the sensor device 4 is adopted as the window alignment direction size 201a and the window vertical direction size 201b.
  • a typical pose of the object 10 on the tray 13 is taken as a reference pose, and the object 10 in the reference pose on the tray 13 (referred to as the reference object 10) is captured by the camera 40a or the camera 41.
  • the photographed image obtained is called a reference photographed image.
  • the window alignment direction size 201a and the window vertical direction size 201b each match, for example, the size of the reference object 10 on the reference captured image.
  • each of the window alignment direction size 201a and the window vertical direction size 201b matches the size of the reference object 10 in the specific direction on the reference captured image.
  • the window arrangement direction size 201a may match the width direction size of the reference object 10 on the reference captured image, or may match the depth direction size of the reference object 10 on the reference captured image.
  • the window vertical direction size 201b may match the width direction size of the reference object 10 on the reference captured image, or may match the depth direction size of the reference object 10 on the reference captured image. , may match the height direction size of the reference object 10 on the reference captured image.
  • the window alignment direction size 201a and the window vertical direction size 201b may be different from each other, or may be the same as each other. In the former case, for example, the window alignment direction size 201a matches the depth direction size of the reference object 10 on the reference captured image, and the window vertical direction size 201b matches the width direction size of the reference object 10 on the reference captured image. You may
  • the size of the minus window 202 corresponds to the size of the object.
  • each of the window alignment direction size 202a and the window vertical direction size 202b of the minus window 202 is a size corresponding to the object size.
  • Each of the window alignment direction size 202a and the window vertical direction size 202b matches, for example, the size of the reference object 10 on the reference captured image, similarly to the window alignment direction size 201a and the window vertical direction size 201b.
  • the window arrangement direction size 202a may match the width direction size of the reference object 10 on the reference captured image, or may match the depth direction size of the reference object 10 on the reference captured image. It may match the height direction size of the reference object 10 on the captured image.
  • the window vertical direction size 202b may match the width direction size of the reference object 10 on the reference captured image, or may match the depth direction size of the reference object 10 on the reference captured image. , may match the height direction size of the reference object 10 on the reference captured image.
  • the window alignment direction size 202a and the window vertical direction size 202b may match each other or may differ from each other.
  • the size corresponding to the object size matches the size of the reference object 10 on the reference captured image, but it does not have to match.
  • the size corresponding to the object size may be 1/2 to 2 times the size of the reference object 10 on the reference captured image.
  • the window alignment direction size 202a of the minus window 202 matches the window alignment direction size 201a of the plus window 201, but it does not have to match the window alignment direction size 201a.
  • the window vertical size 202b of the minus window 202 matches the window vertical size 201b of the plus window 201, but it does not have to match the window vertical size 201b.
  • the sizes 202a in the window arrangement direction of the two minus windows 202 match each other, but they may differ from each other.
  • the window vertical direction sizes 202b of the two minus windows 202 are the same, but they may be different.
  • the window arrangement direction size 201a corresponding to the object size may be stored in the storage unit 63 of the information processing device 6 in advance. Also, the window arrangement direction size 201a may be input to the information processing device 6 by the user. In this case, if the information processing device 6 has an input unit that receives user input, the user may directly input the window alignment direction size 201a to the information processing device 6 . Further, when the external device includes an input device that receives input from the user, the user may input the window alignment direction size 201a to the information processing device 6 through the input device. Alternatively, the control unit 60 of the information processing device 6 or the robot control unit 5 may determine the window arrangement direction size 201a.
  • the control unit 60 or the robot control unit 5 calculates the size of the reference object 10 on the reference captured image based on the image data representing the reference captured image obtained by the camera 40a or the camera 41, for example.
  • the window alignment direction size 201a may be determined based on the calculated size.
  • the control unit 60 or the robot control unit 5 can perform reference photography based on CAD (Computer Aided Design) data representing the outer shape of the object 10 and characteristic data of the camera 40a or camera 41 without using image data.
  • the size of the reference object 10 on the image may be calculated, and the window alignment direction size 201a may be determined based on the calculated size.
  • the above description of the window alignment direction size 201a can also be applied to the window vertical direction size 201b, the window alignment direction size 202a, and the window vertical direction size 202b.
  • FIGS. 7 and 8 are schematic diagrams for explaining an example of unit filtering.
  • a graph 900 shows the relationship between each pixel position of the focused row and the pixel value of the pixel at that pixel position (that is, the first pixel value). It is shown.
  • each position of the target row of the depth image 706 is indicated on the horizontal axis, and the first pixel value is indicated on the vertical axis.
  • FIG. 7 also shows the filter window 200 set for the area including part of the line of interest in the depth image 706 in line with the position of the horizontal axis of the graph 900 .
  • the camera 40a of the depth sensor 40 photographs the tray 13 from above.
  • the depth to the measurement point on the surface of the object in the measurement space 100 is the depth from the depth sensor 40 to the measurement point. Therefore, it can be said that a measurement point with a large depth is a measurement point with a small height from the depth side of the measurement space 100 when viewed from the depth sensor 40 . Therefore, the first pixel value of the depth image 706 can be regarded as representing the height of the corresponding measurement point. That is, it can be said that the depth image 706 is an image representing the height of each measurement point, and the depth information 705 is information representing the height of each measurement point.
  • the depth information 700 can also be said to be information representing the height of each measurement point.
  • the larger the first pixel value the smaller the depth to the corresponding measurement point. Therefore, the greater the first pixel value, the greater the height of the corresponding measurement point.
  • a partial image corresponding to the object height position portion included in the overall height position portion that is, a partial image corresponding to a portion of one target object 10 at a higher position than the others, is processed as a first
  • This is sometimes called an object height position equivalent image.
  • a plurality of measurement points respectively corresponding to a plurality of pixels forming the first object height position equivalent image are positioned on the target object 10 . Since the higher the first pixel value, the higher the height of the corresponding measurement point, the first pixel value of the first object height position equivalent image is the first pixel value of the surrounding portion of the first object height position equivalent image. The value is greater than one pixel value.
  • the area where the value is larger than the surrounding area is the first pixel value of the first object height position equivalent image. It can be said that the line connecting the points indicating the first pixel values in the graph 900 represents the shape of the surface of the object group 10A when the object group 10A is viewed from the depth sensor 40 .
  • the filter window 200 is set in the depth image 706 so that the first pixel of interest is positioned at the center of the plus window 201, for example.
  • the filter window 200 is set so that the first pixel of interest is included in the first object height equivalent image.
  • the measurement point corresponding to the first pixel of interest is a measurement point positioned higher than the others.
  • the filter window 200 is set so that the first pixel of interest is not included in the first object height position equivalent image.
  • the measurement point corresponding to the first pixel of interest is not located higher than the other measurement points.
  • a measurement point positioned higher than other measurement points may be referred to as a high-position measurement point.
  • the absolute value of the positive total value based on the positive window 201 is a relatively large value.
  • the absolute value of the negative total value based on the negative window 202 is not so large. Therefore, the calculated value, which is the sum of the positive total value and the two negative total values, indicates a positive value.
  • the pixel value of the second pixel of the high-position image which corresponds to the first pixel of interest included in the first object-high-position-equivalent image, indicates a positive value.
  • the absolute value of the positive total value based on the positive window 201 is a relatively small value.
  • the absolute value of the negative total value based on the negative window 202 is a relatively large value. Therefore, the calculated value, which is the sum of the positive total value and the two negative total values, indicates a negative value. As a result, the pixel value of the second pixel of the high position image corresponding to the target first pixel not included in the first object high position corresponding image becomes zero.
  • the calculated value will be zero, for example.
  • FIG. 9 is for explaining an example of unit filter processing when the size of the plus window 201 and the minus window 202 does not correspond to the size of the object and the size of the plus window 201 and the minus window 202 is too large.
  • 1 is a schematic diagram of FIG.
  • the filter window 200 is set so that the first pixel of interest is included in the first object height position corresponding image.
  • the positive window 201 since the positive window 201 includes the image surrounding the first object height position equivalent image including the first pixel of interest, the absolute value of the positive total value is different from that in the example of FIG. small value.
  • the minus window 202 includes the first pixel having a large first pixel value
  • the absolute value of the minus total value based on the minus window 202 is larger than in the example of FIG. Therefore, the calculated value is smaller than in the example of FIG. For example, the calculated value becomes a negative value.
  • the pixel value of the second pixel corresponding to the first pixel of interest included in the first object height position equivalent image of the depth image is reduced.
  • FIG. 10 is for explaining an example of unit filter processing when the size of the plus window 201 and the minus window 202 does not correspond to the size of the object and the size of the plus window 201 and the minus window 202 is too small.
  • 1 is a schematic diagram of FIG.
  • the filter window 200 is set so that the first pixel of interest is included in the first object height position equivalent image.
  • the plus window 201 does not include the image around the first object height position equivalent image including the first pixel of interest, the absolute value of the plus total value is the same as in the example of FIG. similarly large value.
  • the minus window 202 includes the first pixel of the first object height position equivalent image including the first pixel of interest
  • the absolute value of the minus total value based on the minus window 202 is compared with the example in FIG. and grow. Therefore, the calculated value is smaller than in the example of FIG.
  • the pixel value of the second pixel corresponding to the first pixel of interest included in the first object height position equivalent image of the depth image is reduced.
  • the size of the plus window 201 is the size corresponding to the object size
  • the size of the plus window 201 is set to the size of the image corresponding to the first object height position of the depth image. It is possible to get closer.
  • the first pixel of interest is included in the first object height equivalent image as in the example of FIG.
  • the absolute value of the negative total value tends to be small. Therefore, the calculated value tends to increase.
  • the pixel value of the second pixel corresponding to the first pixel of interest in the high position image tends to increase.
  • the measurement point corresponding to the first pixel of interest is positioned higher than the others, so the measurement point located on the object 10 is higher than the others. position, the second pixel value corresponding to the measurement point tends to increase.
  • the absolute value of the plus sum value is small, and at least one of the two minus sum values is The absolute value tends to increase. Therefore, the calculated value tends to be small. If the first pixel of interest is not included in the first image corresponding to the object height position, the measurement point corresponding to the first pixel of interest is not at a higher position than the others. If the measurement point is not positioned higher than the others, the second pixel value corresponding to the measurement point tends to be small.
  • the second pixel values corresponding to the high-position measurement points tend to be large.
  • the second pixel values corresponding to measurement points that are not high-position measurement points tend to be small. That is, the second pixel value of the measurement point changes depending on whether or not the measurement point is higher than the others.
  • the high position information 720 can appropriately represent a high position portion of the object group 10A, that is, a portion of the object group 10A that is positioned higher than the others. Therefore, it is possible to appropriately identify the high position portion based on the high position information 720 .
  • a partial image having a second pixel value equal to or greater than the threshold is an image corresponding to the high-position portion of the object group 10A (also referred to as a second high-position equivalent image).
  • a second high-position equivalent image of the high-position image a continuous region composed of a plurality of pixels whose positions are continuous is specified, and the specified continuous region is an image corresponding to the object high position portion of one target object 10. (also referred to as a second object height position equivalent image).
  • each object high position portion can be specified individually from the high position information 720 .
  • the minus window 202 tends to include pixels with a large first pixel value.
  • the absolute value of the negative total value tends to increase, and as a result, the calculated value tends to decrease. Therefore, there is a possibility that the measurement point corresponding to the first pixel of interest will not be identified as the high position measurement point.
  • the size of the minus window 202 corresponds to the size of the object, as in the example of FIG. Therefore, when the first pixel of interest is included in the first image corresponding to the object height position, the absolute value of the negative total value is less likely to increase, and the calculated value is less likely to decrease. This makes it easier to identify the measurement point corresponding to the first pixel of interest as the high position measurement point.
  • the size of the minus window 202 is too small, the number of pixels included in the minus window 202 is reduced. In this case, if the pixel values of the specific pixels included in the minus window 202 contain a lot of noise, the error of the minus total value may increase. As the error of the negative total value increases, the error of the calculated value increases. As a result, the measurement point corresponding to the first pixel of interest may not be identified as the high position measurement point.
  • the size of the minus window 202 corresponds to the size of the object, even if the pixel values of specific pixels included in the minus window 202 contain a lot of noise, The error of the negative total value can be reduced, and as a result, the error of the pixel value of the second pixel corresponding to the first pixel of interest can be reduced. Therefore, the measurement point corresponding to the first pixel of interest is likely to be identified as the high position measurement point.
  • the plus window 201 is sandwiched between the two minus windows 202, it is possible to specify the object height position portion having spaces on both sides in the surroundings from the height position information 720. This point will be explained below.
  • the direction corresponding to the window alignment direction of the filter windows 200 set in the depth image is called the filter direction.
  • the directions corresponding to the row direction and column direction of the depth image in the measurement space 100 are called the x direction and the y direction, respectively.
  • the filter direction matches the x direction.
  • partial images with small first pixel values are located on both sides of the first object height position equivalent image including the first pixel of interest in the window arrangement direction. Therefore, on both sides in the filter direction of the object height portion corresponding to the first object height equivalent image including the first pixel of interest (also referred to as the object height portion of interest), there is a space where no object exists.
  • the filter window 200 in which the positive window 201 is sandwiched between two negative windows 202 is set as shown in FIG. 7, the pixel value of the second pixel corresponding to the first pixel of interest increases.
  • the pixel value of the second pixel corresponding to the first pixel of interest changes when a partial image with a small first pixel value is located only on one side of the position-equivalent image in the window arrangement direction.
  • the second pixel corresponding to the target first pixel pixel value changes.
  • the filter direction coincides with the x direction corresponding to the row direction of the depth image in the measurement space 100
  • at least the object height position portion having spaces on both sides in the x direction is identified as the high position information 720 easier to identify from
  • the end effector 21 can easily grip the target object 10 including the object high position portion specified from the high position information 720 from both sides in the filter direction.
  • the filter window 200 does not have to include one of the two minus windows 202 .
  • the window alignment direction size 202 a of the minus window 202 determines the size of the space around the object high position portion specified from the high position information 720 .
  • the window arrangement direction size 202a of the minus window 202 on the right side is large, the space on the right side of the filter direction of the object high position portion specified from the high position information 720 becomes large. That is, when the window arrangement direction size 202a of the minus window 202 on the right side is large, an object high position portion with a large space on the right side in the filter direction is specified from the high position information 720.
  • FIG. By adjusting the window arrangement direction size 202a of the minus window 202, it is possible to acquire the high position information 720 in consideration of the size of the space around the object high position portion. This point will be explained below.
  • each minus window 202 When the window alignment direction size 202a of each minus window 202 is large, as shown in FIG. If the partial image is spread out, it becomes difficult for the first pixels with large first pixel values to be included in each minus window 202 . In other words, if partial images with small first pixel values are spread on both sides of the first object height position equivalent image including the first pixel of interest in the window arrangement direction, when the window arrangement direction size 202a of the minus window 202 is large. However, the pixel value of the second pixel corresponding to the first pixel of interest increases.
  • the object height position has a large space on the left side in the filter direction and a small space on the right side in the filter direction.
  • a portion is identified from high position information 720 .
  • the size of the space around the object height position portion specified from the height position information 720 can be adjusted.
  • the window arrangement direction size 202a is set to a size corresponding to the object size, as in the above example, the height position information 720 indicates that the size of the surrounding space corresponds to the object size. can be specified.
  • the size 202a of the minus window 202 in the window alignment direction may be a size corresponding to the thickness of the end effector 21 fingers.
  • the window arrangement direction size 202a of the minus window 202 may be matched with the thickness of the finger of the end effector 21 on the captured image obtained by the camera 40a or the camera 41, as described above.
  • the high position information 720 that can specify the object high position portion whose size of the surrounding space corresponds to the thickness of the finger of the end effector 21 . Therefore, the object 10 that can be gripped by the end effector 21 with a plurality of fingers can be determined as the object 10 to be held.
  • the identifying unit 603 that identifies an object image in which the target object 10 is captured in the color image 711 binarizes the high position image represented by the high position information 720 to generate a binarized image, for example. For example, for each of the second pixel values of the plurality of second pixels forming the high-position image, the identifying unit 603 converts the second pixel value equal to or greater than the threshold to "1", and converts the second pixel value equal to or greater than the threshold to "1". 2 Change the pixel value to "0". As a result, the high-position image is converted into a binarized image.
  • a region having a pixel value of "1" in the binarized image corresponds to the high position portion of the object group 10A.
  • a region having a pixel value of "1” will be referred to as a high-value region
  • a region having a pixel value of "0” will be referred to as a low-value region.
  • the identifying unit 603 identifies a continuous region made up of a plurality of pixels with continuous positions, which is included in the high value region corresponding to the overall high position portion.
  • a continuous region can also be called an independent region or an island region.
  • a plurality of continuous regions included in the high-value region correspond to object high position portions of the plurality of target objects 10, respectively.
  • a region corresponding to one continuous region of the binarized image corresponds to one object high position portion and is a second object high position corresponding image.
  • the identification unit 603 can identify a plurality of continuous regions included in the high price region by labeling the high price region using 4-connection, 8-connection, or the like.
  • the specifying unit 603 specifies, for each specified continuous area, a partial image corresponding to the continuous area in the color image 711 . Then, the specifying unit 603 sets the specified partial image as an object image including the target object 10 .
  • the partial image corresponding to the continuous area corresponding to the object height position portion of the target object 10 is the object image in which the target object 10 is shown.
  • a partial image corresponding to the continuous area in the color image 711 is a partial image (third object high position It is also called an equivalent image). Therefore, the third object height position corresponding image included in the color image 711 is the object image in which the target object 10 is shown. Since the object height position portion captured in the third object height position equivalent image constitutes at least a portion of the target object 10, at least a portion of the target object 10 is captured in the object image.
  • FIG. 13 is a schematic diagram showing an example of how a range 713 extended from the third object height position corresponding image 712 in the color image 711 is used as the object image 714 .
  • An example of the third object height equivalent image 712 is shown on the left side of FIG.
  • the right side of FIG. 13 shows a range 713 expanded from the third object height equivalent image 712 shown on the left side.
  • the specifying unit 603 may use the range 713 extended from the third object height position corresponding image 712 as the object image 714 based on the color information of the measurement space 100 .
  • the specifying unit 603 may use the range 713 extended from the third object height position equivalent image 712 as the object image 714 based on the color image data 710 .
  • the specifying unit 603 specifies, for example, a region in the color image 711 that is continuous with the third object high position equivalent image 712 and that has the same color as the third object high position equivalent image 712 . do.
  • the specifying unit 603 sets a range 713 extended from the third object height position equivalent image 712 to the specified region as an object image 714 .
  • the specifying unit 603 adds the specified region to the third object height position equivalent image 712 to obtain the object image 714 . This enlarges the size of the target object 10 appearing in the object image.
  • the specifying unit 603 may use the range 713 extended from the third object height position equivalent image 712 as the object image 714 based on the edge image of the measurement space 100 .
  • the specifying unit 603 generates an edge image of the measurement space 100 by performing edge detection on the color image 711 showing the measurement space 100, for example.
  • the Sobel method, the Laplacian method, the Canny method, or the like is used as the edge image generation method.
  • the identifying unit 603 identifies a portion of the edge image corresponding to the third object height position equivalent image 712 (also referred to as a corresponding partial edge image).
  • the specifying unit 603 specifies a range extending from the corresponding partial edge image to the edge corresponding to the contour of the target object 10 (the region where the luminance changes rapidly) in the edge image. Then, the specifying unit 603 sets a portion corresponding to the specified range in the color image 711 as a range 713 extended from the third object height position equivalent image 712 based on the edge image, and sets the range 713 as an object image 714. .
  • the target object 10 of interest among the plurality of target objects 10 is called the attention target object 10 .
  • the degree acquisition unit 604 may, for example, obtain an evaluation value representing the ease of holding the target object 10 and employ the obtained evaluation value as the degree of ease of holding the target object 10 .
  • the evaluation value can be obtained from various viewpoints.
  • the degree acquisition unit 604 identifies a partial image (also referred to as an object-corresponding image) corresponding to the specific object image in which the target object 10 is shown in the depth image 706 .
  • the degree obtaining unit 604 obtains the average value or the maximum value of the pixel values of the plurality of first pixels forming the specified object-based image.
  • the mean value and maximum value for the object-oriented image are called the first mean value and the first maximum value, respectively.
  • the degree acquisition unit 604 may use the first average value or the first maximum value as the evaluation value.
  • the degree acquisition unit 604 identifies a partial image 707b (also referred to as a surrounding partial image 707b) surrounding the object corresponding image 707a in the depth image 706, and extracts the identified surrounding partial image 707b.
  • An average value or a maximum value of the pixel values of a plurality of constituent first pixels may be obtained.
  • the average value and maximum value for the surrounding partial image 707b are called the second average value and the second maximum value, respectively.
  • the degree acquiring unit 604 may use a difference value obtained by subtracting either the second average value or the second maximum value from either the first average value or the first maximum value as the evaluation value. Since the difference value increases as the target object 10 is positioned higher than its surroundings, it can be said that the target object 10 is easier to hold as the difference value increases.
  • the width 707bb of the surrounding partial image 707b may be a size corresponding to the thickness of the end effector 21 fingers.
  • the width 707bb may match the thickness of the finger of the end effector 21 on the captured image obtained by the camera 40a or the camera 41.
  • FIG. 14 the surrounding partial image 707b surrounds the object corresponding image 707a, but it does not have to surround the object corresponding image 707a.
  • the degree acquisition unit 604 may also identify the size of the space around the target object 10 based on the depth image 706 and determine the evaluation value based on the identified size. In this case, the degree obtaining unit 604 increases the evaluation value, for example, as the size of the space around the target object 10 increases.
  • the degree acquisition unit 604 can specify the size of the space around the target object 10 by specifying, for example, a range of small first pixel values around the object corresponding image 707a in the depth image 706 .
  • the hold target determination unit 605 determines the hold target object 10 as the hold target object 10 that has the highest degree of ease of holding among the plurality of target objects 10 . If there are a plurality of target objects 10 with the highest degree of ease of holding, the hold target determination unit 605 may determine the hold target object 10 based on other information. For example, for each of the plurality of target objects 10 having the highest degree of ease of holding, the exposed area of the target object 10 is obtained based on the specific object image in which the target object 10 is captured. Then, the degree acquisition unit 604 may set the target object 10 having the largest exposed area as the hold target object 10 among the plurality of target objects 10 having the largest holdability degree.
  • robot system 1 has been described above, the robot system 1 is not limited to the above example. Another example of the robot system 1 will be described below.
  • the generation unit 602 may generate the high position information 720 by filtering the depth image 706 using multiple types of spatial filter functions according to the object size. For example, the generation unit 602 may generate the high position information 720 by filtering the depth image 706 using the first spatial filter function and the second spatial filter function according to the object size. good.
  • FIG. 15 is a schematic diagram showing an example of a filter window 220 of the second spatial filter function.
  • the filter window 220 has, for example, a plus window 201 and two minus windows 202, similar to the filter window 200. Unlike the filter window 200, the window alignment direction and window vertical direction of the filter window 220 match the column direction and row direction of the depth image 706, respectively.
  • the generation unit 602 filters the depth image using the filter window 200 in the same manner as described above.
  • the filtered depth image generated using the filter window 200 is referred to as a first filtered image.
  • the generator 602 also filters the depth image using the filter window 210 in the same manner as the filter window 200 is used. This filtered depth image is called a second filtered image. Then, the generation unit 602 sets a synthesized image obtained by synthesizing the first filtered image and the second filtered image as a high position image.
  • the generation unit 602 generates a value obtained by adding the pixel value of the pixel at the pixel position (m, n) in the first filtered image and the pixel value of the pixel at the pixel position (m, n) in the second filtered image. , the pixel value of the pixel at the pixel position (m, n) of the synthesized image.
  • the generation unit 602 generates a composite image by performing this process for each pixel position.
  • FIG. 16 is a schematic diagram illustrating an example filter window 220 for a third spatial filter function and an example filter window 230 for a fourth spatial filter function.
  • Each of the filter windows 220 and 230 has a plus window 201 and two minus windows 202, similar to the filter windows 200 and 210.
  • the window alignment direction of the filter windows 220 is set to the right oblique direction of the depth image 706 .
  • the window alignment direction of the filter windows 230 is set to the left oblique direction of the depth image 706 .
  • the generation unit 602 When using the filter windows 200, 210, 220, and 230, the generation unit 602 generates the first filtered image and the second filtered image as described above.
  • the generator 602 also filters the depth image using the filter window 220 .
  • the depth image after this filtering process is called a third filtered image.
  • the generator 602 also filters the depth image using the filter window 230 .
  • the depth image after this filtering process is called a fourth filtered image.
  • the generation unit 602 sets a synthesized image obtained by synthesizing the first filtered image, the second filtered image, the third filtered image, and the fourth filtered image as the high position image.
  • the generating unit 602 generates, for example, the pixel value of the pixel at the pixel position (m, n) of the first filtered image, the pixel value of the pixel at the pixel position (m, n) of the second filtered image, and the pixel value of the pixel at the pixel position (m, n) of the third filtered image.
  • a value obtained by adding the pixel value of the pixel at the pixel position (m, n) and the pixel value of the pixel at the pixel position (m, n) of the fourth filtered image is defined as the pixel position (m, n) of the synthesized image. is the pixel value of the pixel in
  • the generation unit 602 generates a composite image by performing this process for each pixel position. From the high position information 720 representing the high position image generated in this way, it becomes easy to identify the object high position portion having a space in at least one of the four directions around it.
  • the generation unit 602 may generate the high position information 720 using five or more types of filter windows with different window alignment directions. Also, the generator 602 may generate the high position information 720 using at least one of the filter windows 200 , 210 , 220 and 230 .
  • the generation unit 602 may generate the high position information 720 using multiple types of spatial filter functions corresponding to the sizes of the object 10 in multiple directions.
  • FIG. 17 is a schematic diagram showing an example of filter windows of multiple types of spatial filter functions corresponding to sizes of the object 10 in multiple directions.
  • FIG. 17 shows a filter window 240 of the spatial filter function corresponding to the height direction size of the object 10, a filter window 250 of the spatial filter function corresponding to the depth direction size of the object 10, and a width direction size of the object 10.
  • a filter window 260 of the spatial filter is shown.
  • FIG. 17 shows an example in which the size in the width direction, the size in the depth direction, and the size in the height direction of the object 10 are larger in this order.
  • Each of the filter windows 240 , 250 , 260 has the same configuration as the filter window 200 .
  • the window arrangement direction size and window vertical direction size of the plus window 201 and the minus window 202 of the filter window 240 match the height direction size of the reference object 10 on the reference captured image.
  • the window alignment direction size and window vertical direction size of the plus window 201 and the minus window 202 of the filter window 250 match the depth direction size of the reference object 10 on the reference captured image.
  • the window alignment direction size and window vertical direction size of the plus window 201 and the minus window 202 of the filter window 260 match the width direction size of the reference object 10 on the reference captured image.
  • the generation unit 602 generates a depth image filtered using the filter window 240 (also referred to as a fifth filtered image), a depth image filtered using the filter window 250 (also referred to as a sixth filtered image), and a filter A filtered depth image (also referred to as a seventh filtered image) is prepared using window 260 . Then, the generation unit 602 generates a synthesized image by synthesizing the fifth filtered image, the sixth filtered image, and the seventh filtered image in the same manner as described above, and sets the synthesized image as the high position image.
  • the orientation of the object 10 in the object group 10A on the tray 13 is determined. It becomes easy to specify the high-position portion of the object group 10A regardless of the variation in . In other words, regardless of how each object 10 looks when the object group 10A is viewed from the sensor device 4, it becomes easier to identify the high position portion of the object group 10A. Furthermore, in other words, regardless of how each object 10 appears in the captured images obtained by the cameras 40a and 41, it becomes easier to identify the high-position portion of the object group 10A. This makes it easier to identify the object 10 that is easy to hold. Therefore, the end effector 21 can easily hold the object 10 .
  • the configuration of the filter window used by the generation unit 602 is not limited to the above example.
  • the filter windows of at least one type of spatial filter function used by the generator 602 may include the filter window 270 shown in the upper part of FIG. 18 or the filter window 280 shown in the middle of FIG. may be included, or the filter window 290 shown at the bottom of FIG. 18 may be included.
  • filter window 270 plus window 201 is surrounded by minus window 202 .
  • minus windows 202 are arranged above and below and to the left and right of the plus window 201 . If a filter window 280 is used, high object portions with surrounding x and y spaces are more likely to be identified.
  • the outer shape of the filter window 290 is circular. In addition, the outer shape of the filter window may be other than rectangular and circular.
  • the arrangement of the plurality of minus partial windows 202 included in the filter window may be determined according to the arrangement of the fingers of the end effector 21 of the robot 2 .
  • the plurality of negative partial windows 202 of the filter window may be arranged like the filter window 200 in FIG. may be arranged like the filter window 210 of , or may be arranged like the filter window 290 of FIG.
  • the plurality of minus partial windows 202 of the filter window may be arranged like the filter window 280 in FIG. It may be arranged like the filter window 270 of FIG.
  • the predetermined plus value for the plus window 201 may be other than +1.0, and the predetermined minus value for the minus window 202 may be other than -0.5.
  • the pixel values of the plurality of first pixels in the plus window 201 are multiplied by a predetermined plus value, but a predetermined plus value may be added.
  • a predetermined negative value may be added to the pixel values of the plurality of first pixels within the negative window 202 .
  • the plus window 201 may be changed to the minus window 202 and the minus window 202 may be changed to the plus window 201 .
  • the second pixel value corresponding to the measurement point positioned higher than the others is The second pixel value corresponding to a measurement point that is not positioned higher than the others tends to increase. Therefore, in the high position image represented by the high position information 720, the partial image having the second pixel value less than the threshold is the image corresponding to the high position portion of the object group 10A (that is, the second high position corresponding image). Become.
  • a plurality of filter images respectively corresponding to a plurality of types of filter windows are synthesized, but they do not have to be synthesized.
  • a certain type of filter window is used to generate a filtered image (step s3), the generated filtered image is used as a high position image, and steps s4 to s7 are performed. may be executed.
  • step s4 if the binarized image of the high position image does not include a high value region, that is, if the high position portion of the object group 10A is not detected, steps s5 to s7 are not executed and another type
  • the holding pose determination process may be performed again using a filter window of .
  • step s4 of this holding posture determination process if the binarized image of the high position image does not include a high-value region, another type of filter window may be used and the holding posture determination process may be executed again. .
  • the specifying unit 603 uses the object image as a range extended from the third object high position equivalent image based on the edge image or color information of the measurement space 100, but based on the depth information 705, the third A range extended from the image corresponding to the object height position may be used as the object image.
  • the generation unit 602 functions as part of the identification unit 603 .
  • the identifying unit 603 filters the depth image 706 using, for example, two types of filter windows whose window alignment directions are perpendicular to each other. Then, the specifying unit 603 determines, as the object image, the range extended from the third object height position equivalent image based on the result of the filter processing in the color image 711 .
  • FIG. 19 is a schematic diagram for explaining an example of the operation of the identification unit 603.
  • FIG. 19 schematically shows an example of a partial image 706a included in the depth image 706 when the object 10 is elongated, such as a pencil.
  • a partial image 706 a is an image representing the depth of one object 10 .
  • a filter window 200 and a filter window 210 are set for the partial image 701a.
  • the window alignment direction size and window vertical direction size of the plus window 201 and the minus window 202 of the filter windows 200 and 210 are set according to the size of the elongated object 10 in the width direction.
  • Filter windows 200 and 210 are set to include the first pixel of interest at the same pixel location.
  • the window alignment direction of the filter windows 200 matches the longitudinal direction of the object 10 corresponding to the partial image 706 a in the depth image 706 .
  • the window alignment direction of the filter windows 210 coincides with the direction corresponding to the lateral direction of the object 10 corresponding to the partial image 706 a in the depth image 706 .
  • the calculated value (also referred to as the first calculated value) obtained using the filter window 200 is Although it increases, the calculated value obtained using the filter window 210 (also referred to as a second calculated value) becomes smaller. Therefore, when the difference value obtained by subtracting the second calculated value from the first calculated value is equal to or greater than the threshold value, the certain object 10 is positioned in the measurement space 100 in the direction corresponding to the window arrangement direction of the filter windows 210 (the second 2 filter direction).
  • the specifying unit 603 generates a plurality of first calculation values obtained when the filter window 200 is used in the generation unit 602 to filter the depth image 706, and the filter window 210 is used in the generation unit 602. Based on a plurality of second calculation values obtained when the image 706 is filtered, the direction in which the object 10 shown in the third object height position equivalent image extends is identified. Then, the specifying unit 603 determines, as the object image, the range extended from the third object height position equivalent image based on the specified direction.
  • the first calculated value obtained when the pixel position of the first pixel of interest is the pixel position (m, n) and the pixel position (m, n) correspond to each other.
  • the second calculation value obtained when the pixel position of the first pixel of interest is the pixel position (m, n) and the pixel position (m, n) are expressed as mutually corresponding.
  • the identifying unit 603 selects a plurality of first calculated values respectively corresponding to a plurality of third pixels forming a third object height position corresponding image from a plurality of first calculated values obtained by filtering based on the filter window 200. Identify. The first calculated value corresponding to the third pixel is the first calculated value corresponding to the same pixel position as that of the third pixel. Further, the identifying unit 603 selects a plurality of second calculation values corresponding to a plurality of third pixels forming the third object height position equivalent image from a plurality of second calculation values obtained by filtering based on the filter window 210. Identify a value. The second calculated value corresponding to the third pixel is the second calculated value corresponding to the same pixel position as that of the third pixel.
  • the identifying unit 603 obtains a difference value between the first calculated value and the second calculated value corresponding to the same pixel position with respect to the specified plurality of first calculated values and the plurality of second calculated values. This difference value is a difference value obtained by subtracting the second calculated value from the first calculated value. Then, when the obtained plurality of difference values include a difference value larger than the threshold value, the identifying unit 603 determines that the object 10 captured in the third object height position equivalent image extends along the second filter direction. judge that there is When the identification unit 603 finds that the object 10 captured in the third object height position equivalent image extends along the second filter direction, the identification unit 603 determines the position of the filter window 210 from the third object height position equivalent image in the color image 711 .
  • the object image is defined as the range expanded along the window arrangement direction.
  • the specifying unit 603 is positioned in the window alignment direction of the filter window 210 with respect to the third object height position equivalent image, and Identify regions that have pixel values. Then, the identifying unit 603 determines, as an object image, a range expanded from the third object height position equivalent image to the identified region in the color image 711 .
  • the representative pixel value of the third object height position equivalent image may be, for example, an average value of pixel values of a plurality of third pixels forming the third object height position equivalent image.
  • filter windows 200 and 210 are not limited to the filter windows 200 and 210.
  • the filter windows 220 and 230 shown in FIG. 16 above may be used.
  • a grayscale image may be used instead of the color image 711.
  • the specifying unit 603 may specify an object image in which the target object 10 is shown in the grayscale image based on the high position information 720 in the same manner as described above.
  • the holding posture determination unit 606 may determine the direction in which the end effector 21 grips the holding target object 10 based on the window arrangement direction of the filter windows used in the holding posture determination processing. For example, when the end effector 21 grips the object 10 with two fingers and the filter window 200 of FIG. It may be decided to grip the hold target object 10 from a direction parallel to the filter direction.
  • the robot control unit 5 may control the opening/closing width of the finger of the end effector 21 based on the window arrangement direction size 202a of the minus partial window 202 of the filter window used in the holding posture determination process. For example, the robot control unit 5 increases the finger opening/closing width of the end effector 21 when the window alignment direction size 202a is large, and decreases the finger opening/closing width of the end effector 21 when the window alignment direction size 202a is small. good too.
  • robot control unit 6 information processing device 200, 210, 220, 230, 240, 250, 260, 270, 280, 290 filter window 201 plus partial window 202 minus partial window 600 information acquisition unit 602 generation unit 603 identification Unit 604 Degree acquisition unit 605 Holding object determination unit 606 Holding posture determination unit 630 Program 705 Depth information 706 Depth image 710 Color information 711 Color image 712 Third object height position equivalent image 714 Object image 720 High position information

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Geometry (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Orthopedic Medicine & Surgery (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)
PCT/JP2023/007351 2022-02-28 2023-02-28 情報処理装置、ロボット制御システム及びプログラム Ceased WO2023163219A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
CN202380021452.5A CN118696207A (zh) 2022-02-28 2023-02-28 信息处理装置、机器人控制系统以及程序
US18/841,599 US20250037289A1 (en) 2022-02-28 2023-02-28 Information processing device, robot control system, and non-transitory computer-readable recording medium
JP2024503317A JPWO2023163219A1 (https=) 2022-02-28 2023-02-28
EP23760217.2A EP4488616A1 (en) 2022-02-28 2023-02-28 Information processing device, robot control system, and program

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022029790 2022-02-28
JP2022-029790 2022-02-28

Publications (1)

Publication Number Publication Date
WO2023163219A1 true WO2023163219A1 (ja) 2023-08-31

Family

ID=87766342

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/007351 Ceased WO2023163219A1 (ja) 2022-02-28 2023-02-28 情報処理装置、ロボット制御システム及びプログラム

Country Status (5)

Country Link
US (1) US20250037289A1 (https=)
EP (1) EP4488616A1 (https=)
JP (1) JPWO2023163219A1 (https=)
CN (1) CN118696207A (https=)
WO (1) WO2023163219A1 (https=)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH09255158A (ja) * 1996-03-22 1997-09-30 Kobe Steel Ltd 物体配置認識装置
JP2014164579A (ja) * 2013-02-26 2014-09-08 Oki Electric Ind Co Ltd 情報処理装置、プログラム及び情報処理方法
JP2016148558A (ja) 2015-02-10 2016-08-18 キヤノン株式会社 情報処理装置、情報処理方法
WO2021177239A1 (ja) * 2020-03-05 2021-09-10 ファナック株式会社 取り出しシステム及び方法

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH03202290A (ja) * 1989-12-27 1991-09-04 Toyota Motor Corp ばら積み物体の取出装置
JP5458807B2 (ja) * 2009-10-30 2014-04-02 富士電機株式会社 対象物把持領域抽出装置および対象物把持領域抽出装置を用いたロボットシステム

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH09255158A (ja) * 1996-03-22 1997-09-30 Kobe Steel Ltd 物体配置認識装置
JP2014164579A (ja) * 2013-02-26 2014-09-08 Oki Electric Ind Co Ltd 情報処理装置、プログラム及び情報処理方法
JP2016148558A (ja) 2015-02-10 2016-08-18 キヤノン株式会社 情報処理装置、情報処理方法
WO2021177239A1 (ja) * 2020-03-05 2021-09-10 ファナック株式会社 取り出しシステム及び方法

Also Published As

Publication number Publication date
US20250037289A1 (en) 2025-01-30
EP4488616A1 (en) 2025-01-08
CN118696207A (zh) 2024-09-24
JPWO2023163219A1 (https=) 2023-08-31

Similar Documents

Publication Publication Date Title
US11948282B2 (en) Image processing apparatus, image processing method, and storage medium for lighting processing on image using model data
CN106683071B (zh) 图像的拼接方法和装置
CN105608671B (zh) 一种基于surf算法的图像拼接方法
KR101803712B1 (ko) 화상처리장치, 제어 방법, 프로그램 및 기록 매체
US11282232B2 (en) Camera calibration using depth data
CN105180836B (zh) 控制装置、机器人、以及控制方法
JP6570296B2 (ja) 画像処理装置、画像処理方法およびプログラム
US20150278996A1 (en) Image processing apparatus, method, and medium for generating color image data
US20200104969A1 (en) Information processing apparatus and storage medium
JP7298687B2 (ja) 物体認識装置及び物体認識方法
CN108550167B (zh) 深度图像生成方法、装置及电子设备
JP6305232B2 (ja) 情報処理装置、撮像装置、撮像システム、情報処理方法およびプログラム。
US20160073089A1 (en) Method for generating 3d image and electronic apparatus using the same
WO2023163219A1 (ja) 情報処理装置、ロボット制御システム及びプログラム
US20140152862A1 (en) Image processing apparatus, image pickup apparatus, image pickup system, image processing method, and non-transitory computer-readable storage medium
JPWO2018012524A1 (ja) 投影装置、投影方法および投影制御プログラム
CN105488845A (zh) 产生三维图像的方法及其电子装置
JP6320165B2 (ja) 画像処理装置及びその制御方法、並びにプログラム
JP6351364B2 (ja) 情報処理装置、情報処理方法およびプログラム
CN103905722A (zh) 图像处理装置以及图像处理方法
CN110677597A (zh) 图像处理方法及装置
JP2013200840A (ja) 映像処理装置、映像処理方法、映像処理プログラム、及び映像表示装置
KR102497593B1 (ko) 정보 처리장치, 정보 처리방법 및 기억매체
JP2019159470A (ja) 推定装置、推定方法、及び推定プログラム
JP7197211B2 (ja) 三次元グラフィックスデータ作成方法、プログラム及び三次元グラフィックスデータ作成システム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23760217

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2024503317

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 202380021452.5

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 18841599

Country of ref document: US

WWE Wipo information: entry into national phase

Ref document number: 2023760217

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2023760217

Country of ref document: EP

Effective date: 20240930