WO2025129666A1 - 自动抓取方法和自动抓取系统 - Google Patents

自动抓取方法和自动抓取系统 Download PDF

Info

Publication number
WO2025129666A1
WO2025129666A1 PCT/CN2023/141165 CN2023141165W WO2025129666A1 WO 2025129666 A1 WO2025129666 A1 WO 2025129666A1 CN 2023141165 W CN2023141165 W CN 2023141165W WO 2025129666 A1 WO2025129666 A1 WO 2025129666A1
Authority
WO
WIPO (PCT)
Prior art keywords
point cloud
graspable
objects
predetermined area
target range
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2023/141165
Other languages
English (en)
French (fr)
Inventor
杨剑
蔡迪
张学超
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Schaeffler Technologies AG and Co KG
Original Assignee
Schaeffler Technologies AG and Co KG
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Schaeffler Technologies AG and Co KG filed Critical Schaeffler Technologies AG and Co KG
Priority to PCT/CN2023/141165 priority Critical patent/WO2025129666A1/zh
Publication of WO2025129666A1 publication Critical patent/WO2025129666A1/zh
Anticipated expiration legal-status Critical
Pending legal-status Critical Current

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional [3D] objects

Definitions

  • the present invention relates to the field of robot technology, and in particular to an automatic grasping method and an automatic grasping system for a robot.
  • Robots are increasingly used in scenarios such as assembly line production and cargo sorting.
  • Robots can identify items in a predetermined area and accurately grab the identified items for further operations (such as placement, processing, etc.).
  • Currently commonly used automatic grasping systems are usually composed of computers, three-dimensional image acquisition components, robot operating systems, machine vision software and algorithms, AI (Artificial Intelligence) models, and robot path planning software.
  • This automatic grasping system is based on a three-dimensional vision solution, that is, three-dimensional images are used to identify graspable items in a predetermined area.
  • the automatic grasping method is used to control a robot to automatically grasp objects in a predetermined area.
  • the automatic grasping method includes: a model building step, in which an AI model for identifying graspable objects in a predetermined area is built; a two-dimensional image acquisition step, in which a two-dimensional image of a predetermined area is acquired; a target range determination step, in which a target range corresponding to one or more graspable objects is determined in the two-dimensional image based on the AI model; a three-dimensional point cloud generation step, in which, when the target range meets a first confidence level, the target range is determined.
  • the grasping method firstly, the graspable objects are identified in the two-dimensional image and the corresponding target range is determined, and then only the graspable objects within the target range are further identified in the three-dimensional image. In this way, the object of the three-dimensional image processing is reduced from the entire predetermined area to the target range obtained by the two-dimensional image processing, thereby improving the speed and accuracy of object recognition.
  • the model building step may include training the AI model, and the training of the AI model includes: providing a predetermined number of training two-dimensional images, wherein each training two-dimensional image includes multiple objects; manually marking the graspable objects in each training two-dimensional image; and using a verification set to verify the AI model.
  • the recognition accuracy of the AI model can be improved through training.
  • the automatic grasping method may further include an object positioning adjustment step, wherein, after the target range determination step, when the target range does not meet the first confidence condition, and/or, after the three-dimensional point cloud generation step, when the point cloud of one or more graspable objects does not meet the second confidence condition, the position and/or posture of the objects in the predetermined area are adjusted, and then the two-dimensional image acquisition step is returned.
  • an object positioning adjustment step wherein, after the target range determination step, when the target range does not meet the first confidence condition, and/or, after the three-dimensional point cloud generation step, when the point cloud of one or more graspable objects does not meet the second confidence condition, the position and/or posture of the objects in the predetermined area are adjusted, and then the two-dimensional image acquisition step is returned.
  • the first confidence condition may be that the confidence of the target range is greater than 0.99; and/or the second confidence condition may be that the confidence of the point cloud is greater than 0.99. Performing recognition at a sufficiently high confidence level is conducive to accurately grasping objects.
  • the three-dimensional point cloud generation step may further include optimizing the point cloud using an interpolation algorithm.
  • the interpolation algorithm may improve the accuracy of the point cloud, thereby providing more accurate positioning information for the robot.
  • the positioning information determining step may further include selecting the at least one target from the one or more graspable objects based on the confidence of the point cloud of the one or more graspable objects when the point cloud of the one or more graspable objects meets the second confidence condition.
  • selecting the at least one target from the one or more graspable objects based on the confidence of the point cloud of the one or more graspable objects when the point cloud of the one or more graspable objects meets the second confidence condition.
  • the automatic grasping system includes a robot for automatically grasping items in a predetermined area.
  • the automatic grasping system also includes: a model building module, which is configured to establish an AI model for identifying graspable items in a predetermined area; a two-dimensional camera, which is configured to collect a two-dimensional image of a predetermined area; a target range determination module, which is configured to determine a target range corresponding to one or more graspable items in a two-dimensional image based on the AI model; a three-dimensional camera, which is configured to collect a three-dimensional image of the target range when the target range meets a first confidence condition; a point cloud generation module, which is configured to generate a point cloud of one or more graspable items based on the three-dimensional image collected by the three-dimensional camera; and a positioning information determination module, which is configured to determine the positioning information about the position and posture of at least one target item among the one or more graspable items based on the corresponding point cloud when
  • FIG. 4 shows a schematic diagram of a point cloud of a self-automatic grasping method according to an exemplary embodiment of the present invention.
  • an automatic grasping method for a robot is provided. Specifically, the automatic grasping method is used to identify graspable objects from a predetermined area where one or more objects are placed, so as to control the robot to automatically grasp the objects from the predetermined area.
  • Fig. 1 shows a flow chart of an automatic grasping method according to an exemplary embodiment of the present invention.
  • the automatic grasping method is described below in conjunction with Fig. 1.
  • the automatic grasping method mainly includes steps S1 to S6.
  • the model building step S1 is first performed.
  • the content of the model building step S1 is to build an AI (artificial intelligence) model for identifying graspable items in the predetermined area 1.
  • the AI model is the basis for performing two-dimensional image recognition, which can be built based on a deep learning neural network algorithm.
  • the AI model can be stored as a software program in a processor of a computer, especially a graphics processing unit (GPU).
  • the AI model affects the accuracy of image recognition.
  • an original AI model can be preliminarily established first, and then the original AI model can be trained to obtain an improved AI model.
  • training the AI model includes the following steps. First, in step S11, a predetermined number of training two-dimensional images are provided, wherein each training two-dimensional image includes multiple items.
  • the training two-dimensional images can be collected on-site, or can also come from a stored database.
  • the number of training two-dimensional images can be set as needed, for example, 600, or more or less.
  • Each training two-dimensional image includes one or more graspable items, that is, items placed on the top layer and can therefore be directly grasped by the robot.
  • these training two-dimensional images can include items of various sizes and/or shapes and/or distributed in different ways.
  • the operator manually marks (for example, via a human-machine interaction tool such as a touch screen) graspable items in each training two-dimensional image.
  • the AI model thus knows which items in each training two-dimensional image are graspable.
  • the AI model is verified using a verification set.
  • the verification process includes determining the accuracy and loss rate of the AI model.
  • the verified AI model can be used as a complete model for subsequent image processing and object recognition.
  • the two-dimensional image acquisition step S2 is performed.
  • the content of the two-dimensional image acquisition step S2 is to collect a two-dimensional image of the predetermined area 1.
  • One or more items 2 are placed in the predetermined area 1. These items 2 may have the same or different shapes and/or sizes, and may be distributed in the predetermined area 1 in any manner (for example, in a discrete manner, an overlapping manner, or a combination thereof).
  • the acquired two-dimensional image of the predetermined area 1 contains image information of the items 2 therein.
  • An imaging device such as a two-dimensional camera can be used to collect a two-dimensional image of the predetermined area 1.
  • the two-dimensional camera is arranged above the predetermined area 1 and the items 2 therein.
  • a reverse light source arranged below the item 2 and an enhanced light source arranged above the item 2 can be used to provide lighting.
  • the enhanced light source can preferably be integrated with the two-dimensional camera.
  • the target range determination step S3 includes determining the target range corresponding to one or more graspable items 2a in the acquired two-dimensional image based on the AI model. That is, the one or more graspable items 2a and the corresponding target range are identified in the acquired two-dimensional image by the AI model.
  • the target range here is not necessarily the exact graphic range that corresponds to the identified graspable items 2a, but can be slightly larger than the target range.
  • the target range can be a continuous area or a set of discrete areas consisting of multiple discrete graspable items.
  • multiple discrete graspable items 2a and multiple discrete areas containing these graspable items 2a are identified in the two-dimensional image of the predetermined area 1, and these discrete areas together constitute the target range 1a.
  • the target area of the subsequent three-dimensional image processing is the target range 1a determined here, rather than the entire predetermined area 1.
  • the automatic grasping method will make a confidence judgment on the target range 1a obtained in step S3, and execute the three-dimensional point cloud generation step S4 when the target range 1a meets the first confidence condition.
  • the first confidence condition is usually that the confidence of the target range 1a reaches a predetermined numerical range, for example, greater than 0.99, which means that the recognition result of the two-dimensional image has a higher accuracy.
  • the three-dimensional point cloud generation step S4 includes collecting a three-dimensional image of the target range 1a, and thereby generating a point cloud corresponding to the graspable object 2a determined in step S3.
  • the acquisition of the three-dimensional image can be achieved, for example, by a three-dimensional camera, which can be arranged above the predetermined area 1 and the object 2 therein, and preferably can be integrated with a two-dimensional camera.
  • a three-dimensional camera When using a three-dimensional camera to collect a three-dimensional image, lighting can be provided by a structured light source.
  • the structured light source can preferably be turned on only when the three-dimensional camera collects a three-dimensional image and turned off when the two-dimensional camera collects a two-dimensional image.
  • FIG4 shows a schematic diagram of a point cloud of a plurality of graspable items 2a generated in step S4.
  • the point cloud generated based on the three-dimensional image represents a collection of surface points of the graspable item 2a. Since the resolution of the initially generated point cloud is limited, preferably, the three-dimensional point cloud generation step S4 may also include optimizing the point cloud initially generated based on the three-dimensional image using an interpolation algorithm to improve the accuracy of the contour and feature recognition of the point cloud. The optimized point cloud may be used to perform subsequent steps.
  • the automatic grasping method will make a confidence judgment on the point cloud obtained in step S4, and execute the positioning information determination step S5 when the point cloud of the graspable objects 2a obtained in step S4 meets the second confidence condition.
  • the second confidence condition is usually that the confidence of the point cloud of the graspable objects 2a reaches a predetermined numerical range, such as greater than 0.99, which means that the recognition result of the three-dimensional image has a higher accuracy.
  • the positioning information determination step S5 includes determining the positioning information of at least one target object based on the point cloud.
  • the target object is an object to be grasped selected from the above-mentioned graspable objects 2a.
  • the determined target object The number of target objects can be singular (grabbing only one object at a time) or plural (grabbing multiple objects at a time), which can be determined according to a predetermined number and is not greater than the determined number of graspable objects 2a.
  • the target objects can be selected based on the confidence of the point cloud of these graspable objects 2a.
  • the object with the highest confidence can be selected from these graspable objects 2a according to the predetermined single-grab number in order of confidence from high to low as the target object; when the number of graspable objects 2a is less than or equal to the predetermined single-grab number, all graspable objects 2a can be directly selected as target objects.
  • the robot grasping step S6 is executed.
  • the content of the robot grasping step S6 is that the robot grasps the corresponding target object from the predetermined area 1 according to the received positioning information.
  • the robot can perform subsequent operations on the grasped target object, such as placing it in a designated area or processing it.
  • the automatic grasping method may further include an object positioning adjustment step S7, so as to change the position and/or posture of the object 2 in the predetermined area 1 when there is a lack of accurately identified graspable objects 2a.
  • the object positioning adjustment step S7 includes: after the target range determination step S3, when the target range 1a does not meet the first confidence condition, and/or, after the three-dimensional point cloud generation step S4, when the point cloud of one or more graspable objects 2a obtained does not meet the second confidence condition, adjusting the position and/or posture of the object in the predetermined area 1, and then returning to the two-dimensional image acquisition step S2.
  • the position and/or posture of the objects in the predetermined area 1 can be adjusted, for example, by a flexible feeding device.
  • the predetermined area 1 can be arranged on the flexible feeding device, and the objects in the predetermined area 1 are thus carried by the flexible feeding device.
  • the flexible feeding device can change the position and/or posture of the objects in the predetermined area 1 by shaking.
  • an automatic grasping system is also provided, which is used to correspondingly execute the above-mentioned automatic grasping method according to the present invention.
  • the automatic grasping system includes a model building module, a two-dimensional camera, a target range determination module, a three-dimensional camera, a point cloud generation module, a positioning information determination module and a robot.
  • the model building module is used to correspondingly execute the model building step S1 of the above-mentioned automatic grasping method.
  • the model building module is configured to establish an AI model for identifying graspable items in the predetermined area 1.
  • the model building module can train the AI model.
  • the process of establishing and training the AI model by the model building module is as described in step S1 of the automatic grasping method, and will not be repeated here.
  • the two-dimensional camera is used to correspondingly perform the two-dimensional image acquisition step S2 of the above-mentioned automatic grasping method.
  • the two-dimensional camera is configured to acquire a two-dimensional image of a predetermined area 1.
  • the automatic grasping system may also preferably include a reverse light source for providing reverse light to the two-dimensional camera and an enhanced light source for providing enhanced light to the two-dimensional camera.
  • the process of the two-dimensional camera acquiring a two-dimensional image is as described in step S2 of the automatic grasping method, and will not be repeated here.
  • the model building module, the target range determination module, the point cloud generation module and the positioning information determination module can be integrated into the processor of the computer as software programs.
  • the automatic grasping method and automatic grasping system of the present invention combine two-dimensional vision technology and three-dimensional vision technology, first use two-dimensional vision technology to determine the target area, and then generate a point cloud for the specified target area.
  • the automatic grasping method and automatic grasping system of the present invention significantly reduce the object area of three-dimensional image processing, thereby shortening the generation time of the point cloud, reducing the difficulty of target recognition, and improving the grasping accuracy.
  • This automatic grasping method and automatic grasping system can especially improve the recognition and grasping efficiency of small-sized and thin-thick objects.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Manipulator (AREA)

Abstract

一种自动抓取方法和自动抓取系统。该自动抓取方法包括：建立用于识别预定区域（1）中的可抓取物品（2a）的AI模型；采集预定区域（1）的二维图像；基于AI模型在二维图像中确定对应于一个或多个可抓取物品（2a）的目标范围（1a）；当目标范围（1a）满足第一置信度条件时，采集目标范围（1a）的三维图像并且生成一个或多个可抓取物品（2a）的点云；当该一个或多个可抓取物品（2a）的点云满足第二置信度条件时，基于相应的点云确定关于该一个或多个可抓取物品（2a）中的至少一个目标物品的位置和姿态的定位信息，并且将定位信息发送到机器人；机器人根据定位信息从预定区域抓取该至少一个目标物品。该自动抓取方法和自动抓取系统可以改善物品识别效果。

Description

自动抓取方法和自动抓取系统 技术领域
本发明涉及机器人技术领域。具体地，本发明涉及一种用于机器人的自动抓取方法和自动抓取系统。
背景技术
当前，机器人越来越多地应用在流水线生产和货物分拣等场景中。机器人可以识别预定区域中的物品并且精准地抓取所识别的物品以进行进一步的操作(例如摆放、加工等)。目前常用的自动抓取系统通常由计算机、三维图像采集组件、机器人操作系统、机器视觉软件和算法、AI(人工智能，Artificial Intelligence)模型以及机器人路径规划软件等组成。这种自动抓取系统是基于三维视觉解决方案，即通过三维图像来识别预定区域中的可抓取物品。但是，对于厚度较小的物品，由于视野区域较大，生成三维点云需要较长的时间；而对于尺寸较小的物品，由于点云场景的面积相对较大，因此难以准确地从预定区域中识别可抓取的物品，从而导致识别率较低。
发明内容
因此，本发明需要解决的技术问题是，提供一种能够改善物品识别效果的自动抓取方法和自动抓取系统。
上述技术问题通过根据本发明的一种自动抓取方法而得到解决。该自动抓取方法用于控制机器人来自动抓取预定区域中的物品。该自动抓取方法包括：模型建立步骤，其中，建立用于识别预定区域中的可抓取物品的AI模型；二维图像采集步骤，其中，采集预定区域的二维图像；目标范围确定步骤，其中，基于AI模型在二维图像中确定对应于一个或多个可抓取物品的目标范围；三维点云生成步骤，其中，当目标范围满足第一置信 度条件时，采集目标范围的三维图像并且生成该一个或多个可抓取物品的点云；定位信息确定步骤，其中，当该一个或多个可抓取物品的点云满足第二置信度条件时，基于相应的点云确定关于该一个或多个可抓取物品中的至少一个目标物品的位置和姿态的定位信息；和机器人抓取步骤，其中，机器人根据定位信息从预定区域抓取该至少一个目标物品。在该抓取方法中，首先在二维图像中识别可抓取物品并确定相应的目标范围，然后在三维图像中仅对目标范围内的可抓取物品进行进一步的识别。由此将三维图像处理的对象由整个预定区域缩小到二维图像处理得到的目标范围，从而可以提高物品识别的速度和精确度。
根据本发明的一个优选实施例，模型建立步骤可以包括对AI模型进行训练，对AI模型进行训练包括：提供预定数量的训练二维图像，其中，每个训练二维图像包括多个物品；人工标记每个训练二维图像中的可抓取物品；和使用验证集来验证AI模型。通过训练可以提高AI模型的识别精确度。
根据本发明的另一优选实施例，该自动抓取方法还可以包括物品定位调整步骤，其中，在目标范围确定步骤之后当目标范围不满足第一置信度条件时，和/或，在三维点云生成步骤之后当一个或多个可抓取物品的点云不满足第二置信度条件时，调整预定区域中的物品的位置和/或姿态，然后返回二维图像采集步骤。由此可以在预定区域中不存在能够精确识别的可抓取物品时重新分布物品并重新开始识别可抓取物品。
根据本发明的另一优选实施例，第一置信度条件可以为目标范围的置信度大于0.99；和/或第二置信度条件可以为点云的置信度大于0.99。在足够高的置信度下执行识别有利于准确抓取物品。
根据本发明的另一优选实施例，三维点云生成步骤还可以包括使用插值算法对点云进行优化。通过插值算法可以提高点云的精度，从而为机器人提供更准确的定位信息。
根据本发明的另一优选实施例，定位信息确定步骤还可以包括当一个或多个可抓取物品的点云满足第二置信度条件时，基于该一个或多个可抓取物品的点云的置信度从该一个或多个可抓取物品中选择该至少一个目标 物品。由此可以在识别出多个可抓取物品时基于置信度选出预定数量的物品来进行抓取。
上述技术问题还通过根据本发明的一种自动抓取系统而得到解决。该自动抓取系统包括用于自动抓取预定区域中的物品的机器人。该自动抓取系统还包括：模型建立模块，其配置为建立用于识别预定区域中的可抓取物品的AI模型；二维相机，其配置为采集预定区域的二维图像；目标范围确定模块，其配置为基于AI模型在二维图像中确定对应于一个或多个可抓取物品的目标范围；三维相机，其配置为当目标范围满足第一置信度条件时，采集目标范围的三维图像；点云生成模块，其配置为基于三维相机采集的三维图像生成一个或多个可抓取物品的点云；和定位信息确定模块，其配置为当该一个或多个可抓取物品的点云满足第二置信度条件时，基于相应的点云确定关于该一个或多个可抓取物品中的至少一个目标物品的位置和姿态的定位信息；其中，机器人配置为根据定位信息从预定区域抓取该至少一个目标物品。该自动抓取系统首先在二维图像中识别可抓取物品并确定相应的目标范围，然后在三维图像中仅对目标范围内的可抓取物品进行进一步的识别。由此将三维图像处理的对象由整个预定区域缩小到二维图像处理得到的目标范围，从而可以提高物品识别的速度和精确度。
根据本发明的一个优选实施例，该自动抓取系统还可以包括柔性进给装置，柔性进给装置配置为承载预定区域中的物品并且在目标范围不满足第一置信度条件和/或该一个或多个可抓取物品的点云不满足第二置信度条件时调整预定区域中的物品的位置和/或姿态。预定区域可以设置在柔性进给装置的承载表面上，柔性进给装置可以晃动承载表面以使预定区域中的物品改变位置和/或姿态。
根据本发明的另一优选实施例，该自动抓取系统还可以包括用于预定区域中的物品的反向光源和增强光源，反向光源可以布置在柔性进给装置中。反向光源和增强光源用于在二维相机采集二维图像时提供合适的光照条件。
根据本发明的另一优选实施例，模型建立模块、目标范围确定模块、 点云生成模块和定位信息确定模块可以集成在计算机的处理器中。处理器可以通过有线或无线的方式与其他硬件装置通信连接，以便发送或接收相关信号。
附图说明
以下结合附图进一步描述本发明。图中以相同的附图标记来代表功能相同的元件。其中：
图1示出根据本发明的示例性实施例的自动抓取方法的流程图；
图2示出根据本发明的示例性实施例的模型训练的流程图；
图3示出根据本发明的示例性实施例的自动抓取方法的目标范围的示意图；和
图4示出根据本发明的示例性实施例的自自动抓取方法的点云的示意图。
具体实施方式
以下将结合附图描述根据本发明的自动抓取方法和自动抓取系统的具体实施方式。下面的详细描述和附图用于示例性地说明本发明的原理，本发明不限于所描述的优选实施例，本发明的保护范围由权利要求书限定。
根据本发明的实施例，提供了一种用于机器人的自动抓取方法。具体而言，这种自动抓取方法用于从放置有一个或多个物品的预定区域中识别出可抓取的物品，以便控制机器人来从预定区域中自动抓取物品。
图1示出根据本发明的示例性实施例的自动抓取方法的流程图。下面结合图1来说明该自动抓取方法。如图1所示，该自动抓取方法主要包括步骤S1至S6。
在该自动抓取方法中，首先执行模型建立步骤S1。模型建立步骤S1的内容是建立用于识别预定区域1中的可抓取物品的AI(人工智能)模型。AI模型是执行二维图像识别的基础，其可以基于深度学习神经网络算法来建立。AI模型可以作为软件程序存储在计算机的处理器、特别是图形处理器(GPU)中。AI模型影响图像识别的精确性。
优选地，在模型建立步骤S1中，可以首先初步建立一个原始的AI模型，然后对原始的AI模型进行训练，以便得到改善的AI模型。具体而言，如图2所示，对AI模型进行训练包括以下步骤。首先，在步骤S11，提供预定数量的训练二维图像，其中，每个训练二维图像包括多个物品。训练二维图像可以现场采集，或者也可以来自于存储的数据库。训练二维图像的数量可以根据需要来设定，例如600张，也可以更多或更少。每个训练二维图像都包括一个或多个可抓取的物品，即放置在顶层并且因此可以由机器人直接抓取的物品。优选地，这些训练二维图像可以包括各种不同尺寸的和/或不同形状的和/或以不同方式分布的物品。接下来，在步骤S12，由操作人员在每个训练二维图像中人工标记出(例如，经由触屏等人机交互工具)可抓取的物品。AI模型由此得知每个训练二维图像中的哪些物品是可抓取的。最后，在步骤S13，使用验证集来验证AI模型。验证的过程包括确定AI模型的准确性和损失率。通过验证的AI模型可以作为完善的模型来应用于后续的图像处理和物品识别。
如图1所示，在获得了所需的AI模型之后，执行二维图像采集步骤S2。二维图像采集步骤S2的内容是采集预定区域1的二维图像。预定区域1中放置有一个或多个物品2。这些物品2可以具有相同或不同的形状和/或尺寸，并且可以以任意方式分布(例如，以离散方式、重叠方式或其组合的方式)在预定区域1中。所采集的预定区域1的二维图像包含了其中的物品2的图像信息。可以使用例如二维相机等成像装置来采集预定区域1的二维图像。二维相机布置在预定区域1及其中物品2的上方。在使用二维相机采集二维图像时，可以使用布置在物品2下方的反向光源以及布置在物品2上方的增强光源来提供照明。增强光源优选地可以与二维相机集成在一起。
在获得二维图像之后，执行目标范围确定步骤S3。目标范围确定步骤S3包括基于AI模型在所采集的二维图像中确定对应于一个或多个可抓取物品2a的目标范围。也就是说，通过AI模型在所采集的二维图像中识别出一个或多个可抓取物品2a以及相应的目标范围。这里的目标范围未必是精确对应于所识别出的可抓取物品2a的准确图形范围，而是可以略大于所 识别出的可抓取物品2a的实际覆盖范围的区域。目标范围可以是一个连续的区域，也可以是由多个离散的可抓取物品组成的离散区域集合。例如，如图3所示，在预定区域1的二维图像中识别出了多个离散的可抓取物品2a以及分别包含这些可抓取物品2a的多个离散区域，由这些离散区域共同构成了目标范围1a。后续的三维图像处理的目标区域是这里确定的目标范围1a，而非整个预定区域1。
接下来，该自动抓取方法将对步骤S3中获得的目标范围1a进行置信度判断，并且在目标范围1a满足第一置信度条件时执行三维点云生成步骤S4。第一置信度条件通常是，目标范围1a的置信度达到预定的数值范围，例如大于0.99，这意味二维图像的识别结果具有较高的精确度。三维点云生成步骤S4包括采集目标范围1a的三维图像，并且由此生成对应于步骤S3中确定的可抓取物品2a的点云。三维图像的采集例如可以通过三维相机来实现，三维相机可以布置在预定区域1及其中物品2的上方，并且优选地可以与二维相机集成在一起。在使用三维相机采集三维图像时，可以通过结构光源来提供照明。结构光源优选地可以仅在三维相机采集三维图像时开启并且在二维相机采集二维图像时关闭。
图4示出了在步骤S4中生成的多个可抓取物品2a的点云的示意图。如图4所示，根据三维图像生成的点云代表可抓取物品2a的表面点的集合。由于初始生成的点云的分辨率是有限的，优选地，三维点云生成步骤S4还可以包括使用插值算法对基于三维图像初步生成的点云进行优化，以便提高点云的轮廓和特征识别的准确性。优化后的点云可以用于进行后续步骤。
然后，该自动抓取方法将对步骤S4中获得的点云进行置信度判断，并且在步骤S4中得到的这些可抓取物品2a的点云满足第二置信度条件时执行定位信息确定步骤S5。类似于第一置信度条件，第二置信度条件通常是，这些可抓取物品2a的点云的置信度达到预定的数值范围，例如大于0.99，这意味三维图像的识别结果具有较高的精确度。定位信息确定步骤S5包括基于点云确定至少一个目标物品的定位信息。目标物品是从上述的可抓取物品2a中选择的待抓取的物品。每次执行步骤S5时，所确定的目 标物品的数量可以是单数(一次仅抓取一个物品)或复数(一次抓取多个物品)，其可以根据预定数量来确定并且不大于所确定的可抓取物品2a的数量。优选地，可以基于这些可抓取物品2a的点云的置信度来选择目标物品。具体地，当可抓取物品2a的数量大于预定的单次抓取数量时，可以按照置信度由高至低的顺序从这些可抓取物品2a中按照预定的单次抓取数量选择具有最高置信度的物品作为目标物品；当可抓取物品2a的数量小于或等于预定的单次抓取数量时，可以直接将所有可抓取物品2a作为目标物品。
基于对应于目标物品的点云得到目标物品的定位信息。目标物品的定位信息包括目标物品在预定区域1中的位置和姿态。目标物品在预定区域1中的位置和姿态可以以三维坐标数据的形式来表达。该过程可以基于PCL(点云图书馆)点云处理和坐标生成算法来进行。所确定的目标物品的定位信息将发送至机器人。
在机器人接收到目标物品的定位信息之后，执行机器人抓取步骤S6。机器人抓取步骤S6的内容是，机器人根据所接收的定位信息从预定区域1抓取相应的目标物品。机器人可以对抓取的目标物品进行后续操作，例如摆放到指定区域或者进行加工等。
如图1所示，优选地，该自动抓取方法可以另外包括物品定位调整步骤S7，以便在缺少精确识别的可抓取物品2a时改变物品2预定区域1中的位置和/或姿态。物品定位调整步骤S7包括：在目标范围确定步骤S3之后当目标范围1a不满足第一置信度条件时，和/或，在三维点云生成步骤S4之后当所获得的一个或多个可抓取物品2a的点云不满足第二置信度条件时，调整预定区域1中的物品的位置和/或姿态，然后返回二维图像采集步骤S2。
优选地，预定区域1中的物品的位置和/或姿态的调整例如可以通过柔性进给装置来实现。具体而言，预定区域1可以布置在柔性进给装置上，预定区域1中的物品由此由柔性进给装置承载。柔性进给装置可以通过晃动来改变物品在预定区域1中的位置和/或姿态。在物品定位调整步骤S7之后，预定区域1中的物品重新分布，自动抓取方法将从二维图像采集步 骤S2开始重新执行。
根据本发明的实施例，还提供了一种自动抓取系统，该自动抓取系统用于对应地执行根据本发明的上述自动抓取方法。该自动抓取系统包括模型建立模块、二维相机、目标范围确定模块、三维相机、点云生成模块、定位信息确定模块和机器人。
模型建立模块用于对应地执行上述自动抓取方法的模型建立步骤S1。具体地，模型建立模块配置为建立用于识别预定区域1中的可抓取物品的AI模型。在建立AI模型的过程中，模型建立模块可以对于AI模型进行训练。模型建立模块建立和训练AI模型的过程如自动抓取方法的步骤S1所述，在此不再赘述。
二维相机用于对应地执行上述自动抓取方法的二维图像采集步骤S2。具体地，二维相机配置为采集预定区域1的二维图像。该自动抓取系统还可以优选地包括用于为二维相机提供反向光的反向光源以及用于为二维相机提供增强光的增强光源。二维相机采集二维图像的过程如自动抓取方法的步骤S2所述，在此不再赘述。
目标范围确定模块用于对应地执行上述自动抓取方法的目标范围确定步骤S3。目标范围确定模块配置为基于AI模型在二维图像中确定对应于一个或多个可抓取物品2a的目标范围1a。目标范围确定模块确定目标范围1a的过程如自动抓取方法的步骤S3所述，在此不再赘述。
三维相机和点云生成模块用于对应地执行上述自动抓取方法的三维点云生成步骤S4。三维相机配置为当目标范围1a满足第一置信度条件时，采集目标范围1a的三维图像，而点云生成模块配置为基于三维相机采集的三维图像生成上述一个或多个可抓取物品2a的点云。优选地，三维相机和二维相机可以集成在一起。该自动抓取系统还可以优选地包括用于为三维相机提供结构光的结构光源。三维相机和点云生成模块的具体工作过程如自动抓取方法的步骤S4所述，在此不再赘述。
定位信息确定模块用于对应地执行上述自动抓取方法的定位信息确定步骤S5。定位信息确定模块配置为当上述的一个或多个可抓取物品2a的点云满足第二置信度条件时，基于相应的点云确定关于这些可抓取物品2a 中的至少一个目标物品的位置和姿态的定位信息。定位信息确定模块确定定位信息的过程如自动抓取方法的步骤S5所述，在此不再赘述。
机器人用于对应地执行上述自动抓取方法的机器人抓取步骤S6。机器人获取定位信息确定模块提供的目标物品的定位信息并且根据定位信息从预定区域1抓取所确定的目标物品。
优选地，自动抓取系统还可以包括柔性进给装置。如自动抓取方法的实施例中所述，柔性进给装置配置为承载预定区域1中的物品并且在目标范围不满足第一置信度条件和/或上述的一个或多个可抓取物品2a的点云不满足第二置信度条件时调整预定区域1中的物品的位置和/或姿态。优选地，用于二维相机的反向光源可以布置在柔性进给装置中。
在上述自动抓取系统中，模型建立模块、目标范围确定模块、点云生成模块和定位信息确定模块可以作为软件程序集成在计算机的处理器中。
本发明的自动抓取方法和自动抓取系统将二维视觉技术和三维视觉技术结合起来，首先利用二维视觉技术确定目标区域，然后针对指定的目标区域生成点云。相较于直接对整个场景生成点云的现有技术，本发明的自动抓取方法和自动抓取系统显著缩小了三维图像处理的对象区域，由此缩短了点云的生成时间，降低了目标识别的难度，并且提高了抓取精度。这种自动抓取方法和自动抓取系统尤其可以提高小尺寸、小厚度的物品的识别和抓取效率。
虽然在上述说明中示例性地描述了可能的实施例，但是应当理解到，仍然通过所有已知的和此外技术人员容易想到的技术特征和实施方式的组合存在大量实施例的变化。此外还应该理解到，示例性的实施方式仅仅作为一个例子，这种实施例绝不以任何形式限制本发明的保护范围、应用和构造。通过前述说明更多地是向技术人员提供一种用于转化至少一个示例性实施方式的技术指导，其中，只要不脱离权利要求书的保护范围，便可以进行各种改变，尤其是关于所述部件的功能和结构方面的改变。
附图标记表
1   预定区域
1a  目标范围
2   物品
2a  可抓取物品

Claims (10)

  1. 一种自动抓取方法，用于控制机器人来自动抓取预定区域(1)中的物品，其特征在于，所述自动抓取方法包括：
    模型建立步骤(S1)，其中，建立用于识别所述预定区域(1)中的可抓取物品(2a)的AI模型；
    二维图像采集步骤(S2)，其中，采集所述预定区域(1)的二维图像；
    目标范围确定步骤(S3)，其中，基于所述AI模型在所述二维图像中确定对应于一个或多个可抓取物品(2a)的目标范围(1a)；
    三维点云生成步骤(S4)，其中，当所述目标范围(1a)满足第一置信度条件时，采集所述目标范围(1a)的三维图像并且生成所述一个或多个可抓取物品(2a)的点云；
    定位信息确定步骤(S5)，其中，当所述一个或多个可抓取物品(2a)的点云满足第二置信度条件时，基于相应的点云确定关于所述一个或多个可抓取物品(2a)中的至少一个目标物品的位置和姿态的定位信息；和
    机器人抓取步骤(S6)，其中，所述机器人根据所述定位信息从所述预定区域(1)抓取所述至少一个目标物品。
  2. 根据权利要求1所述的自动抓取方法，其特征在于，所述模型建立步骤(S1)包括对所述AI模型进行训练，对所述AI模型进行训练包括：
    提供预定数量的训练二维图像，其中，每个训练二维图像包括多个物品；
    人工标记每个训练二维图像中的可抓取物品(2a)；和
    使用验证集来验证所述AI模型。
  3. 根据权利要求1所述的自动抓取方法，其特征在于，所述自动抓取方法还包括物品定位调整步骤(S7)，其中，在所述目标范围确定步骤(S3)之后当所述目标范围(1a)不满足所述第一置信度条件时，和/或， 在所述三维点云生成步骤(S4)之后当所述一个或多个可抓取物品(2a)的点云不满足所述第二置信度条件时，调整所述预定区域(1)中的物品的位置和/或姿态，然后返回所述二维图像采集步骤(S2)。
  4. 根据权利要求1所述的自动抓取方法，其特征在于，
    所述第一置信度条件为所述目标范围(1a)的置信度大于0.99；和/或
    所述第二置信度条件为所述点云的置信度大于0.99。
  5. 根据权利要求1所述的自动抓取方法，其特征在于，所述三维点云生成步骤(S4)还包括使用插值算法对所述点云进行优化。
  6. 根据权利要求1至5中任一项所述的，其特征在于，所述定位信息确定步骤(S5)还包括当所述一个或多个可抓取物品(2a)的点云满足第二置信度条件时，基于所述一个或多个可抓取物品(2a)的点云的置信度从所述一个或多个可抓取物品(2a)中选择所述至少一个目标物品。
  7. 一种自动抓取系统，包括用于自动抓取预定区域(1)中的物品的机器人，其特征在于，所述自动抓取系统还包括：
    模型建立模块，其配置为建立用于识别所述预定区域(1)中的可抓取物品(2a)的AI模型；
    二维相机，其配置为采集所述预定区域(1)的二维图像；
    目标范围确定模块，其配置为基于所述AI模型在所述二维图像中确定对应于一个或多个可抓取物品(2a)的目标范围(1a)；
    三维相机，其配置为当所述目标范围(1a)满足第一置信度条件时，采集所述目标范围(1a)的三维图像；
    点云生成模块，其配置为基于所述三维相机采集的所述三维图像生成所述一个或多个可抓取物品(2a)的点云；和
    定位信息确定模块，其配置为当所述一个或多个可抓取物品(2a)的点云满足第二置信度条件时，基于相应的点云确定关于所述一个或多个可抓取物品(2a)中的至少一个目标物品的位置和姿态的定位信息；
    其中，所述机器人配置为根据所述定位信息从所述预定区域(1)抓取所述至少一个目标物品。
  8. 根据权利要求7所述的自动抓取系统，其特征在于，所述自动抓取 系统还包括柔性进给装置，所述柔性进给装置配置为承载所述预定区域(1)中的物品并且在所述目标范围(1a)不满足所述第一置信度条件和/或所述一个或多个可抓取物品(2a)的点云不满足所述第二置信度条件时调整所述预定区域(1)中的物品的位置和/或姿态。
  9. 根据权利要求8所述的自动抓取系统，其特征在于，所述自动抓取系统还包括用于所述预定区域(1)中的物品的反向光源和增强光源，所述反向光源布置在所述柔性进给装置中。
  10. 根据权利要求7至9中任一项所述的自动抓取系统，其特征在于，所述模型建立模块、所述目标范围确定模块、所述点云生成模块和所述定位信息确定模块集成在计算机的处理器中。
PCT/CN2023/141165 2023-12-22 2023-12-22 自动抓取方法和自动抓取系统 Pending WO2025129666A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2023/141165 WO2025129666A1 (zh) 2023-12-22 2023-12-22 自动抓取方法和自动抓取系统

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2023/141165 WO2025129666A1 (zh) 2023-12-22 2023-12-22 自动抓取方法和自动抓取系统

Publications (1)

Publication Number Publication Date
WO2025129666A1 true WO2025129666A1 (zh) 2025-06-26

Family

ID=96136234

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/141165 Pending WO2025129666A1 (zh) 2023-12-22 2023-12-22 自动抓取方法和自动抓取系统

Country Status (1)

Country Link
WO (1) WO2025129666A1 (zh)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102019122790A1 (de) * 2018-08-24 2020-02-27 Nvidia Corp. Robotersteuerungssystem
CN111178250A (zh) * 2019-12-27 2020-05-19 深圳市越疆科技有限公司 物体识别定位方法、装置及终端设备
CN114758337A (zh) * 2022-06-16 2022-07-15 山东海量信息技术研究院 一种语义实例重建方法、装置、设备及介质
CN114952809A (zh) * 2022-06-24 2022-08-30 中国科学院宁波材料技术与工程研究所 工件识别和位姿检测方法、系统及机械臂的抓取控制方法
CN115713514A (zh) * 2022-11-17 2023-02-24 北京有竹居网络技术有限公司 一种对象抓取方法、装置、计算机设备及存储介质
EP4155036A1 (en) * 2021-09-24 2023-03-29 Toyota Jidosha Kabushiki Kaisha A method for controlling a grasping robot through a learning phase and a grasping phase
WO2023068929A1 (en) * 2021-10-19 2023-04-27 Fizyr B.V. Automated bin-picking based on deep learning
CN117036470A (zh) * 2023-06-28 2023-11-10 南京信息工程大学 一种抓取机器人的物体识别与位姿估计方法

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102019122790A1 (de) * 2018-08-24 2020-02-27 Nvidia Corp. Robotersteuerungssystem
CN111178250A (zh) * 2019-12-27 2020-05-19 深圳市越疆科技有限公司 物体识别定位方法、装置及终端设备
EP4155036A1 (en) * 2021-09-24 2023-03-29 Toyota Jidosha Kabushiki Kaisha A method for controlling a grasping robot through a learning phase and a grasping phase
WO2023068929A1 (en) * 2021-10-19 2023-04-27 Fizyr B.V. Automated bin-picking based on deep learning
CN114758337A (zh) * 2022-06-16 2022-07-15 山东海量信息技术研究院 一种语义实例重建方法、装置、设备及介质
CN114952809A (zh) * 2022-06-24 2022-08-30 中国科学院宁波材料技术与工程研究所 工件识别和位姿检测方法、系统及机械臂的抓取控制方法
CN115713514A (zh) * 2022-11-17 2023-02-24 北京有竹居网络技术有限公司 一种对象抓取方法、装置、计算机设备及存储介质
CN117036470A (zh) * 2023-06-28 2023-11-10 南京信息工程大学 一种抓取机器人的物体识别与位姿估计方法

Similar Documents

Publication Publication Date Title
CN111462154B (zh) 基于深度视觉传感器的目标定位方法、装置及自动抓取机器人
CN111679291A (zh) 基于三维激光雷达的巡检机器人目标定位配置方法
CN114630734B (zh) 具有专用硬件加速的支持机器学习的视觉伺服
JP5743499B2 (ja) 画像生成装置、画像生成方法、およびプログラム
CN113469108B (zh) 一种基于边缘计算的智能增强现实装配辅助方法及系统
CN115070781B (zh) 一种物体抓取方法及双机械臂协作系统
CN112347837B (zh) 图像处理系统
JP7376318B2 (ja) アノテーション装置
CN109079777B (zh) 一种机械臂手眼协调作业系统
CN114419437A (zh) 基于2d视觉的工件分拣系统及其控制方法和控制装置
CN110942083B (zh) 拍摄装置以及拍摄系统
CN112947458A (zh) 基于多模态信息的机器人准确抓取方法及计算机可读介质
CN114463244A (zh) 一种视觉机器人抓取系统及其控制方法
WO2025000778A1 (zh) 一种试管的抓取控制方法及装置
CN120306273A (zh) 一种基于机械臂视觉抓取的垃圾智能分拣系统及方法
CN117021099B (zh) 一种面向任意物体基于深度学习和图像处理的人机交互方法
JP2021094085A (ja) 制御システム及び制御プログラム
CN114419451A (zh) 电梯内外识别方法、装置、电子设备及存储介质
CN109934155B (zh) 一种基于深度视觉的协作机器人手势识别方法及装置
CN119515972A (zh) 运动机构轨迹的生成与位置校准和装置
JP2025037479A (ja) 物体の姿勢を推定する方法、ロボットが物体を把持する方法、制御装置、およびロボットシステム
JPH05150835A (ja) ロボツトによる組み立て装置
CN114358187B (zh) 一种多物体识别及分拣系统和方法
CN117850639A (zh) 视觉引导定位系统的配置管理方法及系统
CN118894277B (zh) 贴标机用自适应试管定位参数协同调控系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23961988

Country of ref document: EP

Kind code of ref document: A1