WO2025166552A1

WO2025166552A1 - Method, apparatus, device, medium and product for generating three dimensional model

Info

Publication number: WO2025166552A1
Application number: PCT/CN2024/076397
Authority: WO
Inventors: Tongshuai ZHU; Ye Tian; Qizhen LV; Wenbin Li; Chen Li; Xinyu Fang; Yingjie Qian
Original assignee: ABB Schweiz AG
Current assignee: ABB Schweiz AG
Priority date: 2024-02-06
Filing date: 2024-02-06
Publication date: 2025-08-14
Anticipated expiration: 2026-08-06

Abstract

There is a method, an apparatus, an electronic device, a computer-readable storage device, and a computer program product for generating a three dimensional (3D) model. The method comprises obtaining a first 3D model, wherein the first 3D model at least represents a robot and a tool attached to the robot (302). The method further comprises obtaining pose information of the robot and a computer assistant model of the robot (304). The method further comprises identifying the robot in the first 3D model based on the pose information and the computer assistant model (306). The method further comprises generating a second 3D model representing the tool by removing the robot from the first 3D model (308). In this way, a 3D model of a tool can be separated from a robot and thus can be processed individually for use of any following application, thereby the performance of the robot can be improved.

Description

METHOD, APPARATUS, DEVICE, MEDIUM AND PRODUCT FOR GENERATING THREE DIMENSIONAL MODEL

FIELD

Embodiments of the present disclosure generally relate to the field of computer technology and in particular, to a method, an apparatus, an electronic device, a computer-readable medium and a computer program product for generating a three dimensional (3D) model.

BACKGROUND

A robot is an intelligent machine that can work semi-autonomously or fully autonomously. Robots can perform tasks such as operations or movements through programming and automatic control. Robots have basic characteristics such as perception, decision-making, and execution, and can assist or even replace human beings in performing dangerous, heavy, and complex tasks, improving work efficiency and quality, serving human life, and expanding or extending human activities and capabilities.

Point clouds is a set of points in a 3D coordinate system, typically represented as a set of X, Y, and Z coordinates. These points are used to represent the external surface shape of an object. Each point is positioned by a set of Cartesian coordinates (X, Y, Z) , and some may contain color information (R, G, B) or intensity information about the object's surface reflectance. Point clouds can be captured using various 3D scanning devices such as LiDAR, stereo cameras, or time-of-flight cameras. Point cloud processing involves various tasks such as filtering, segmentation, classification, and reconstruction. This processing can be supported by various algorithms. Point clouds have a wide range of applications in fields like reverse engineering, robotic navigation, and 3D modeling.

SUMMARY

In general, various example embodiments of the present disclosure provide a method, an apparatus, an electronic device, a computer-readable storage device, and a computer program product for generating a 3D model.

In a first aspect, it is provided a method for generating a 3D model. The method comprises obtaining a first 3D model, wherein the first 3D model at least represents a robot and a tool attached to the robot. The method further comprises obtaining pose information of the robot and a computer assistant model of the robot. The method further comprises identifying the robot in the first 3D model based on the pose information and the computer assistant model. The method further comprises generating a second 3D model representing the tool by removing the robot from the first 3D model.

In a second aspect, it is provided an apparatus for generating a 3D model. The apparatus comprises a first obtaining module configured to obtain a first 3D model, wherein the first 3D model at least represents a robot and a tool attached to the robot. The apparatus further comprises a second obtaining module configured to obtain pose information of the robot and a computer assistant model of the robot. The apparatus further comprises an identifying module configured to identify the robot in the first 3D model based on the pose information and the computer assistant model. The apparatus further comprises a generating module configured to generate a second 3D model representing the tool by removing the robot from the first 3D model.

In a third aspect, it is provided an electronics device. The electronics device comprises a processor; and a memory coupled to the processor, wherein the memory has instructions stored therein, and the instructions, when executed by the processor, cause the device to execute actions of the first aspect.

In a forth aspect, it is provided a computer-readable medium. The computer-readable medium comprises instructions stored therein, which when executed by a processor, cause the processor to perform methods of the first aspect.

In a fifth aspect, it is provided a computer program product. The computer program product comprises instructions stored therein, which when executed by a processor, cause the processor to perform methods of the first aspect.

It is to be understood that the Summary is not intended to identify key or essential features of embodiments of the present disclosure, nor is it intended to be used to limit the scope of the present disclosure. Other features of the present disclosure will become readily comprehensible through the description below.

DESCRIPTION OF DRAWINGS

Through the following detailed descriptions with reference to the accompanying drawings, the above and other objectives, features and advantages of the example embodiments disclosed herein will become more comprehensible. In the drawings, several example embodiments disclosed herein will be illustrated in an example and in a non-limiting manner, wherein:

FIG. 1 illustrates a schematic diagram of an example environment in which a plurality of embodiments of the present disclosure can be implemented;

FIG. 2 illustrates an example scenario in which some embodiments of the present disclosure can be implemented;

FIG. 3 illustrates a flowchart of an example method for generating a 3D model in accordance with some embodiments of the present disclosure;

FIG. 4 illustrates an example of a second 3D model in accordance with some embodiments of the present disclosure;

FIG. 5 illustrates an example second coordinate in accordance with some embodiments of the present disclosure;

FIG. 6 illustrates an example first coordinate in accordance with some embodiments of the present disclosure;

FIG. 7 illustrates a block diagram of an example apparatus for generating a 3D model in accordance with some embodiments of the present disclosure; and

FIG. 8 illustrates a block diagram illustrating an electronic device in accordance with some embodiments of the present disclosure.

Throughout all the drawings, the same or similar reference numerals represent the same or similar elements.

DETAILED DESCRIPTION OF EMBODIMENTS

Principles of the present disclosure will now be described with reference to several example embodiments shown in the drawings. Though example embodiments of the present disclosure are illustrated in the drawings, it is to be understood that the embodiments are described only to facilitate those skilled in the art in better understanding and thereby achieving the present disclosure, rather than to limit the scope of the disclosure in any manner.

The term comprises "or" includes "and" its variants are to be read as open terms that mean "includes, but is not limited to" . The term "or" is to be read as "and/or" unless the context clearly indicates otherwise. The term "based on" is to be read as "based at least in part on" . The term "being operable to" is to mean a function, an action, a motion or a state can be achieved by an operation induced by a user or an external mechanism. The term "one embodiment" and "an embodiment" are to be read as "at least one embodiment" . The term "another embodiment" is to be read as "at least one other embodiment" . The terms "first" , "second" , and the like may refer to different or same objects. Other definitions, explicit and implicit, may be included below. A definition of a term is consistent throughout the description unless the context clearly indicates otherwise.

The functions or algorithms described herein may be implemented in software in one embodiment. The software may consist of computer executable instructions stored on computer readable media or computer readable storage device such as one or more non-transitory memories or other type of hardware-based storage devices, either local or networked. Further, such functions correspond to modules, which may be software, hardware, firmware or any combination thereof. Multiple functions may be performed in one or more modules as desired, and the embodiments described are merely examples. The software may be executed on a digital signal processor, ASIC, microprocessor, or other type of processor operating on a computer system, such as a personal computer, server or other computer system, turning such computer system into a specifically programmed machine.

The functionality can be configured to perform an operation using, for instance, software, hardware, firmware, or the like. For example, the phrase "configured to" can refer to a logic circuit structure of a hardware element that is to implement the associated functionality. The phrase "configured to" can also refer to a logic circuit structure of a hardware element that is to implement the coding design of associated functionality of firmware or software. The term "module" refers to a structural element that can be implemented using any suitable hardware (e.g., a processor, among others) , software (e.g., an application, among others) , firmware, or any combination of hardware, software, and firmware. The term "logic" encompasses any functionality for performing a task. For instance, each operation illustrated in the flowcharts corresponds to logic for performing that operation. An operation can be performed using, software, hardware, firmware, or the like. The terms, "component" , "system" , and the like may refer to computer-related entities, hardware, and software in execution, firmware, or combination thereof. A component may be a process running on a processor, an object, an executable, a program, a function, a subroutine, a computer, or a combination of software and hardware. The term, "processor" may refer to a hardware component, such as a processing unit of a computer system.

The terms "a" or "an" as used herein, are defined as one or more than one. Also, the use of introductory phrases such as "at least one" and "one or more" in the claims should not be construed to imply that the introduction of another claim element by the indefinite articles "a" or "an" limits any particular claim containing such introduced claim element to disclosures containing only one such element, even when the same claim includes the introductory phrases "one or more" or "at least one" and indefinite articles such as "a" or "an" . The same holds true for the use of definite articles.

Furthermore, the claimed subject matter may be implemented as a method, apparatus, or article of manufacture using standard programming and engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computing device to implement the disclosed subject matter. Computer-readable storage media can include, but are not limited to, magnetic storage devices, e.g., hard disk, floppy disk, magnetic strips, optical disk, compact disk (CD) , digital versatile disk (DVD) , smart cards, flash memory devices, among others. In contrast, computer-readable media, i.e., not storage media, may additionally include communication media such as transmission media for wireless signals and the like.

As discussed above, with the increasing demand for intelligence and digitization of robotic systems, automated generation of collision-free paths and construction of visual virtual environments are increasingly used. The 3D model of a robotic system can be designed using a computer assistant software (such as a computer aided design (CAD) software) for simulation or 3D modeling using vision devices. Three-dimensional models of robotic systems can be simulated using CAD software or modeled using vision equipment. Simulation modeling using CAD software requires the use of professional drawing software and requires the user to have the appropriate professional and technical skills, in addition, CAD modeling requires the physical object to be mapped, modeling efficiency is relatively low. It is difficult to determine the relationship between the environment model and the robot's spatial position when the model is built by vision equipment. Thus, the tool of the robot is always treated as a part of the robot. The tool cannot be processed individually and this may affect other applications such as collision-free paths planning.

Therefore, the present disclosure proposed a new solution for generating a 3D model, especially for a tool of a robot. The proposed solution uses vision sensors to reconstruct a 3D model of the robot's working environment and automatically identifies the 3D model data into the robot's coordinate system. The proposed solution can build 3D models of the robot's working environment (such as the tool attached to the arm of the robot and the work object) more accurately, and thus the 3D model of the tool can be separated from the whole 3D. In this way, the 3D model of a tool can be separated from a robot and thus can be processed individually, thereby the performance of the robot can be improved.

FIG. 1 illustrates a schematic diagram of an example environment 100 in which a plurality of embodiments of the present disclosure can be implemented. The example environment 100 is only illustrated and is not intended to suggest any limitations as to scope of use or functionality of embodiments of the disclosure described herein.

As shown, the example environment 100 comprises a robot 102 and a computing device 110. An example of the computing device 110 may be a server or a computer. A tool 104 is mounted on an end (such as an arm) of the robot 102. The tool 104 may be a welding gun, a suction gripper and so on.

The example environment 100 comprises a vision sensor 106. The vision sensor 106 may be a camera, a RGBD camera or a stereovision device. The vision sensor 106 may be used for generating image data of the robot 102 and its surrounding environment, such as a work object 108. It is to be understood that the vision sensor 106 is optional. The image data of the robot 102 and its surrounding environment can be obtained in advance. a depth measuring sensor, such as a laser, Lidar, a RGBD camera or a stereovision device.

The computing device 110 obtains data from the vision sensor 106 to reconstruct the robot and its working environment in 3D and realizes the segmentation of the surrounding environment model and the tool model by identifying and localizing the robot body and eliminating the robot portion of the reconstructed model. In some example embodiments, the surrounding environment model is converted to the robot base coordinate system, and the tool model is converted to the robot end coordinate system. The tool 104 is fixed at an end of the robot 102 and the work object us in the reachable space. The vision sensor 106 is used to acquire and help reconstruct the overall 3D model of the robot system. The computing device 110 connects with the robot 102 and the vision sensor 106 respectively and acquires the model and pose information of the robot 102 as well as the 3D model of the robot system.

Based on the CAD model of the robot 102 and the pose (or attitude) information, the robot 102 is identified and localized in the reconstructed 3D model. The part of the reconstructed 3D model that overlaps with the CAD model of the robot 102 is eliminated, the 3D model of the tool that is fixed to the end of the robot 102 is set as an independent entity from the first 3D model (e.g., the robot environment model) . Finally, the robot environment model is converted to the robot base coordinate system and the 3D model of the tool (also referred to as a second 3D model) is converted to the robot end coordinate system.

FIG. 2 illustrates an example scenario 200 in which some embodiments of the present disclosure can be implemented. A robot 202 in FIG. 2 may correspond to the robot 102 in FIG. 1. A tool 204 in FIG. 2 may correspond to the tool 104 in FIG. 1. A vision sensor 206 in FIG. 2 may correspond to the vision sensor 106 in FIG. 1. A work object 208 in FIG. 2 may correspond to the work object 108 in FIG. 1. The computing device 210 may correspond to the computing device 110 in FIG. 1.

As shown in FIG. 2, the point clouds of the tool 204 can be split from the whole point clouds of the robot 202 and its surroundings including the work object 208. In this way, the 3D model of a tool can be separated from a robot and thus can be processed individually. Hence, the subsequent applications such as collision-free paths planning can be more accurate, thereby the performance of the robot can be improved.

FIG. 3 illustrates a flowchart of an example method 300 for generating a 3D model in accordance with some embodiments of the present disclosure. For the purposed of better description, FIG. 3 will be described with reference to FIG. 1.

At block 302, the computing device 110 obtains a first 3D model, wherein the first 3D model at least represents a robot and a tool attached to the robot. For example, the computing device 110 receives a 3D model represented by a plurality of point clouds of the robot 102, the tool 104 and the work object 108.

At block 304, the computing device 110 obtains pose information of the robot and a computer assistant model (also referred to as a CAD model thereafter) of the robot. For example, the computing device 110 receives pose information of the robot 102 from the robot 102. The CAD model of the robot 102 may be stored in the computing device 110 in advance. In some example embodiments, the pose information of the robot may comprise a plurality of parameter values of the robot 102, and the plurality of parameter values may indicate at least one of movements and rotations of a plurality of joints of the robot 102.

At 306, the computing device 110 identifies the robot in the first 3D model based on the pose information and the computer assistant model. For example, based on the pose information and the CAD model, the computing device may know the pose of the robot 102 at the moment, and match the pose of the robot 102 to the first 3D mode. As such, the respective portion of the first 3D model which represents the robot 102 can be recognized.

In some example embodiments, the computing device 110 may determine a pose of the robot 102 in the CAD model based on the pose information of the robot 102 and the CAD model of the robot 102. The computing device 110 may determine a location of the robot 102 in the first 3D model based on the pose of the robot 102 in the CAD model. The computing device 110 may determine a mapping between the robot 102 in the CAD model and the robot 102 in the first 3D model based on the location of the robot 102 in the first 3D model.

At 308, the computing device 110 generates a second 3D model representing the tool by removing the robot from the first 3D model. For example, the computing device 110 eliminates the respective portion of the first 3D model which represents the robot 102 from the first 3D model.

By implementing the embodiments of the method 300, a 3D model of the robot's working environment (such as the tool attached to the arm of the robot and the work object) can be built more accurately, and thus the 3D model of the tool can be separated from the whole 3D. In this way, the 3D model of a tool can be separated from a robot and thus can be processed individually, thereby the performance of the robot can be improved.

FIG. 4 illustrates an example of a second 3D model 400 in accordance with some embodiments of the present disclosure. As shown in FIG. 4, a block 402 represents the first 3D model. The first 3D model 402 may comprise a plurality of point clouds of the robot, the tool mounted on the arm of the robot, and the work object. The respective point clouds of the robot 404 may be removed from the first 3D model 402, and the respective point clouds of the work object 406 may be removed from the first 3D model 402. Thus, the respective point clouds of the tool 408 can be separated from the whole point clouds. A detail of the point clouds of the tool 408 is shown in block 410.

Reference coordinate systems are based on geometrical features of real objects such as pinpoints, right-angled edges, etc. The most common method of reference coordinate system calibration is manual calibration by the pinpoint method, in addition to CAD model simulation or the design of specific tools for special cases and other methods.

FIG. 5 illustrates an example second coordinate 500 in accordance with some embodiments of the present disclosure. FIG. 6 illustrates an example first coordinate 600 in accordance with some embodiments of the present disclosure. Herein, the first coordinate may refer to a coordinate which is external to the robot, and the second coordinate may refer to a coordinate which is fixed to a robot end. In some example embodiments, the tool is attached to the robot end, and the work object is placed in a fixed place. In some example embodiments, the work object is attached to the robot end, and the tool is placed in a fixed place.

Reference coordinate systems are based on geometrical features of real objects such as pinpoints, right-angled edges, etc. The most common approach of reference coordinate system calibration is manual calibration by the pinpoint approach, in addition to CAD model simulation or the design of specific tools for special cases and other approaches. In some example embodiments, the method 300 further may use a general robot reference calibration method based on the 3D vision, which utilizes 3D imaging technology to generate a 3D model of the robot body, its work pieces, and tools, and the method 300 may further calculate the position of the reference coordinate system in the robot coordinate system by 3D positioning technology. This is also called the 3D vision-based reference coordinate system calibration approach, which can simplify the process of calibrating the reference coordinate system, enable more flexible coordinate system calibration, and can improve the safety of use by eliminating the need for equipment such as a needle tip, and reduce the technical requirements for the user.

In some example embodiments, the vision sensor may be fixed to the robot body or outside the robot but may remain in position with the robot base, the vision sensor and the robot may determine the hand-eye relationship through hand-eye calibration techniques. In some example embodiments, the robot may move to a new position to collect the point cloud data used to define the target reference coordinate system. The computing device may stitch the collected point cloud data according to the robot position and the hand-eye relationship and convert it to the robot coordinate system until the stitched point cloud is good enough to define target reference coordinate. The stitched point cloud data may be displayed in the human-computer interface. The user can define the pose of the target reference coordinate system by extracting the geometric features in the point cloud, such as at least one of the following: the center of a circle, a plane, a straight line, a corner point associated with the tool or the work object, an axis of a rotationally symmetric object, an axis of a cylinder, an axis of a cone, a center point of a regular polygon, a center point of a line segment, an angle bisector, an intersection point of an extension line, an intersection line of an extended surface, a normal to a plane, or a perpendicular line to a straight line.

The user can also use the CAD model of the work piece or tool with the defined reference coordinate system to determine the position of the reference coordinate system by matching the point cloud and setting the offsets as needed.

As shown in FIG. 5, circle 502 is the input from the user. A point 504 can be extracted. Three planes 506, 508 and 510 which are each other can be extracted afterwards. Then a tool-based coordinate can be built based on the planes 506, 508 and 510.

When the target reference coordinate is separated with the robot, the robot may hold a vision sensor (may be other than the vision sensor 106) to scan the features which used to define the reference coordinate in target object. The point cloud of these features can be translated to the robot base coordinate with the below equation (1) :

whereinrepresents the position of the work object in the robot-based coordinate; represents the position of the tool in the robot-based coordinate; represents the position of the vision sensor in the tool-based coordinate; andrepresents the position of the work object in the vision sensor-based coordinate.

Similarly, as shown in FIG. 6, circle 602 is the input from the user. A point 604 of the work object can be extracted. Three planes along the point 604 which are each other can be extracted afterwards. Then a tool-based coordinate can be built based on the extracted planes.

When the target reference coordinate is attached to the robot, the robot may hold the tool move the features which used to define the reference coordinate to vision sensor work area. The point clouds of these features can be translated to the robot tool0 coordinate with the below equation (2) :

whereinrepresents the tool center position (TCP) in the tool-based coordinate; represents the position of the robot in the tool-based coordinate; represents the position of the vision sensor in the robot-based coordinate; andrepresents the TCP in the vision sensor-based coordinate.

By implementing of the embodiments of FIGS. 5 and 6, the calibration of the reference coordinate system can also be accomplished when the target object does not have pinpoints and corner points. The calibration process is free from the dependence on pinpoints and corners, making the setting of the reference coordinate system more flexible. Further, it can make the reference coordinate setting easier without making sure pin to pin with eye and result more robust with low dependence on human skill. The calibration process can be automated. In this way, the calibration result can be updated regularly. There is no requirement on pinpoint which makes calibration process more safety. The present disclosure applies 3D vision technology to the calibration of the robot's reference coordinate system, and the method 300 is applicable to objects of arbitrary shapes, making the setting of the robot's reference coordinate system more flexible.

Reference is made to FIG. 7, which illustrates a block diagram of an example apparatus 700 for generating a 3D model in accordance with some embodiments of the present disclosure. The apparatus 700 comprises a first obtaining module 702 configured to obtain a first 3D model, wherein the first 3D model at least represents a robot and a tool attached to the robot. The apparatus further comprises a second obtaining module 704 configured to obtain pose information of the robot and a computer assistant model of the robot. The apparatus further comprises an identifying module 706 configured to identify the robot in the first 3D model based on the pose information and the computer assistant model. The apparatus further comprises a generating module 708 configured to generate a second 3D model representing the tool by removing the robot from the first 3D model.

In some example embodiments, the second 3D model may be aligned to a robot-end based coordinate, and the pose information of the robot may comprise a plurality of parameter values of the robot, and the plurality of parameter values may indicate at least one of movements and rotations of a plurality of joints of the robot.

In some example embodiments, the identifying module 706 may further comprise a module configured to determine a pose of the robot in the computer assistant model based on the pose information of the robot and the computer assistant model of the robot; a module configured to determine a location of the robot in the first 3D model based on the pose of the robot in the computer assistant model; and a module configured to determine a mapping between the robot in the computer assistant model and the robot in the first 3D model based on the location of the robot in the first 3D model.

In some example embodiments, the 3D model may comprise a plurality of point clouds. The apparatus may further comprise a module configured to obtain a user input indicating an area where the user expects first coordinate which is external to the robot or a second coordinate which is fixed to a robot end to be generated, wherein the tool or the work object is at least partly included in the area; a module configured to determine a shape of the tool or the work object based on respective point clouds of the tool or the work object in the plurality of point clouds; and a module configured to generate the first coordinate or the second coordinate based on the shape of the tool and the respective point clouds of the tool or the work object.

In some example embodiments, the generating module 708 may further comprise a module configured to extract one or more geometric features of the tool or the work object from the respective point clouds of the tool or the work object; and a module configured to generate the first coordinate or the second coordinate based on the one or more geometric features of the tool or the work object.

In some example embodiments, the one or more geometric features may comprise one or more of the following: a center of a circle associated with the tool or the work object; a plane associated with the tool or the work object; a straight line associated with the tool or the work object; or a corner point associated with the tool or the work object, an axis of a rotationally symmetric object, an axis of a cylinder, an axis of a cone, a center point of a regular polygon, a center point of a line segment, an angle bisector, an intersection point of an extension line, an intersection line of an extended surface, a normal to a plane, or a perpendicular line to a straight line.

In some example embodiments, the generating module 708 may further comprise a module configured to determine respective positions of the one or more geometric features in a robot-based coordinate; and a module configured to generate the the first coordinate with respective to the robot-based coordinate based on the respective positions.

In some example embodiments, the generating module 708 may further comprise a module configured to determine respective positions of the one or more geometric features in a robot end-based coordinate; and a module configured to generate the second coordinate with respective to the robot end-based coordinate based on the respective positions.

In some example embodiments, the apparatus 700 may further comprise a module configured to determine a plurality of offsets along the one or more geometric features with respective to the robot-based coordinate, the generating module 708 may further comprise a module configured to generate the first coordinate with respective to the robot-based coordinate based on the plurality of offsets and the respective positions.

In some example embodiments, the apparatus 700 may further comprise a module configured to determine a plurality of offsets along the one or more geometric features with respective to the robot end-based coordinate, the generating module 708 may further comprise a module configured to generate the second coordinate with respective to the robot end-based coordinate based on the plurality of offsets and the respective positions.

In some example embodiments, the 3D model may comprise a plurality of point clouds, and a device for scanning the 3D model is attached to the robot or placed in a fixed position, and the apparatus 700 may further comprise a module configured to obtain a plurality of point clouds of the tool and the work object collected by the device; a module configured to obtain a mapping between a position of the robot and a position of the device; and a module configured to generate a first coordinate with respective to a robot-based coordinate based on the mapping and the plurality of point clouds of the tool and the work object; and/or a module configured to generate a second coordinate with respective to a robot end-based coordinate based on the mapping and the plurality of point clouds of the tool and the work object.

In some example embodiments, the mapping between the position of the robot and the position of the device may be pre-determined based on a hand-eye calibration.

In some example embodiments, the plurality of point clouds of the tool and the work object may be matched to the robot-based coordinate or the robot end-based coordinate by stitching the collected point cloud data at different locations around the robot based on a hand-eye relationship determined by the hand-eye calibration.

By implementing the example embodiments of FIG. 7, a 3D model of the robot's working environment (such as the tool attached to the arm of the robot and the work object) can be built more accurately, and thus the 3D model of the tool can be separated from the whole 3D. In this way, the 3D model of a tool can be separated from a robot and thus can be processed individually, thereby the performance of the robot can be improved. In some embodiments, the calibration of the reference coordinate system can also be accomplished when the target object does not have pinpoints and corner points. The calibration process can be free from the dependence on pinpoints and corners, making the setting of the reference coordinate system more flexible. Further, it can make the reference coordinate setting easier without making sure pin to pin with eye and result more robust with low dependence on human skill. The calibration process can be automated.

FIG. 8 illustrates a block diagram illustrating an electronic device 800 in accordance with some embodiments of the present disclosure. As indicated, the device 800 includes a central processing unit (CPU) 801, which can execute various appropriate actions and processing based on the computer program instructions stored in a read-only memory (ROM) 802 or the computer program instructions loaded into a random access memory (RAM) 803 from a storage unit 808. The RAM 803 also stores all kinds of programs and data required by operating the electronic device 800. CPU 801, ROM 802 and RAM 803 are connected to each other via a bus 804, to which an input/output (I/O) interface 805 is also connected.

A plurality of components in the device 800 are connected to the I/O interface 805, comprising: an input unit 806, such as a keyboard, a mouse and the like; an output unit 807, such as various types of displays, loudspeakers and the like; a storage unit 808, such as a storage disk, an optical disk and the like; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver and the like. The communication unit 809 allows the device 800 to exchange information/data with other devices through computer networks such as Internet and/or various telecommunication networks.

Each procedure and processing described above, such as the method 300, can be executed by a processing unit 801. For example, in some embodiments, the method 300 can be implemented as computer software programs, which are tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, the computer program can be partially or completely loaded and/or installed to the device 800 via the ROM 802 and/or the communication unit 809. When the computer program is loaded to the RAM 803 and executed by the CPU 801, one or more steps of the above described method 300 are implemented. Alternatively, in other embodiments, the CPU 801 may also be configured in any proper manner to implement the above process/method.

The present disclosure may be a method, a device, a system and/or a computer program product. The computer program product can include a computer-readable storage medium loaded with computer-readable program instructions thereon for executing various aspects of the present disclosure.

The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (anon-exhaustive list) of the computer readable storage medium would include: a portable computer diskette, a hard disk, a random access memory (RAM) , a read-only memory (ROM) , an erasable programmable read-only memory (EPROM or Flash memory) , a static random access memory (SRAM) , a portable compact disc read-only memory (CD-ROM) , a digital versatile disk (DVD) , a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination thereof. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable) , or electrical signals transmitted through a wire.

Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium, or downloaded to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.

Computer readable program instructions for carrying out operations of the present disclosure may be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN) , or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider) . In some embodiments, by means of state information of the computer readable program instructions, an electronic circuitry including, for example, programmable logic circuitry (PLC) , field-programmable gate arrays (FPGA) , or programmable logic arrays (PLA) can be personalized to execute the computer readable program instructions, thereby implementing various aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference to flowchart and/or block diagrams of methods, apparatus (systems) , and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which are executed via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which are executed on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, snippet, or portion of codes, which comprises one or more executable instructions for implementing the specified logical function (s) . In some alternative implementations, the functions noted in the block may be implemented in an order different from those illustrated in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or by combinations of special purpose hardware and computer instructions.

Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are contained in the above discussions, these should not be construed as limitations on the scope of the present disclosure, but rather as descriptions of features that may be specific to particular embodiments. Certain features that are described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination.

A person of ordinary skill in the art may be aware that, in combination with the examples described in the embodiments disclosed in this specification, units and algorithm steps can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on particular applications and design constraints of the technical solutions. A person skilled in the art may use different methods to implement the described functions for each particular application, but it should not be considered that the implementation goes beyond the scope of this application.

It may be clearly understood by a person skilled in the art that, for the purpose of convenient and brief description, for a detailed working process of the foregoing system, apparatus, and unit, refer to a corresponding process in the foregoing method embodiment. Details are not described herein again.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the described apparatus embodiment is merely an example. For example, the unit division is merely logical function division and may be other division in actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented through some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.

The units described as separate parts may be or may not be physically separate, and parts displayed as units may be or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

In addition, functional units in the embodiments of this application may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.

When the functions are implemented in a form of a software functional unit and sold or used as an independent product, the functions may be stored in a computer readable storage medium. Based on such an understanding, the technical solutions in this application essentially, or the part contributing to the prior art, or some of the technical solutions may be implemented in a form of a software product. The computer software product is stored in a storage medium, and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or some of the steps of the methods described in the embodiments of this application. The foregoing storage medium includes: any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (Read-Only Memory, ROM) , a random access memory (Random Access Memory, RAM) , a magnetic disk, or an optical disc.

The foregoing descriptions are merely specific implementations of this application, but are not intended to limit the protection scope of this application. Any variation or replacement readily figured out by a person skilled in the art within the technical scope disclosed in this application shall fall within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

A method for generating a three dimensional (3D) model, comprising:

obtaining a first 3D model at least representing a robot and a tool attached to the robot;

obtaining pose information of the robot and a computer assistant model of the robot;

identifying the robot in the first 3D model based on the pose information and the computer assistant model; and

generating a second 3D model representing the tool by removing the robot from the first 3D model.
The method of claim 1, wherein the second 3D model is aligned to a robot-end based coordinate, and wherein the pose information of the robot comprises a plurality of parameter values of the robot, and wherein the plurality of parameter values indicates at least one of movements and rotations of a plurality of joints of the robot.
The method of claim 1, wherein identifying the robot in the first 3D model based on the pose information and the computer assistant model comprises:

determining a pose of the robot in the CAD model based on the pose information of the robot and the computer assistant model of the robot;

determining a location of the robot in the first 3D model based on the pose of the robot in the computer assistant model; and

determining a mapping between the robot in the computer assistant model and the robot in the first 3D model based on the location of the robot in the first 3D model.
The method of claim 1, wherein the 3D model comprises a plurality of point clouds, and the method further comprises:

obtaining a user input indicating an area where the user expects a first coordinate which is external to the robot or a second coordinate which is fixed to a robot end to be generated, wherein the tool or the work object is at least partly included in the area;

determining a shape of the tool or the work object based on respective point clouds of the tool or the work object in the plurality of point clouds; and

generating the first coordinate or the second coordinate based on the shape of the tool and the respective point clouds of the tool or the work object.
The method of claim 4, wherein generating the first coordinate or the second coordinate based on the shape of the tool or the work object and the respective point clouds of the tool or the work object comprises:

extracting one or more geometric features of the tool or the work object from the respective point clouds of the tool or the work object; and

generating the first coordinate or the second coordinate based on the one or more geometric features of the tool or the work object.
The method of claim 5, wherein the one or more geometric features comprises one or more of the following:

a center of a circle associated with the tool or the work object;

a plane associated with the tool or the work object;

a straight line associated with the tool or the work object;

a corner point associated with the tool or the work object;

an axis of a rotationally symmetric object;

an axis of a cylinder;

an axis of a cone;

a center point of a regular polygon;

a center point of a line segment;

an angle bisector;

an intersection point of an extension line;

an intersection line of an extended surface;

a normal to a plane; or

a perpendicular line to a straight line.
The method of claim 5, wherein generating the first coordinate or the second coordinate based on the one or more geometric features of the tool or the work object comprises:

for the first coordinate:

determining respective positions of the one or more geometric features in a robot-based coordinate; and

generating the first coordinate with respective to the robot-based coordinate based on the respective positions;

for the second coordinate:

determining respective positions of the one or more geometric features in a robot end-based coordinate; and

generating the second coordinate with respective to the robot-end based coordinate based on the respective positions.
The method of claim 7, further comprising:

determining a plurality of offsets along the one or more geometric features with respective to the robot-based coordinate; and

wherein generating the first coordinate with respective to the robot-based coordinate comprises:

generating the first coordinate with respective to the robot-based coordinate based on the plurality of offsets and the respective positions; or

determining a plurality of offsets along the one or more geometric features with respective to the robot end-based coordinate; and

wherein generating the second coordinate with respective to the robot end-based coordinate comprises:

generating the second coordinate with respective to the robot end-based coordinate based on the plurality of offsets and the respective positions.
The method of claim 1, wherein the 3D model comprises a plurality of point clouds, and a device for scanning the 3D model is attached to the robot or placed in a fixed position, and wherein the method further comprises:

obtaining a plurality of point clouds of the tool and the work object collected by the device;

obtaining a mapping between a position of the robot and a position of the device; and

generating a first coordinate with respective to a robot-based coordinate based on the mapping and the plurality of point clouds of the tool and the work object; or

generating a second coordinate with respective to a robot end-based coordinate based on the mapping and the plurality of point clouds of the tool and the work object.
The method of claim 9, wherein the mapping between the position of the robot and the position of the device is pre-determined based on a hand-eye calibration.
The method of claim 10, wherein the plurality of point clouds of the tool and the work object are matched to the robot-based coordinate or the robot end-based coordinate by stitching the collected point cloud data at different locations around the robot based on a hand-eye relationship determined by the hand-eye calibration.
An apparatus for generating a three dimensional (3D) model, comprising:

a first obtaining module configured to obtain a first 3D model, wherein the first 3D model at least represents a robot and a tool attached to the robot;

a second obtaining module configured to obtain pose information of the robot and a computer assistant model of the robot;

an identifying module configured to identify the robot in the first 3D model based on the pose information and the computer assistant model; and

a generating module configured to generate a second 3D model representing the tool by removing the robot from the first 3D model.
An electronic device, comprising:

a processor; and

a memory coupled to the processor, wherein the memory has instructions stored therein, and the instructions, when executed by the processor, cause the device to execute actions of any of claims 1-11.
A computer-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform a method of any of claims 1-11.
A computer program product having instructions stored therein, which when executed by a processor, cause the processor to perform a method of any of claims 1-11.