KR20230160276A - Autonomous welding robot - Google Patents

Abstract

In various embodiments, a computer-implemented method of generating commands for a welding robot. The computer-implemented method includes: identifying expected locations for candidate consideration of the part based on a Computer Aided Design (CAD) model of the part to be welded; scanning a workspace containing the part to create a representation of the part; Identifying a candidate seam on the part based on the expected location of the candidate seam and a representation of the part; Determining an actual location of the candidate seam; and At least in part based on the actual position of the candidate seam to the welding robot. This includes generating welding commands for:

Description

Autonomous welding robot

This application is related to U.S. Provisional Patent Application No. 63/153,109, filed February 24, 2021, entitled “SYSTEMS AND METHODS FOR OPERATING AND CONTROLLING A WELDING ROBOT” Claims the benefit of U.S. Provisional Patent Application Serial No. 63/282,827, filed November 24, 2021, the entire contents of which are hereby incorporated by reference for all purposes.

Robotic manufacturing involves using robots to perform one or more aspects of a manufacturing process. Robot welding is one application area in robotic manufacturing. In robotic welding, a robot welds two or more components together along one or more seams. These robots offer significant advantages in terms of production time, reliability, efficiency and cost because they automate processes previously performed by humans or by machines directly controlled by humans.

Various examples include computer-implemented methods for generating commands for welding robots. The computer-implemented method includes identifying expected locations of candidate seams on a part to be welded based on a Computer Aided Design (CAD) model of the part; scanning a workspace containing the part to create a representation of the part; identifying candidate Sims for the part based on the expected location of the candidate Sims and the representation of the part; determining the actual location of said candidate deliberation; and generating a welding command for the welding robot based at least in part on the actual location of the candidate seam.

For example, there is a computer-implemented method for generating welding commands for a welding robot. The method includes obtaining image data of a workspace including a part to be welded through a sensor; identifying a plurality of points on the part to be welded based on the image data; identifying candidate seams on parts to be welded from the plurality of points; and generating a welding command for the welding robot based at least in part on the candidate deliberation identification.

1 is a block diagram of an autonomous robotic welding system according to various embodiments.
2 is a schematic diagram of an autonomous robotic welding system according to various embodiments.
3 is a schematic diagram of an autonomous robotic welding system according to various embodiments.
4 is an example point cloud of a part with weldable seams according to various embodiments.
5 is an example point crowd of a part with weldable seams according to various embodiments.
FIG. 6 is a block diagram illustrating a registration process flow according to various embodiments.
7 is a schematic diagram of a graph-search technique that can determine a path plan for a robot according to various embodiments.
8 is a mathematical model of a robot arm according to various embodiments.
Figure 9 is a flowchart of a method for performing autonomous welding according to various embodiments.
Figure 10 is a flowchart of a method for performing autonomous welding according to various embodiments.
Figure 11 is a flowchart of a method for performing autonomous welding according to various embodiments.

Conventional welding techniques are tedious, labor-intensive and inefficient. Additionally, existing welding technologies are not flexible enough to accommodate the irregularities that commonly occur in manufacturing processes, resulting in undesirable downtime and inefficiencies. For example, existing welding technology requires a skilled programmer to generate the instructions needed for a welding robot to perform the welding task. These commands instruct the welding robot what motions, paths, trajectories, and welding parameters it should use to perform a specific welding task. The commands were written under the assumption that a large amount of work would be performed, repeating the same welding operation multiple times. Therefore, any aberrations that occur during the welding process (e.g., other parts) may result in incorrect welding positions. Misplaced welds ultimately increase inefficiencies, costs and other negative aspects of mass production.

In some cases, a computer-aided design (CAD) model of a part may be useful to a welding robot to facilitate the welding operation. For example, a CAD model of the part to be welded may be provided to the welding robot, and the welding robot may use the CAD model to guide movements such as the position of the seam to be welded. The seam to be welded is assigned an annotation to the CAD model (e.g., an annotation containing a user-selected edge where each corner represents a seam), and the welding robot uses a sensor to find the seam and then welds according to the annotations. This approach can reduce or eliminate the need for skilled programmers or manufacturing engineers, but it has limitations. For example, welding work is a very precise work. In general, to produce an acceptable weld, it is desirable for the welding tip to be located within 1 mm of the target position relative to the seam. When guiding a welding robot based on a CAD model, even if the part closely matches the CAD model, the actual shim may be more than 1 mm away from the modeled position, making precise positioning of the welding tip to produce an acceptable weld difficult. Or it may be impossible. If the CAD model is a simplification of the actual part (which is common in low-volume production), the shim may be removed by more than 1 cm from the modeled location. Therefore, using known techniques, it can be difficult to accurately find seams based on a CAD model. Therefore, welding robots can produce unacceptable welds, resulting in defective parts.

Additionally, previous solutions for controlling welding robots required that the welding robot (more specifically the robotic arm) move to the first point, such as a seam, within the manufacturing workspace to avoid collisions with other components (e.g. parts, sensors, clamps, etc.). A skilled operator was needed to give specific commands to the welding robot as it moved along the path from to the second point. Identifying a path free of obstacles and collisions (e.g. a path that a robot can follow to weld a seam) is referred to here as path planning. Requiring skilled workers to perform path planning for tens or hundreds of potential paths for a robotic arm is inefficient, tedious, and expensive. Moreover, existing welding robots are often programmed to repeatedly follow the same path, same motion, and same trajectory. This repetitive process may be acceptable in a high-volume manufacturing environment where the manufacturing process is highly mature; however, in low or medium volume settings, components may be placed in unexpected positions relative to the welding robot, resulting in collisions, Misalignment, poor tolerances, and other problems can occur. Therefore, in certain settings, a skilled operator may be required to facilitate welding.

The welding technology described herein is superior to previous welding robots and technologies because it can automatically and dynamically generate commands useful to the welding robot to accurately and accurately identify and weld seams. Unlike previous systems and techniques, the welding techniques described here do not necessarily require a CAD model of the part to be welded (although CAD models may be useful in some examples and as described below), nor do they necessarily require a CAD model of the part or manufacturing workpiece. A priori information about space is not necessarily necessary. Rather, the welding technology described here uses mobile sensors to map the manufacturing workspace (particularly parts and seams) in three-dimensional (3D) space and uses these maps to locate and weld seams with a high level of accuracy and precision. The welding technology described here includes the ability to identify multiple candidate seams for welding, interact with the user to select candidate seams for welding, dynamically change welding parameters, and provide feedback on the welding operation. Additionally, the welding technology described herein is configured to automatically and dynamically perform path planning using data acquired by sensors, i.e. automatically and dynamically plan one or more paths in the manufacturing workspace followed by the robot without any a priori information. identified, the arm can move without colliding with other components. The welding technology described herein is also configured to dynamically perform path planning and welding using a combination of data acquired by sensors and prior information (e.g., an annotated CAD model). These and other embodiments are now described below with reference to the drawings.

Figure 1 is a block diagram of an autonomous robotic welding system 100 according to various embodiments. System 100 includes a manufacturing workspace 101, a user interface 106, a controller 108, and storage 109 that stores a database 112. System 100 may include other components or subsystems not explicitly described herein. Manufacturing workspace 101 is an area or enclosure where robotic arm(s) operate on one or more parts positioned, coupled or supported by a platform or positioner while aided by information received through one or more sensors. In embodiments, workspace 101 may be any suitable welding area designed with appropriate safety measures for welding. For example, the workspace 101 may be a welding area located in a workshop, workshop, manufacturing plant, manufacturing plant, etc. In an embodiment, manufacturing workspace 101 (or more generally workspace 101 ) includes sensors 102 , robots 110 configured to perform welding-type processes such as welding, brazing, and bonding, and parts to be welded ( 114) (e.g., a portion with a seam) and a fixture 116. Fixture 116 may hold, position, and/or manipulate part 114 and may be, for example, a clamp, platform, positioner, or other type of fixture. Fixture 116 may be configured to securely secure part 114. In examples, fixture 116 can be adjusted manually by a user or automatically by a motor. For example, fixture 116 may dynamically adjust its position, orientation, or other physical configuration before or during the welding process. In some examples, robot 110 may include one or more sensors 102. For example, one or more sensors 102 may be located on an arm of the robot 110 (e.g., a welding head attached to the arm). In another example, one or more sensors 102 may be located on a movable, unwelded robot arm (which may be different from robot 110). In another example, one of the one or more sensors 102 may be located on an arm of the robot 110 and another of the one or more sensors 102 may be located on a movable piece of equipment in the workspace. In another example, one of the one or more sensors 102 may be located on an arm of the robot 110 and the other of the one or more sensors 102 may be located on a movable, non-welding robot arm.

Sensor 102 is configured to capture information about workspace 101 . In an embodiment, sensor 102 is an image sensor configured to capture visual information (e.g., a two-dimensional (2D) image) about workspace 101. For example, the sensor 102 may include a camera (eg, a camera with a built-in laser), a scanner (eg, a laser scanner), etc. Sensor 102 may include a sensor, such as a LiDAR (Light Detection and Ranging) sensor. Alternatively or additionally, sensor 102 may be an audio sensor configured to emit and/or capture sound, such as a Sound Navigation and Ranging (SONAR) device. Alternatively or additionally, sensor 102 may be an electromagnetic sensor configured to emit and/or capture electromagnetic (EM) waves, such as a radio detection and ranging (RADAR) device. Sensors 102 may collect information about the physical structure of workspace 101 through visual, auditory, electromagnetic and/or other sensing technologies. (moving structures of the workspace 101), and in another example, sensors 102 collect a combination of static and dynamic information. Sensors 102 may collect any suitable combination of information regarding the physical structure of workspace 101 and provide such information to other components (e.g., controller 108) to provide workspace 101 ) can create a 3D representation of the physical structure. As described above, sensor 102 may capture and convey any of a variety of information types, but this description does not apply to sensor 102 primarily capturing visual information (e.g., 2D images) of workspace 101. It is assumed that they are captured and used en masse to create a 3D representation of the workspace 101, as described later.

To create a 3D representation of workspace 101, sensor 102 captures 2D images of the physical structure of workspace 101 from various angles. For example, although a single 2D image of a fixture 116 or part 114 may be inadequate for creating a 3D representation of that component, likewise a single 2D image of a fixture 116 or part 114 may be viewed from a single angle, view, or plane. While a set of multiple 2D images 114 may be inadequate for creating a 3D representation of a given component, multiple 2D images captured from multiple angles at various locations within the workspace 101 can be used to create a fixture 116 or part 114 ) may be suitable for creating 3D representations of components such as Capturing 2D images in multiple orientations allows you to view a part conceptually similar to how a top view of a component, including front, profile, and top-down views of the component, provides all the information needed to create a 3D representation of that component. This is because it provides three-dimensional spatial information. Accordingly, in embodiments, sensors 102 are configured to move around workspace 101 to capture information suitable for creating a 3D representation of structures within workspace 101 . In the example, the sensors are stationary but present in an appropriate number and appropriately varied locations around the workspace 101 such that the appropriate information is captured by the sensors 102 to generate the 3D representation described above. In instances where sensor 102 is movable, any suitable structure may be useful to facilitate such movement around workspace 101. For example, one or more sensors 102 may be located in a powered track system. The track system itself may be stationary, but the sensors 102 are configured to move around the workspace 101 of the track system. However, in some examples, the sensors 102 are movable in the track system and the track system itself is movable around the workspace 101. In another example, one or more mirrors are arranged within the workspace 101 with a sensor 102 capable of pivoting, pivoting, rotating or translating about and/or along a point or axis, so that the sensor 102 When in the first configuration, capture a 2D image from an initial vantage point, and when in the second configuration, use a mirror to capture a 2D image from a different vantage point. In another example, the sensor 102 may be suspended from an arm that may be configured to pivot, pivot, rotate, or translate about and/or along a point or axis, and the sensor 102 may be configured to move these arms throughout their entire range of motion. When extended through, it can be configured to capture 2D images from various vantage points.

In some embodiments, some or all of the sensor 102 configurations described above are implemented. Other sensor 102 configurations are contemplated and are included within the scope of this disclosure.

Still referring to FIG. 1 , robot 110 (e.g., a welding head of robot 110) is configured to move within workspace 101 according to a path plan received from controller 108, as described below. It is composed. Robot 110 is further configured to perform one or more suitable manufacturing processes (e.g., welding operations) on part 114 in accordance with instructions received from controller 108. In some embodiments, robot 110 may be a six-axis robot equipped with a welding arm. Robot 110 may be any suitable robotic welding equipment, such as a YASKAWA® robot arm, ABB® IRB robot, KUKA® robot, etc. The robot 110 may be used for arc welding, resistance welding, spot welding, TIG (tungsten inert gas) welding, MAG (metal activated gas) welding, MIG (metal inert gas) welding, laser welding, plasma welding, combinations thereof, and/or It could be configured to do something similar.

Still referring to FIG. 1 , workspace 101 , particularly sensor(s) 102 and robots 110 within workspace 101 , are connected to controller 108 . Controller 108 herein is any suitable machine that is specifically and specially configured (e.g., programmed) to perform the operations assigned to controller 108 or, more generally, system 100. In some embodiments, controller 108 is not a general-purpose computer, but is instead specifically programmed and/or hardware configured to perform tasks assigned herein to controller 108, or more generally to system 100. In some embodiments, controller 108 is or includes an application specific integrated circuit (ASIC) configured to perform operations assigned herein to controller 108 or, more generally, to system 100. In some embodiments, controller 108 includes or is a processor, such as a central processing unit (CPU). In some embodiments, controller 108 is a field programmable gate array (FPGA). In embodiments, the controller 108 includes executable code that, when executed by the controller 108, causes the controller 108 to perform one or more of the operations attributed herein to the controller 108 or, more generally, to the system 100. Includes memory to store . Controller 108 is not limited to the specific examples described herein.

The controller 108 controls the sensor(s) 102 and the robot 110 within the workspace 101. In some embodiments, controller 108 controls fixture(s) 116 within workspace 101. For example, controller 108 may move within workspace 101 as described above and/or control sensor(s) 102 to capture 2D images, audio data, and/or EM data as described above. can do. For example, the controller 108 may control the robot 110 described herein to perform welding operations and move within the workspace 101 according to path planning techniques described below. For example, the controller 108 may manipulate fixture(s) 116 such as positioners (e.g., platforms, clamps, etc.) to rotate, translate, or move one or more parts within the workspace 101. there is. Controller 108 may also control other aspects of system 100. For example, the controller 108 may further interact with the user interface (UI) 106 by providing a graphical interface on the UI 106, such as providing various types of information to and/or from the user. and/or by receiving (e.g., identified seams that are candidates for welding, possible routes during route planning, welding parameter options or selections, etc.), which allows the user to interact with the system 100 and provide input to the system 100. and through this the controller 108 can interact with the user. UI 106 may be any type of interface, including a touchscreen interface, voice activated interface, keypad interface, combinations thereof, etc.

Additionally, controller 108 may interact with database 112 , such as by storing data in and/or retrieving data from database 112 . Database 112 may be stored in any suitable type of storage 109 more generally configured to store any type of information. In some embodiments, database 112 may include random access memory (RAM), a memory buffer, a hard drive, erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), or read-only memory (ROM). ), may be stored in storage 109 such as flash memory, etc. In some embodiments, database 112 may be stored on a cloud-based platform. Database 112 may store any information useful to system 100 when performing a welding operation. For example, database 112 may store a CAD model of part 114. As another example, database 112 may store an annotated version of a CAD model of part 114. Database 112 may also store a point cloud of part 114 created using a CAD model (also referred to herein as a CAD model point cloud). Similarly, part 114 is generated based on a 3D representation of part 114 and/or user input provided regarding part 114 (e.g., regarding the seam of part 114 to be welded, welding parameters, etc.). ) may be provided and stored in the database 112. In an example, storage 109 stores executable code 111 that, when executed, causes controller 108 to perform one or more operations attributed herein to controller 108 or, more generally, to system 100. In an example, executable code 111 is a single, self-contained program, and in other embodiments, executable code is a program that has one or more function calls to other executable code, which may be stored in storage 109 or elsewhere. am. In some embodiments, one or more functions resulting from execution of executable code 111 may be implemented by hardware. For example, multiple processors may be useful for performing one or more individual tasks of executable code 111.

2 is a schematic diagram of an illustrated autonomous robotic welding system 200 according to various embodiments. System 200 is an example of system 100 of Figure 1, with like reference numbers referring to like elements. For example, system 200 includes workspace 201 . The workspace 201 optionally includes a movable sensor 202, a robot 210 (which may include one or more sensors 202 mounted thereon in addition to the movable sensor 202), and a fixture 216. ). The robot 210 includes multiple joints and members (e.g., shoulders, arms, elbows, etc.) that allow the robot 210 to move in any suitable degree of freedom. The robot 210 has parts , for example, includes a welding head 210A that performs a welding operation on a part that can be supported by a fixture 216 (e.g. a clamp). The system 200 has a UI connected to the workspace 201. It further includes 206. In operation, the sensor 202 collects a 2D image of the workspace 201 and provides the 2D image to a controller (not explicitly shown in Figure 2). The controller controls the fixture ( 216), generate a 3D representation (e.g., a point cloud) of the workspace 201, such as portions supported by fixtures 216, and/or other structures within the workspace 201. The controller is described herein. Control the robot 210 to weld the seam as shown, and plan a path for welding the seam so that the robot 210 does not collide with the structure in the workspace 201, supported by a seam (e.g., fixture 216). Use 3D representation to identify parts that are

3 is a schematic diagram of an autonomous robotic welding system 300 according to various embodiments. System 300 is an example of system 100 of FIG. 1 and system 200 of FIG. 2, with like numbers referring to similar elements. For example, system 300 includes workspace 301. Workspace 301 optionally includes a movable sensor 302, a robot 310 (which may be equipped with one or more sensors 302), and a fixture 316 (e.g., a platform or positioner). Robot 310 includes multiple joints and members (eg, shoulders, arms, elbows, etc.) that allow robot 310 to move with any suitable degree of freedom. The robot 310 includes a welding head 310A that performs a welding operation on a part, for example, a part that can be supported by the fixture 316. System 300 further includes UI 306 connected to workspace 301. In operation, sensor 302 collects a 2D image of workspace 301 and provides the 2D image to a controller (not explicitly shown in Figure 3). The controller generates a 3D representation (e.g., a point cloud) of the workspace 301, such as the fixtures 316, the parts supported by the fixtures 316, and/or other structures within the workspace 301. The controller uses the 3D representation to identify a seam (e.g., a part supported by fixture 316) and plan a path for robot 310 to weld the seam without colliding with structures within workspace 301. , the robot 310 is controlled to perform welding the seam as described herein.

Referring back to Figure 1, as described above, controller 108 receives a 2D image (and possibly other data, such as audio data or EM data) from sensor 102 and displays a 3D image of the structure depicted in the 2D image. It is configured to create a representation. A 3D representation can be called a point cloud. A point cloud may be a set of points each representing a 3D spatial location of a point on the surface of part 114 and/or fixture 116. In some embodiments, one or more 2D images (e.g., image data captured by sensor(s) 102 in a particular orientation relative to part 114) are superimposed and/or stitched together by controller 108. 3D image data of the workspace 101 can be reconstructed and generated. The 3D image data may be collated to create a point cloud with associated image data for at least some of the points in the point cloud.

In embodiments, 3D image data may be collated by controller 108 in such a way that a point cloud generated from the data may have six degrees of freedom. For example, each point in a point cloud may represent an infinitesimal location in 3D space. As described above, sensor(s) 102 may capture multiple 2D images of a point from various angles. These multiple 2D images can be collated by controller 108 to determine the average image pixel for each point. Averaged image pixels can be attached to points. For example, if sensor(s) 102 is a color camera with red, green, and blue channels, the six degrees of freedom are {x-position, y-position, z-position, red intensity, green intensity, and blue intensity. } can be. For example, if sensor(s) 102 is a black and white camera with a black and white channel, four degrees of freedom can be created.

4 is an example point crowd 400 of a part with weldable seams according to various embodiments. More specifically, point cloud 400 represents parts 402 and 404 to be welded together along seams 406. 5 is an example point crowd 500 of a part with weldable seams according to various embodiments. More specifically, point cloud 500 represents parts 502 and 504 to be welded together along seams 506 . Controller 108 (FIG. 1) is configured to generate 3D point clouds 400, 500 based on 2D images captured by sensor 102, as described above. The controller 108 then plans the welding path and uses the robot 110 to place the weld along the seams 406, 506 (FIG. 1). ) (or, in some embodiments, image data useful for generating point clouds 400, 500) may be used to identify and locate seams such as ). How controller 108 executes executable code 111 (Figure 1) to perform these tasks, including seam identification and path planning, is now described in detail.

When executing executable code 111 , the controller 108 uses a neural network to determine pixel-by-pixel values (e.g., using images captured by sensor 102 or based on images captured by sensor 102 ). ) and/or point-by-point (e.g., one or more point cloud) classification to identify and classify structures within the workspace 101. For example, the controller 108 may be used on a pixel-by-pixel basis to identify each imaged structure within the workspace 101 as a part 114, as a seam of a part 114, or at an interface between multiple parts 114. and/or can perform point-by-point classification (reference: here candidate shim), and are used as fixtures 116, robots 110, etc. For example, the controller 108 may identify and classify pixels and/or points based on a neural network (e.g., a U-net model) trained using appropriate training data. Neural networks can be trained on image data, point cloud data, geospatial data, or a combination of these. Because the point cloud and/or image data includes information captured from various vantage points within the workspace 101, the neural network can simulate fixtures 116 or candidate images of part(s) 114 from multiple angles and/or perspectives. It can be operated to classify. In some embodiments, a neural network can be trained to operate directly on a set of points, and a dynamic graph convolutional neural network (k), a neural network, can be implemented to analyze disorganized points on a point cloud. In some embodiments, a first neural network may be trained on point cloud data to perform point-by-point classification and a second neural network may be trained on image data to perform pixel-by-pixel classification. The first neural network and the second neural network may individually identify candidate Sims and localize candidate Sims. The outputs of the first neural network and the second neural network may be combined into a final output for determining the location and direction of one or more candidate deltas of part 114.

In some embodiments, once pixel-by-pixel classification is performed, the results may be projected onto 3D point cloud data and/or a mesh version of the point cloud data, thereby providing information about the location of fixture 116 in workspace 101. can be provided. When the input data is image data (for example, a color image), spatial information such as depth information may be included along with the color data for pixel-level segmentation. In some embodiments, pixel-by-pixel classification may be performed to identify candidate seams and locate candidate seams relative to part 114, as described further below.

As described above, controller 108 may identify and classify pixels and/or points into specific structures within workspace 101, such as fixtures 116, parts 114, candidate seams for part 114, etc. . Portions of the image and/or point cloud data classified as non-part and non-candidate seam structures, such as fixtures 116, may be segmented (e.g., modified or removed) from the data, thereby forming parts 114 and/or Alternatively, data identified and classified to correspond to the candidate Sim of part 114 may be isolated. In some embodiments, after identifying the candidate Sims and partitioning the non-part 114 and non-candidate Sim data as described above (or optionally prior to such partitioning), the neural network may be configured to analyze each candidate Sim to determine the Sim type. You can. For example, a neural network can be configured to determine whether a candidate seam is a butt joint, corner joint, edge joint, lap joint, tee joint, etc. A model (e.g., a U-net model) may classify deliberation types based on data captured from multiple vantage points within workspace 101.

When pixel-by-pixel classification is performed using image data, controller 108 projects pixels of interest (e.g., pixels representing part 114 and candidate seams for part 114) into 3D space to classify part 114. ) can generate a 3D point set representing the candidate seam and part 114. Alternatively, if classification by point is performed using point cloud data, the point of interest may already exist in the three-dimensional space within the point cloud. In either case, for the controller 108, the 3D points are an unordered set of points and at least some of the 3D points may be lumped together. To remove this noise and generate a continuous and continuous subset of points representing candidate seams, the Manifold Blurring and Mean Shift (MBMS) technique or similar techniques can be applied. These techniques can compress points and remove noise. Controller 108 may then apply a clustering method to decompose the candidate Sims into individual candidate Sims. In other words, instead of having multiple subsets of points representing multiple seams, clustering allows us to divide each subset of points into individual seams. After clustering, controller 108 can fit a spline to each individual subset of points. Therefore, each individual subset of points can be an individual candidate seam.

To summarize and not by way of limitation, using the techniques described above, controller 108 receives image data captured by sensor 102 at various locations and vantage points within workspace 101. The controller 108 uses a neural network to perform pixel-by-pixel and/or point-by-point classification techniques to classify each pixel and/or point as a part 114, a candidate seam for part 114, or multiple parts 114, fixtures 116. ), etc. are classified and identified as interfaces between them. Structures identified as non-part 114 structures and non-candidate seam structures are split, and the controller 108 may perform further processing on the remaining points (e.g., to mitigate noise). By performing these operations, the controller 108 can generate a candidate shim set indicating the location and direction of the shim in the part 114. As will now be described, the controller 108 may determine whether the candidate seam is in fact a seam and may optionally perform further processing using a priori information such as the CAD model of the part and the seam. The resulting data is suitable for use by the controller 108 to plan a path for placing welds along identified seams, as described below.

In some cases, the identified candidate Sim may not be a Sim (i.e., the identified candidate Sim may be a false positive). To determine whether an identified candidate Sim is in fact a Sim, controller 108 uses images captured by sensor 102 at various vantage points within workspace 101 to determine confidence values. Reliability indicates the possibility that the candidate Sim determined at that point is the actual Sim. The controller 108 may then compare the confidence values for the different perspectives and remove candidate Sims that may not be actual Sims. For example, controller 108 may determine the mean, median, maximum, or any other suitable summary statistic of the candidate values associated with a particular candidate Sim. Typically, a candidate Sim that corresponds to a real Sim will have consistently high (e.g., above a threshold) confidence values across the various vantage points used to capture that candidate Sim. If the summary statistics of reliability for the candidate Sim are greater than or equal to the threshold, the control unit 108 may designate the candidate Sim as the actual Sim. Conversely, if the summary statistics of the reliability value for the candidate seam are below the threshold, the candidate seam may be designated as a false positive not suitable for welding.

As mentioned above, after identifying a candidate Sim that is actually a Sim, the controller 108 may use a priori information, such as a CAD model (or a point cloud version of the CAD model), to perform further processing, referred to herein as registration. More specifically, in some cases there may be differences between the seam dimensions of the part and the seam dimensions of the CAD model, and the CAD model must be transformed (e.g., updated) to account for those differences. It is later used to perform route planning as described here. Accordingly, the controller 108 configures a first Sim (e.g., a candidate Sim for part 114 that has been verified as an actual Sim) and a second Sim (e.g., a Sim annotated (e.g., by an operator/user) to a CAD model corresponding to the first Sim). ) is compared to determine the difference between the first and second trials. As described above, you can annotate the seam of a CAD model. If the CAD model and/or controller 108 accurately predicts the position of the candidate shim, the first shim and the second shim may be at approximately the same position. Alternatively, if the CAD model and/or controller 108 is partially accurate, the first shim and the second shim may partially overlap. The controller 108 may perform a comparison between the first and second trials. This comparison of the first and second shims may be based in part on the shapes of the two shims and their relative positions in space. If the first core and the second core have relatively similar shapes and are adjacent to each other, the second core can be identified as being identical to the first core. In this way, controller 108 can account for topography of the part surface that is not accurately represented in the CAD model. In this way, the controller 108 can identify the candidate seams and use the CAD model of the part to sub-select, improve, or update the candidate seams for the part. Each candidate Sim may be an updated set of points indicating the position and direction of the candidate Sim relative to the part.

Figure 6 is a block diagram illustrating a registration process flow 600 according to various embodiments. Some or all steps of registration process flow 600 are performed by controller 108 (Figure 1). Controller 108 may first perform coarse registration 602 using a point cloud of CAD model 604 and a scanned point cloud 606 formed using images captured by sensor 102 . The CAD model point cloud 604 and the scan point cloud 606 may be sampled so that their points have a uniform or nearly uniform distribution and both have the same or approximately the same point density. In the example, controller 108 downsamples point clouds 604, 606 by randomly and uniformly selecting points in the cloud to retain and discard remaining unselected points. For example, in some embodiments, a Poisson Disk Sampling (PDS) down-sampling algorithm may be implemented. The controller 108 may provide the boundaries of the point clouds 604, 606, the minimum distance between samples, and the limit of samples to select before a sample is rejected as inputs to the PDS algorithm. In some embodiments, a delta network may be useful for transforming one model into another during coarse registration 602. A delta network can be a Siamese network that takes a source model and a target model and encodes them into latent vectors. Controller 108 can use latent vectors to predict point-by-point transformations that transform or update one model to another. Delta networks may not require a training data set. Given a CAD model and a scanned point cloud 604, 606, the controller 108 in the context of a delta network spends one or more epochs learning the similarity or degree of similarity between the two. During this epoch, the Delta network learns meaningful features that are useful for subsequent registrations. Delta networks may use skip connections to learn transformations, but some embodiments may not use skip connections. In some cases, CAD models include surfaces that do not exist in the 3D point cloud created using captured images (e.g., scans). In this case, the delta network moves all points corresponding to the missing surfaces in the CAD model point cloud 604 to some points in the scan point cloud 606 (and updates the scan point cloud 606). Accordingly, during registration, the controller 108 (e.g., a delta network) may use the learned features and transform (or update) the original CAD model, or use the learned features and transform the transformed CAD model. You can. In an example, the delta network may include an encoder network such as a Dynamic Graph Convolutional Neural Network (DGCNN). After the point cloud is encoded with features, a concatenated vector consisting of CAD and scan embeddings can be formed. After implementing the pooling operation (e.g., maxpooling), a decoder can be applied to the resulting vector. In some embodiments, the decoder may include five convolutional layers with specific filters (e.g., 256, 256, 512, 1024, Nx3 filters). The resulting output can be concatenated with the CAD model and scan embeddings, max-pulled, and then provided once more to the decoder. The final result may include point-by-point transformations.

Irrelevant data and noise in the data (e.g., the output of coarse registration 602) may affect the registration of part 114. At least for this reason, it is desirable to remove as much irrelevant data and noise as possible. Bounding box 608 is useful for removing such extraneous data and noise (e.g., fixture 116) to limit the area over which registration is performed. In other words, data inside the bounding box is maintained, but all data (3D or otherwise) outside the bounding box is deleted. The bounding box described above may be of any shape that can enclose or encapsulate (eg partially or completely) the CAD model itself. For example, the bounding box may be an expanded or enlarged version of the CAD model. Data outside the bounding box may be removed from the final registration or may still be included but may be weighted to mitigate the impact.

Still referring to Figure 6, during refined registration 610, controller 108 passes the output of bounding box 608 to the patch through a set of convolutional layers of a neural network trained as an autoencoder. More specifically, data may pass through the encoder portion of the autoencoder, and the decoder portion of the autoencoder may be unused. The input data may be, for example, the XYZ location of a patch point in the shape of (128, 3). The output could be a vector of length 1024, for example, which is useful for point-wise features.

The corresponding point set that best supports rigorous conversion between the CAD point cloud and the scanned point cloud model must be determined during registration. The candidate may be stored (e.g., in the database 112) as a matrix in which each element stores the probability or reliability of a match between two points.

Various methods are useful for finding corresponding points in this matrix, such as hard correspondence, soft correspondence, product manifold filter, graph faction, covariance, etc. After the refined registration (610) is completed, the registration process is completed (612).

As previously discussed, in some embodiments the actual position of the shims of part 114 may be determined by the seam position by the controller 108 using sensor imaging (e.g., using a scan point cloud) and/or relative to the CAD model. may differ from the seam position determined by the seam position (e.g., using a CAD model point cloud). In such cases, a scanning procedure (also referred to herein as a pre-scan) is useful to correct the determined seam position to closer or exactly match the actual seam position of the part 114. In the scanning procedure, a sensor 102 located on the robot 110 (herein referred to as an onboard sensor) performs a deliberative scan. In some cases, this scan may be performed using an initial motion and/or path plan generated by the controller 108 using a CAD model, scan, or a combination thereof. For example, sensor 102 may scan some or all areas of workspace 101 . While performing this initial motion and/or path planning, sensor 102 may capture observation images and/or data. Observation images and/or data may be processed by controller 108 to generate seam point cloud data. Controller 108 may use seam point cloud data when processing point cloud(s) 604 and/or 606 to modify seam positions. Controller 108 may also use seam point cloud data for path correction and motion planning.

In some embodiments, the registration techniques described above may be useful in addition to onboard sensor 102 to compare and match a Sim determined using sensor 102 to that identified by onboard sensor 102. By aligning the Sims in this way, the robot 110 (more specifically the head of the robot 110) is positioned relative to the actual Sim as desired.

In some embodiments, the pre-scan trajectory of robot 110 is identical to that planned for welding along the seam. In some such embodiments, the motion taken relative to the robot 110 during the pre-scan may be used to better visualize the seam or key geometry with the onboard sensors 102, to limit the probability or reduce instances of collisions, or to better visualize the seam or key geometry around the seam in question. Can be created separately to scan geometry.

In some embodiments, the pre-scan technique may include scanning more than a specific seam or seams, but rather other geometries of the part(s) 114. Scan data can be used to more accurately apply any or all of the techniques described herein (e.g., registration techniques) to find, locate, and detect the seam and ensure that the head of the robot 110 is positioned and moved along the desired seam. It can be useful.

In some embodiments, scanning techniques (e.g., scanning the actual seam using sensors/cameras mounted on the welding arm/welding head) may be useful for identifying gap variability information about the seam rather than position and orientation information about the seam. there is. For example, scanned images captured by the sensor(s) 102 of the robot 110 during a scanning procedure may be useful for identifying variability in gaps and adjusting the welding trajectory or path plan to account for such gaps. there is. For example, 3D point to 2D image pixels or a combination thereof may be useful for finding variable spacing between parts 114 to be welded. Variable gap finding is useful in some embodiments, where 3D points, 2D image pixels, or combinations thereof are useful for locating, identifying, and measuring variable sizes of various gaps between parts 114 to be welded together. In a temporary weld finder or a general weld finder, material deposits in the gap between previous welds or parts to be welded 114 may be identified using 3D points and/or 2D image pixels. Any or all of these techniques may be useful for optimizing welding, including path planning. In some cases, variability in spacing may be identified within a 3D point cloud generated using images captured by sensor 102. In still other cases, variability in spacing may be identified with scanning techniques performed while performing welding operations on a job (e.g., scanning the actual seam using a sensor/camera mounted on a welding arm/welding head). In either case, the controller 108 may be configured to dynamically adapt the welding command (e.g., welding voltage) based on the determined position and size of the gap. Seams can be precisely welded at variable intervals. Welding command adjustments may include adjusting one or more of welder voltage, welder current, electrical pulse duration, electrical pulse shape, and material feed rate.

In embodiments, user interface 106 may provide users with the option to view candidate Sims. For example, user interface 106 may provide part 114 and/or a graphical representation of candidate deliberations on part 114 . Additionally or alternatively, user interface 106 may group candidate Sims based on Sim type. As described above, controller 108 may identify the type of deliberation. For example, candidate shims identified as lap joints may be grouped under the label “lap joints” and presented to the user through user interface 106 under the label “lap joints.” Similarly, candidate seams identified as edge joints may be grouped under the label “Edge Joints” and presented to the user via user interface 106 under the label “Edge Joints.”

The user interface 106 may further provide the user with options for selecting candidate seams to be welded by the robot 110 . For example, each candidate seam of part 114 may be represented by a push button in user interface 106. When a user taps on a particular candidate Sim, the selection may be transmitted to the controller 108. The controller 108 may generate a command to cause the robot 110 to perform a welding operation on a specific candidate seam.

In some embodiments, the user may be provided with the option to update welding parameters. For example, user interface 106 may provide the user with a list of various welding parameters. Users can select specific parameters to update. Selected parameters can be changed through drop-down menus, text input, etc. This update may be sent to controller 108 so that controller 108 can update commands for robot 110.

In embodiments where system 100 is not provided with prior information (e.g., CAD model) of part 114, sensor(s) 102 may scan part 114. A representation of part 114 may be presented to the user via user interface 106. This representation of part 114 may be a point cloud and/or a mesh of point clouds containing projected 3D data of a scanned image of part 114 acquired from sensor(s) 102. A user can annotate seams to be welded in the representation through the user interface 106. Alternatively, controller 108 may identify candidate Sims in the representation of part 114. Candidate Sims may be presented to the user via user interface 106. The user can select the seam to be welded from among the candidate seams. User interface 106 may annotate the representation based on the user's selections. In some embodiments, annotated representations may be stored in database 112.

After one or more seams of part(s) 114 have been identified and modified to the extent possible using the techniques described above (or using other suitable techniques), the controller 108 provides guidance to the robot 110 during the subsequent welding process. Plan your route. In some embodiments, graph matching and/or graph search techniques may be useful in planning the path of robot 110. As described above, a particular seam identified may contain multiple points, and path planning techniques involve determining a different state of the robot 110 for each of these points along a given seam. The state of the robot 110 may include, for example, the position of the robot 110 within the workspace 101 and the specific configuration of the arm of the robot 110 with any number of degrees of freedom that may be applied. For example, in the case of a robot 110 with an arm having six degrees of freedom, the state of the robot 110 is the location of the robot 110 in the workspace 101 (e.g., the robot in a three-dimensional x-y-z space). (110) of the welding head position) as well as specific substates for the six degrees of freedom of each robot arm. Additionally, when the robot 110 transitions from the first state to the second state, it can change its position within the workspace 101, and in this case, the robot 110 follows a specific path within the workspace 101. (For example, along the seam being welded). Accordingly, specifying a series of states of the robot 110 necessarily entails specifying a path for the robot 110 to move within the workspace 101. Controller 108 may perform a pre-scan technique, or a variation thereof, after path planning is complete, and controller 108 may use information captured during the pre-scan technique to make any of a variety of appropriate adjustments (e.g., seam (along the X-Y-Z axis or coordinate system used to perform the actual welding).

7 is a schematic diagram 700 of a graph search technique by which a path plan for robot 110 may be determined (e.g., by controller 108). Each circle in the diagram 700 represents a different state of the robot 110 (e.g., a specific position of the robot 110 within the workspace 101 (e.g., the welding head position of the robot 110 in 3D space). ) and other configurations of the arm of the robot 110, as well as the location and/or configuration of fixtures supporting components such as positioners, clamps, etc. Each column 702, 706, 710 represents a different point along the seam to be welded. Accordingly, for a seam point corresponding to column 702, robot 110 may be in any of states 704A-704D. Likewise, for a seam point corresponding to column 706, robot 110 may be in any of states 708A-708D. Likewise, for the seam point corresponding to column 710, robot 110 may be in any of states 712A-712D. For example, if robot 110 is in state 704A when it is at a seam point corresponding to column 702, robot 110 will be in states 708A-708D for the next seam point corresponding to column 706. You can switch to either one. Similarly, upon entering states 708A-708D, robot 110 may then transition to any of states 712A-712D for the next seam point corresponding to column 710, and so on. In some embodiments, entering a particular state may make it impossible to enter another state. For example, entering state 704A allows the possibility of entering states 708A-708C but not 708D, while entering state 704B allows the possibility of entering states 708C and 708D without entering states 708A-708B. You can. The scope of the present disclosure is not limited to any particular number of seam points or any particular number of robot 110 states.

In some examples, to determine a path plan for the robot 110 using a graph search technique (e.g., according to the technique depicted in diagram 700), the controller 108 may operate on states 704A-704D. The shortest path from can be determined to a state (e.g., states 712A-712D) corresponding to the seam point (N). An objective function can be designed by the controller 108 by assigning a cost to each state and each transition between states. Controller 108 finds the path that results in the lowest possible cost value for the objective function. Multiple start and end points can be selected, allowing you to implement graph search methods such as Dijkstra's algorithm or A*. In some embodiments, brute force methods may be useful in determining an appropriate route plan. The brute force technique involves the controller 108 calculating all possible paths (e.g. via diagram 700) and selecting the shortest path.

The controller 108 can determine whether the state of each sim point is possible, which means that the controller 108 implements a state chain following the sequence of the sim point of the robot 110 and the structure of the work space 101. This means, at least in part, that it is possible to determine whether to cause a random collision with the robot 110 itself, or with a part of the robot 110 itself. To this end, the concept of realizing different states at different points in a deliberation could alternatively be expressed in the context of a deliberation with multiple waypoints. First, the controller 108 may separate the identified seam into a series of waypoints. The direction of the welding head connected to the robot 120 can be limited by three degrees of freedom (spatial/translational). Typically, constraints on the direction of the welding head of the robot 120 are provided for one or two rotational degrees of freedom for each waypoint, with the aim of producing a weld of some desired quality; Constraints are generally relative to the surface normal vector from the waypoint and the path of the welding seam. For example, the position of the welding head can be limited to about one or two axes of rotation about the there is. In some embodiments, these constraints may be limits or allowable ranges for angles. Those skilled in the art will recognize that the ideal or desired welding angle may vary depending on part or seam geometry, direction of gravity relative to the seam, and other factors. In some embodiments, the controller 108 may limit the 1F or 2F welding position to ensure that the seam is perpendicular to gravity for one or more reasons (such as finding a balance between welding and path planning for optimization purposes). Therefore, the position of the welding head can be maintained (limited) by each waypoint in an appropriate direction with respect to the seam. In general, the welding head is not constrained about the rotation axis (θ), which is coaxial with the welding head axis. For example, each waypoint may define the position of the welding head of the welding robot 120 so that the welding head at each waypoint is at a fixed position and orientation with respect to the welding seam. In some implementations, the waypoints are separated fine enough to make the movement of the welding head substantially continuous.

The controller 108 may divide each waypoint into multiple nodes. Each node can indicate a possible direction of the welding head at that waypoint. As a non-limiting example, the welding head may be unrestrained about a rotation axis coaxial with the axis of the welding head such that the welding head can rotate (e.g., 360 degrees) along the rotation axis θ at each waypoint. there is. Each waypoint can be divided into 20 nodes, where each node of each waypoint represents a welding head with a rotation increment of 18 degrees. For example, the first waypoint-node pair may represent a 0-degree rotation of the welding head, the second waypoint-node pair may represent an 18-degree rotation of the welding head, and the third waypoint-node pair may represent a 0-degree rotation of the welding head. It can indicate the rotation of the welding head at 36 degrees. Each waypoint can be divided into 2, 10, 20, 60, 120, 360, or any appropriate number of nodes. Subdivision of a node may represent a division of direction in one or more degrees of freedom. For example, the direction of the welding machine tip with respect to the waypoint can be defined by three angles. Each waypoint-node pair can be connected to define a welding path. Accordingly, the distance between waypoints and the offset between adjacent waypoint nodes can indicate the amount of translation and rotation of the welding head as it moves between node-waypoint pairs.

Controller 108 may evaluate each waypoint-node pair for adequacy of welding. For example, consider the non-limiting example of dividing a waypoint into 20 nodes. Controller 108 may evaluate whether the first waypoint-node pair representing the welding head maintained at 0 degrees is feasible. Stated differently, the controller 108 may evaluate whether the robot 110 will collide or interfere with a part, a fixture, or the welding robot itself when placed at a position and orientation defined by a waypoint-node pair. In a similar manner, controller 108 may evaluate whether a second waypoint-node pair, a third waypoint-node pair, etc. are feasible. Controller 108 may evaluate each waypoint similarly. In this way, all possible nodes for all waypoints can be determined.

In some embodiments, collision analysis described herein may be performed by comparing the 3D model of workspace 101 with the 3D model of robot 110 to determine whether the two models overlap, and optionally whether some or all of the triangles overlap. It can be done. If the two models overlap, the controller 108 may determine that there is a possibility of conflict. If the two models do not overlap, the controller 108 may determine that there is no possibility of conflict. More specifically, in some embodiments, controller 108 compares the models for each of a set of waypoint-node pairs (such as the waypoint-node pairs described above) and determines whether two of the waypoint-node pairs match the models for the subset. Or you could even possibly decide that they all overlap. For a subset of waypoint-node pairs for which model intersections are identified, the controller 108 may omit that subset's waypoint-node pairs from the planned route and may identify alternatives for those waypoint-node pairs. Controller 108 can repeat this process as needed until a collision-free path is planned. Controller 108 may use a flexible collision library (FCL) that includes various techniques for efficient collision detection and proximity calculation as a tool for collision avoidance analysis. FCL is useful for performing multiple proximity queries over different model representations and can be used to perform probabilistic collision identification between point clouds. Additional or alternative resources may be used in conjunction with or in place of FCL.

Controller 108 may generate one or more executable simulation (or evaluation, as both terms are used interchangeably herein) welding paths if physically feasible. The welding path may be a path used by the welding robot to weld the candidate seam. In some embodiments, the welding path may include all midpoints of the seam. In some embodiments, the welding path may include some waypoints but not all of the candidate deliberations. The welding path may include the motion of the robot and welding head as the welding head moves between each waypoint-node pair. Once a feasible path between a node-waypoint pair is identified, a feasible node-waypoint pair for the next sequential waypoint can be identified, if one exists. Those skilled in the art will recognize that many search trees or other strategies can be used to evaluate the space of feasible node-waypoint pairs. As discussed in more detail herein, cost parameters may be assigned or calculated for movement from each node-stop pair to a subsequent node-stop pair. Cost parameters may be associated with travel time, amount of movement (e.g., including rotation) between node-midpoint pairs, and/or simulated/expected welding quality produced by the welding head during movement.

If there are no nodes capable of welding for one or more waypoints, or if there is no path available to travel between the previous waypoint-node pair and the waypoint-node pair of a particular waypoint, the controller 108 may determine at least some additional waypoint-nodes. Alternative welding parameters can be determined so that the pair becomes available for welding. For example, if the controller 108 determines that none of the waypoint-node pairs for the first waypoint are feasible, making the first waypoint unweldable, the controller 108 determines the alternative weld angle, such as Alternative welding parameters can be determined so that waypoint-node pairs for at least some of the first waypoints can be welded. For example, controller 108 may remove or relax constraints on rotation about x and/or y axes. Similarly, the controller 108 may allow the weld angle to be changed by one or two additional rotational (angular) dimensions. For example, the controller 108 may divide non-weldable waypoints into two-dimensional or three-dimensional nodes. Each node can then evaluate the welding feasibility of the welding robot and the welding maintained at various welding angles and rotation states. The waypoint can approach the welding head so that additional rotations about the X and/or Y axes or other degrees of freedom do not cause the welding head to collide. In some implementations, the controller 108 determines that - if there is no node capable of welding to one or more waypoints or if there is no feasible path to travel between the previous waypoint-node pair and any of the waypoint-node pairs of the particular path. If one does not exist, the degrees of freedom provided by the positioner system can be used when determining possible paths between a waypoint - the previous waypoint-node pair and the waypoint-node pair of a particular waypoint.

Based on the generated welding path, the controller 108 can optimize the welding path for welding. (As used herein, optimal and optimized do not refer to determining the absolute best welding path, but generally refer to techniques that can reduce welding time and/or improve welding quality compared to less efficient welding paths.) Examples: For example, controller 108 may determine a cost function that finds local and/or global minima for the motion of robot 110. In general, the optimal welding path minimizes welding head rotation because welding head rotation can increase seam welding time or reduce welding quality. Accordingly, optimizing the welding path may include determining the welding path through the maximum number of waypoints with the minimum amount of rotation.

When evaluating the adequacy of welding at each partitioned node or node-path pair, the controller 108 may perform multiple calculations. In some embodiments, each of multiple calculations may be mutually exclusive of each other. In some embodiments, the first calculation is a kinematic feasibility calculation of whether an arm of the robot 110 of the employed welding robot can mechanically reach (or be in) a state defined by a node or node-waypoint pair. May include calculations. In some embodiments, in addition to the first calculation, a second calculation may also be performed by the controller 108, which may be mutually exclusive with the first calculation. A second calculation determines whether the arm of robot 110 encounters a collision (e.g., a collision with workspace 101 or a structure in workspace 101) when approaching a seam portion (e.g., the node or node-waypoint pair in question). This may include deciding whether or not to do so.

Controller 108 may perform the first calculation before performing the second calculation. In some examples, the second calculation may be performed if the result of the first calculation is positive (e.g., that the arm of robot 110 can mechanically reach (or exist in) the state defined by the node or node-waypoint pair. In some examples, if the result of the primary operation is negative (e.g., the arm of the robot 110 has mechanically reached the state defined by the node or node-waypoint pair (or If it is determined that it cannot exist, the secondary operation may not be performed.

Kinematic feasibility can be related to the type of robotic arm used. For the purposes of this description, welding robot 110 is assumed to include a six-axis robotic welding arm with a spherical wrist. A six-axis robot arm may have six degrees of freedom, three degrees of freedom in X-, Y-, and Z-cartesian coordinate systems, and three additional degrees of freedom due to the wrist-like nature of the robot 110. For example, the wrist-like characteristics of the robot 110 have a fourth degree of freedom in the wrist raising/lowering method (e.g., when the wrist moves in the +y and -y directions), the list ( wrist) resulting in a 5th degree of freedom in a lateral manner (e.g. a wrist moving in the -x and +x directions) and a 6th degree of freedom in rotation. In some embodiments, the welding torch is attached to the wrist portion of the robot 110.

To determine whether an arm of an employed robot 110 can mechanically reach (or exist in) a state defined by a node or node-waypoint pair (i.e., to perform a first calculation), the robot ( 110) can be mathematically modeled as shown in the example model 800 of FIG. 8. In some embodiments, controller 108 may solve the first three joint variables based on the wrist position and the remaining three joint variables based on the wrist orientation. It is noted that the torch is securely attached to the wrist. Therefore, the strain between the torch tip and the wrist is assumed to be fixed. A geometric approach (e.g., the law of cosines) can be used to find the first three combined variables (e.g., variables S, L, and U of 802, 804, and 806, respectively).

After the first three combined variables (i.e., S, L, and U) are successfully calculated, the controller 108 calculates the last three combined variables (i.e., R, B, and T in 808, 810, and 812, respectively) as follows: can be solved, for example, considering the wrist direction as ZYZ Euler angles. Controller 108 may take some offset of robot 110 into account. These offsets may need to be considered and accounted for due to inconsistencies in the Unified Robot Description Format (URDF) files. For example, in some embodiments, the value of a joint (e.g., an actual joint of robot 110) position (e.g., the joint's X-axis) may not match the value specified in the URDF file. Such offset values may be provided to the controller 108 in a table. In some embodiments, controller 108 may consider these offset values while mathematically modeling robot 110. In some embodiments, after robot 110 has been mathematically modeled, controller 108 determines whether an arm of robot 110 can mechanically reach (or exist in) a state defined by a node or node-path pair. can be determined).

As mentioned above, the controller 108 controls the parts 114 in the workspace 101, including the robot 110 itself, once the robot 110 is placed in the position and orientation defined by the waypoint-node pair. , it can be evaluated whether it collides with or interferes with the fixture 116 or anything else. Once the controller 108 determines a state in which the robot arm can exist, the controller 108 uses the second calculation to make the aforementioned evaluation (e.g., regarding whether the robot will collide with something in its environment). can do.

Figure 9 is a flowchart of a method 900 for performing autonomous welding according to various embodiments. More specifically, Figure 9 is a flow diagram of a method 900 for operating and controlling a welding robot (e.g., robot 110 of Figure 1) according to some examples. At step 902, method 900 collects image data of a workspace (e.g., workspace 101 of FIG. 1) using one or more sensors (e.g., sensor(s) 102 of FIG. 1). It includes the step of obtaining. Image data may include 2D and/or 3D images of the workspace. As described above, one or more parts to be welded, fixtures and/or clamps capable of holding the parts in a secure manner may be located in the workspace. In some embodiments, a point cloud may be generated from image data. For example, three-dimensional image data can be reconstructed and generated by overlapping the images. A point cloud can be created by combining 3D image data.

At step 904, method 900 includes identifying a set of points on the part to be welded based on sensor data, which may be an image. The point set can indicate the possibility of a seam being welded. In some embodiments, the neural network may perform pixel-wise segmentation on the image data to identify a set of points. Fixtures and clamps of image data can be classified through a neural network based on image classification. Portions of the image data associated with fixtures and/or clamps may be segmented such that those portions of the image data are not used to identify sets of points, which reduces the search space and thus frees up computational resources needed to identify sets of points to weld. It can be reduced. In such embodiments, point sets may be identified from different parts of the image data (eg, portions of the unsegmented image data).

At step 906, the method 900 includes identifying candidate Sims from the point set. For example, a subset of points within a point set can be identified as a candidate seam. A neural network can perform image classification and/or depth classification to identify candidate cores. In some embodiments, candidate seams may be located relative to the part. For example, to locate a candidate seam, the location and orientation of the candidate seam may be determined for the part.

Additionally, method 900 further includes verifying whether the candidate Sim is an actual Sim. As explained above, sensors can collect image data from multiple angles. For each image captured at a different angle, a confidence level indicating whether the candidate image determined at that angle is the actual image can be determined. If the reliability value exceeds a threshold based on views captured from multiple angles, the candidate Sim can be verified as an actual Sim. In some embodiments, method 900 also includes classifying candidate Sims into Sim types. For example, a neural network can determine whether a candidate seam is a butt joint, corner joint, edge joint, lap joint, tee joint, etc.

In some embodiments, after candidate seams are identified and verified, subsets of points may be clustered together to form continuous, contiguous seams. At step 908, the method 900 includes generating a welding command for the welding robot based on the candidate seam. For example, a welding command may be generated by tracing a path from one end of a subset of points to the other end of the subset of points. This way, a deliberation path can be created. In other words, a weld can be created by tracing this path with the welding head. Additionally, route planning can be performed based on identified and localized candidate simulations. For example, path planning can be performed based on a deliberative path that can be generated by clustering a subset of points.

In some embodiments, welding instructions may be based on the type of seam (e.g., butt seam, corner seam, corner seam, lap seam, T-shaped seam, etc.). In some embodiments, welding instructions may be updated based on input from the user through a user interface (e.g., user interface 106 of FIG. 1). The user can select a candidate seam to weld from among all available candidate seams through the user interface. Path planning can be performed on the selected candidate seam and welding commands can be created for the selected candidate seam. In some embodiments, a user can update welding parameters through a user interface. Welding commands can be updated according to updated welding parameters.

In this way, the welding robot can be operated and controlled by implementing the method 900 without prior information about the part to be welded (e.g., a CAD model). As the part is scanned to generate welding instructions, one or more candidate seams may be annotated (e.g., via a user interface) to a scanned image representation of the part. You can use annotated representations to define a 3D model of a part. The 3D model of the part can be stored in a database for subsequent welding of additional instances of the part.

Figure 10 is a flowchart of a method 1000 for performing autonomous welding according to various embodiments. More specifically, FIG. 10 is a flow diagram of a method 1000 for operating and controlling a welding robot (e.g., robot 110 of FIG. 1) according to some examples. At step 1002, the method 1000 includes identifying an expected direction and expected location of a candidate seam on a part to be welded based on a CAD model of the part. The expected direction and expected position can be determined using annotations provided to the CAD model by the user/operator. Additionally or alternatively, a controller (e.g., controller 108 of FIG. 1) may be operable to identify candidate seams based on model geometry. You can perform object matching to match components in a part to components in a CAD model. In other words, the expected location and direction of candidate deliberation can be identified based on object matching.

At step 1004, method 1000 uses one or more sensors (e.g., sensor(s) 102 of FIG. 1) to monitor a workspace (e.g., workspace 101 of FIG. 1). It includes acquiring image data. Image data may include 2D and/or 3D images of the workspace. As described above, the workspace may include one or more parts to be welded and fixtures and/or clamps to secure the parts in a secure manner. In some embodiments, a point cloud may be generated from image data. For example, three-dimensional image data can be reconstructed and generated by overlapping images. A point cloud can be created by combining 3D image data.

In some embodiments, the sensor is configured to perform partial scans to reduce processing time to generate welding commands. Put differently, instead of scanning the workspace from every angle, image data is collected from several angles (e.g., the angles at which candidate Sims are expected to be viewed). In this embodiment, the point cloud generated from image data is a partial point cloud. For example, creating a partial point crowd that does not include parts of the part that are marked as not containing seams to be welded in the model can reduce scanning and/or processing time.

At step 1006, method 1000 includes identifying candidate seams based on image data, point clouds, and/or partial point clouds. For example, controller 108 (Figure 1) may identify candidate Sims using the techniques described above.

At step 1008, the method 1000 includes identifying the actual location and actual orientation of the candidate Sim. For example, in step 1002 a first subset of points may be identified as modeled seams. At step 1006, a second subset of points may be identified as candidate seams. In some embodiments, a first subset of points and a second subset of points may be compared (e.g., using the registration technique described above with respect to FIG. 6). The first subset of points may be transformed to determine the actual location and direction of the candidate deliberation. In some embodiments, a comparison between the first subset of points and the second subset of points may help determine a tolerance for the first subset of points (e.g., expected location and expected direction of a candidate deliberation). there is. In this embodiment, a first subset of points may be allowed to transform based on a tolerance to determine the actual location and orientation of the candidate seam. Stated differently, the predicted position and expected direction of the candidate seam may be refined (in some embodiments, based on a tolerance) to determine the actual position and actual direction of the candidate seam. These transformation/refinement techniques may describe topography of the part surface that is not accurately represented in the CAD model (e.g., the CAD model of step 1002).

At step 1010, the method 1000 includes generating a welding command for the welding robot based on the actual location and actual orientation of the candidate seam. For example, route planning can be performed based on the candidate's actual location and actual direction.

As with method 900 of FIG. 9, once the actual location and actual orientation of a candidate Sim are identified, method 1000 may include verifying the candidate Sim using the techniques described above. However, unlike method 900, method 1000 may not require user interaction. This is because one or more seams to be welded may already be annotated in the CAD model. Accordingly, in some cases a welding robot may be operated and controlled by implementing method 1000 without user interaction.

Additionally, or as an alternative to the steps described above with respect to method 1000, a welding robot (e.g., robot 110 in FIG. 1) may be used to perform autonomous welding in the following manner. can be operated and controlled. The controller 108 of the robot 110, particularly the robot 11, determines the position of the part within the workspace (e.g., part position in a positioner) and generates a representation of the part (e.g., a point cloud representation). You can scan workspaces that contain . The controller 108 may be provided with an annotated CAD model of the part. The controller 108 may then determine the expected location and expected direction of the candidate threads on the part according to (or based on) a computer aided design (CAD) model of the part and a representation of the part. For example, the controller 108 may operate to identify candidate seams based on model geometry, i.e., matching components or features of the part representation (e.g., topographic features) to components or features of the CAD model. Object matching may be performed to match and the controller 108 may be operable to use this object matching to determine the expected location and expected direction of the candidate core on the part. Once the expected location and orientation are determined, the controller 108 may determine the actual location and actual orientation of the candidate seam based at least in part on the representation of the part. The actual location and orientation may be determined using the transformation/refinement technique described in step 1008.

Figure 11 is a flowchart of a method 1100 for performing autonomous welding according to various embodiments. More specifically, Figure 11 shows an example method 1100 for operation of a manufacturing robot 110 (Figure 1) in a robotic manufacturing environment. One or more parts 114 on which a manufacturing operation (e.g., welding) is to be performed is positioned using a fixture 116 to another fixture 116 (e.g., a positioner) in the manufacturing workspace 101 and/ Or it is assumed to be fixed. The method 1100 may begin after placing one or more parts 114. As an initial step, method 1100 includes scanning one or more portions (e.g., block 1110). Scanning may be performed by one or more sensors 102 (e.g., a scanner not connected to robot 110); The controller 108 may be configured to determine the location of a part within the manufacturing workspace 101 and identify one or more seams on the part using image data acquired from sensors and/or a point cloud derived from the image or sensor data. (Blocks 1112 and 1114). Parts and shims may be located and identified based on one of the techniques described with respect to FIG. 1 . Once the part and shim positions are determined, the controller 108 plots a path for the manufacturing robot 110 along the identified shims (block 1116). The plotted path may include optimized motion parameters of the manufacturing robot 110 to complete the welding without colliding with itself or anything else in the manufacturing workspace 101 . No human input is required to create optimized operating parameters of the manufacturing robot 110 to complete welding. The path/trajectory can be planned based on one of the path planning techniques described above.

The terms “position” and “orientation” are described as separate entities in the above disclosure. However, the term "position" when used in relation to a part means "the particular way in which the part is placed or arranged." The term "position" when used in relation to seams means "the specific way in which the seams of a part are positioned or oriented." Therefore, the location of the part/seam can essentially describe the direction of the part/seam. Therefore, ‘location’ can include ‘direction’. For example, the location may include the relative physical position or orientation (e.g., angle) of the part or candidate seam.

Unless otherwise specified, the words "approximately", "approximately" or "substantially" preceding a value mean +/- 10% of the stated value. Unless otherwise specified, two objects described as "parallel" are side by side and the distance between them is constant or varies by no more than 10%. Unless otherwise specified, two objects described as perpendicular intersect at an angle between 80 and 100 degrees. Modifications are possible in the described embodiments and other examples are possible within the scope of the claims.

Claims (26)

A computer-implemented method of generating instructions for a welding robot, comprising:
identifying the expected location of a candidate seam on the part based on a Computer Aided Design (CAD) model of the part to be welded;
scanning a workspace containing the part to create a representation of the part;
identifying candidate Sims for the part based on the expected location of the candidate Sims and the representation of the part;
determining the actual location of said candidate deliberation; and
Generating welding commands for the welding robot based at least in part on the actual location of the candidate seam;
Including,
Computer-implemented method. According to paragraph 1,
To determine the actual location of the candidate deliberation,
updating the expected position of the candidate deliberations based at least in part on the representation of the part.
Containing more,
Computer-implemented method. According to paragraph 1,
To determine the actual location of the candidate deliberation,
determining tolerance for the expected position of the candidate deliberation based at least in part on the representation of the part; and
refining the expected location of the candidate deliberation based at least in part on the tolerance;
Containing more,
Computer-implemented method. According to paragraph 1,
Identifying the expected location of the candidate thread further comprises matching a representation of a component in the CAD model to the component on the part to be welded,
Computer-implemented method. According to paragraph 1,
The CAD model includes the candidate deliberation annotation,
Computer-implemented method. According to paragraph 1,
Identifying the candidate Sim is
Identifying at least some points of a plurality of points on the part to be welded, i.e., a plurality of points forming a seam, based on the representation of the part.
Containing more,
Computer-implemented method. According to paragraph 1,
Identifying the candidate Sim is,
identifying at least some points of a plurality of points on a part to be welded, i.e., a plurality of points forming a seam, based on the representation of the part;
Verifying whether the candidate Sim is a Sim; and
Identifying the candidate Sim as a Sim type using a neural network;
Containing more,
Computer-implemented method. In clause 7,
To check whether the candidate Sim is a Sim,
Analyzing at least a subset of image data, wherein the subset of image data includes a plurality of images of the part, wherein each image from the plurality of images represents a portion of the part from a different angle. ) to capture, analyze;
For each image from the plurality of images: determining a confidence value that the candidate Sim is a real Sim; and
confirming whether the candidate Sim is the Sim based on a confidence value for each image from the plurality of images;
Containing more,
Computer-implemented method. A computer-implemented method of generating welding commands for a welding robot, the computer-implemented method comprising:
Obtaining, through a sensor, image data of the workspace containing the part to be welded;
identifying a plurality of points on the part to be welded based on the image data;
identifying candidate seams on parts to be welded from the plurality of points; and
generating welding instructions for the welding robot based at least in part on the candidate deliberation identification;
Including,
Computer-implemented method. According to clause 9,
Identifying the candidate Sim is
Further comprising localizing the shim to the part based on the image data,
Computer-implemented method. According to clause 9,
Identifying the candidate seam further comprises identifying a subset of points among the plurality of points forming the candidate seam.
Computer-implemented method. According to clause 9,
The welding command includes a welding path for a welding robot that welds the part,
Computer-implemented method. According to clause 9,
Using a neural network, classifying one or more objects included in the workspace as one or more of a clamp or fixture based on the image data.
Containing more,
Computer-implemented method. According to clause 13,
Classifying one or more objects includes:
Performing pixel-wise classification of the image data
Containing more,
Computer-implemented method. According to clause 14,
identifying a first portion within one or more images containing the image data including a representation of one or more objects based on the pixel-wise classification; and
identifying another candidate Sim from a second portion of the one or more images;
Containing more,
Computer-implemented method. According to clause 9,
Belonging to a group of seam types consisting of butt joints, corner joints, edge joints, lap joints, and tee joints. Further comprising: identifying the candidate Sim,
The welding command is generated based at least in part on the shim type.
Computer-implemented method. According to clause 9,
Identifying the candidate Sim is
confirming whether the candidate Sim is a Sim; and
Identifying the candidate Sim as a Sim type using a neural network;
Including,
Computer-implemented method. According to clause 17,
Checking whether the candidate is
Analyzing at least a subset of the image data, wherein the subset of image data includes a plurality of images of the part, each image from the plurality of images capturing a portion of the part from a different angle. to do;
For each image from the plurality of images: determining a confidence value whether the candidate Sim is a real Sim; and
confirming whether the candidate Sim is the Sim based on a confidence value for each image from the plurality of images;
Containing more,
Computer-implemented method. According to clause 9,
Identifying the candidate Sim is:
confirming that the shim is a shim;
Identifying the candidate mind as a deliberation type using a neural network; and
in response to confirming that the candidate seam is the seam, defining a cluster of a subset of points within the plurality of points, thereby forming the seam;
Containing more,
Computer-implemented method. According to clause 9,
The candidate Sim is a first candidate Sim, and the computer-implemented method includes:
displaying, via a user interface and to a user, a plurality of candidate seams capable of being welded, the plurality of candidate seams including a first candidate seam;
receiving, through the user interface, an indication that a second candidate seam from the plurality of candidate seams will be welded; and
updating the welding instruction for the welding robot so that the welding robot is instructed to weld the second candidate seam;
Containing more,
Computer-implemented method. According to clause 9,
Receiving changes in welding parameters from the user through a user interface;
and
updating the welding command for the welding robot based at least in part on the change in the welding parameter;
Containing more,
Computer-implemented method. According to clause 9,
scanning the part to be welded;
receiving user input from a user including instructions for welding one or more candidate seams on the part to be welded; and
defining a 3D model of the part to be welded based at least in part on the scan and the user input;
Containing more,
Computer-implemented method. According to clause 9,
scanning the part to be welded;
receiving user input from a user including instructions for welding one or more candidate seams on the part to be welded;
defining a 3D model of the part to be welded based at least in part on the scan and the user input; and
Saving the 3D model of the part to be welded in a database;
Containing more,
Computer-implemented method. A computer-implemented method of generating instructions for a welding robot, the computer-implemented method comprising:
By scanning the workspace containing the above parts:
- determining the location of the part within the workspace; and
- creating a representation of said part;
determining the expected position of a candidate for a part to be welded according to the representation of the part and a Computer Aided Design (CAD) model of the part; and
determining an actual location of the candidate deliberation based at least in part on the representation of the part;
Including,
Computer-implemented method. According to clause 24,
The CAD model includes the candidate deliberation annotation,
Computer-implemented method. According to clause 25,
Determining the actual location of the candidate deliberation includes updating the expected position of the candidate deliberation based at least in part on the representation of the part.
Computer-implemented method.