EP4063081A1 - Verfahren zum ermitteln von steuerungsdaten für eine greifeinrichtung zum greifen eines gegenstands - Google Patents

Verfahren zum ermitteln von steuerungsdaten für eine greifeinrichtung zum greifen eines gegenstands Download PDF

Info

Publication number
EP4063081A1
EP4063081A1 EP21163930.7A EP21163930A EP4063081A1 EP 4063081 A1 EP4063081 A1 EP 4063081A1 EP 21163930 A EP21163930 A EP 21163930A EP 4063081 A1 EP4063081 A1 EP 4063081A1
Authority
EP
European Patent Office
Prior art keywords
model
gripping
data
image
determination
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP21163930.7A
Other languages
German (de)
English (en)
French (fr)
Inventor
Ralf Gross
Ingo Thon
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Siemens AG
Original Assignee
Siemens AG
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Siemens AG filed Critical Siemens AG
Priority to EP21163930.7A priority Critical patent/EP4063081A1/de
Priority to PCT/EP2022/055162 priority patent/WO2022199994A1/de
Priority to CN202280036828.5A priority patent/CN117355394A/zh
Priority to EP22712899.8A priority patent/EP4284604A1/de
Priority to US18/283,226 priority patent/US20240253232A1/en
Publication of EP4063081A1 publication Critical patent/EP4063081A1/de
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1602—Programme controls characterised by the control system, structure, architecture
    • B25J9/161—Hardware, e.g. neural networks, fuzzy logic, interfaces, processor
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1612—Programme controls characterised by the hand, wrist, grip control
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1628—Programme controls characterised by the control loop
    • B25J9/163—Programme controls characterised by the control loop learning, adaptive, model based, rule based expert control
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
    • B25J9/1664—Programme controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/39—Robotics, robotics to robotics hand
    • G05B2219/39484—Locate, reach and grasp, visual guided grasping
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/39—Robotics, robotics to robotics hand
    • G05B2219/39514—Stability of grasped objects
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/39—Robotics, robotics to robotics hand
    • G05B2219/39536—Planning of hand motion, grasping

Definitions

  • US 2020/0164531 A1 for example a system for gripping objects, which comprises a recognition device for recognizing the identity, a location and an orientation of an object and a selection system for selecting a gripping point for a respective object.
  • the gripping point can be selected by a user, for example.
  • a corresponding system can be trained with the information regarding gripping points selected by users in order to automate the determination of gripping points using this information.
  • Such a method or system can, for example, enable safer, more reliable, faster and/or more highly automated gripping compared to the prior art.
  • the method described is based on the finding that when analyzing the image of the object, for example to identify the object or to determine further object parameters, not all possible orientations are taken into account of the object must be taken into account, but it can be assumed that the object is in one of its possible stable positions, e.g. on a flat surface. This considerably limits the number of possible positions of the object in the corresponding image analysis. Therefore, the limitation of the algorithms used for the image analysis to possible stable positions of the object allows a significant reduction in the analysis effort, since a large part of possible positions of the object can be ignored. In this way, corresponding analysis methods for identifying the object and/or determining its position can run more easily and/or faster. This also further simplifies the determination of a corresponding gripping point for this object as compared to the prior art.
  • An object can be any three-dimensional structure with a fixed external shape.
  • Objects can be, for example, pieces of material, components, modules, devices or the like.
  • the image of the object can be captured, for example, by means of a camera, a scanner (for example a laser scanner), a distance radar or a similar device for capturing three-dimensional objects.
  • the captured image can advantageously be a two-dimensional image of the object or a two-dimensional image which includes an image of the object.
  • the captured image can also be or include a three-dimensional representation of the object.
  • the at least one item parameter can be, for example, an identifier relating to the item, ID information regarding the object and/or also a name or a short description or description of the object.
  • the identifier can, for example, be designed and set up in such a way that it allows the object to be identified.
  • ID information can be a unique designation, identifier or the like for the respective object or can include such information.
  • the at least one object parameter can include, for example, a location, position or the like with respect to the detected object.
  • a location can be given, for example, by characteristic points and the location of the characteristic points and/or can also be defined, for example, by a location or position of a virtual boundary frame on the captured image, a so-called “bounding box”.
  • a location or position can also be given, for example, by the location of a central point of the object (e.g. a center of gravity) and a rotation angle in relation to a defined or definable standard position.
  • the at least one object parameter can also include a property of the object, such as a color, a material or a combination of materials or comparable properties.
  • the determination of the at least one object parameter for the detected object relates to the object represented on the detected image.
  • the at least one object parameter is therefore assigned to the object represented on the captured image, just as it is represented on the captured image.
  • a gripping device can be designed and set up, for example, as a robot or robot arm with a corresponding gripper for grasping or mechanically fixing the object.
  • a gripper can, for example, be designed like tongs, have one or more suction devices and/or allow or support the fixing of an object to be gripped with electromagnetic forces.
  • a robot or robot arm can be designed and set up, for example, as a 6-axis robot or 6-axis industrial robot or robot arm. Furthermore, such a robot or robot arm can be designed and set up, for example, as a Cartesian robot or with a corresponding gripper.
  • the control data for the gripping device for gripping the object at the at least one gripping point are data that must be supplied to a control device of the gripping device or to a control device for the gripping device so that, for example, a gripper of the gripping device mechanically picks up the object at the at least one gripping point fixed.
  • a control device for the gripping device can be designed and set up, for example, as a so-called robot controller, a programmable logic controller, a computer or a similar control device.
  • the control data can include, for example, the coordinates of a point in space for the gripper and an orientation of the gripper, which the gripper must assume in order to be able to grip the object. Furthermore, the control data can also contain the coordinates of at least one Be gripping point of the object in space or include these coordinates.
  • control device for the gripping device can then calculate the necessary movement of the gripping device and of the gripper in a known manner.
  • coordinates in space is understood to mean, for example, a coordinate system in which both the object to be gripped and the gripping device are located.
  • Control data can then be, for example, coordinates of the at least one gripping point and/or the at least one model gripping point transformed into this real space.
  • a position of the object in space can also be taken into account when calculating the control data, for example to allow unimpeded access of a gripper of the gripping device to the at least one gripping point.
  • the control data for the gripping device can therefore be determined, for example, as follows: after capturing the image of the object, a position of the object is determined as part of the determination of the at least one object parameter. After determining the at least one model grip point from the 3D model of the object, the coordinates of the at least one model grip point can then be converted into corresponding coordinates of the at least one grip point of the object using the position information of the real object. Using these coordinates of the at least one gripping point of the real object and the information about the position of the object With regard to the gripping device, the control data for the gripping device can then be determined according to the present description.
  • Stable positions of an object for example on a surface (e.g. a substantially horizontal plane or surface), are those one or more positions of the object in which the object can be located without moving from rest by itself ( e.g. to tilt or roll).
  • Such a stable position for the object can be determined, for example, by initially moving it towards the surface (e.g. dropping it onto this surface) and then waiting until the object stops moving. By performing this process multiple times with different initial conditions, the stable positions of an object can be determined in this way. For example, the object can be moved to a corresponding surface under a wide variety of initial conditions (e.g. thrown onto it or dropped onto it). Then you wait until the object stops moving. Subsequently, the assumed stable position is then recorded accordingly.
  • a stable position can be recorded, defined and/or stored, for example, by registering the position taken. This registration can take place, for example, via an image recording, a 3D recording and/or a detection of one or more coordinates of the object in the stable position. Furthermore, the detection of the stable position can also include the allocation of an identifier that is unique to the stable position of the object.
  • All of the recorded position data for a specific object must be assigned to a specific stable position that can be converted into one another by displacement and/or rotation about a surface normal of the support surface on which the object is lying.
  • a specific identifier for the associated stable position can then be assigned to all of these positions, for example.
  • the determination of the stable positions of an object can, for example, take place in a semi-automated manner, in that a specific object is selected by a user and then, for example, is dropped or thrown onto a surface under a wide variety of initial conditions. It then waits until the object has come to rest. An image of the object is then captured and an automatic image analysis method is used to check whether the position of the captured object can be transformed into an already captured position of the object by displacement and/or rotation about a surface normal of the surface. If this is the case, then the identifiers for this stable position are automatically assigned to the image that has now been taken.
  • the determination can also take place automatically, for example. This can be done, for example, by physically simulating a falling motion of a 3D model of an object a surface is used. As part of this simulation, you then wait until the movement of the 3D model of the object has come to rest. A corresponding image of the now stationary 3D model of the object is then recorded and an identifier for a stable position is assigned to the image according to the method already explained above. This process can now be repeated automatically with randomly selected initial conditions until no more new stable positions are found or a sufficient number of images is available for each of the stable positions found.
  • a sufficient number of images can be given, for example, if 2, 10 or even 50 images are available for each stable position. Furthermore, it can be specified that a new stable position is no longer found if no new stable position is found after 10, 50 or even 100 attempts.
  • the images assigned to a specific stable position can also be correspondingly stored in a database.
  • This database can then be used, for example, to assign a specific stable position to a newly detected object by comparing it with these images.
  • the images can be used in order to train a corresponding neural network, which can then be used as part of the image evaluation for newly recorded images of objects.
  • a corresponding neural network for example, a recorded image of a stationary object on a surface can then be supplied to the neural network.
  • the result of the evaluation by the neural network can then at least, among other things be an identifier for the stable position occupied by this object.
  • An advantage of using the stable positions in a method according to the present description is, for example, that only the relatively few stable positions compared to all possible positions have to be considered when identifying, determining the position and/or determining the gripping point. This can reduce the computational effort involved in determining the position, identification and/or determination of the grasping point, often even considerably.
  • the information relating to a stable position of an object can, for example, be designed and set up as one or more images of the object, with the object being in the stable position in each of the images.
  • An identifier for this stable position can also be assigned to each of the images.
  • the information relating to at least one possible stable position can then be designed and set up, for example, as one or more images of an object, in which each image is assigned an identifier for the stable position in which the object is located in this image.
  • the information relating to at least one possible stable position of an object can be designed and set up as an ML model (“Machine Learning” model), the ML model using a machine learning method to ascertained information relating to the at least one possible Stable position has been trained and/or set up.
  • ML model Machine Learning
  • a possible design and a corresponding handling of such ML models will be discussed in more detail below.
  • a such an ML model can be designed and set up as a neural network.
  • information relating to the at least one possible stable position of the object is understood to mean any use of such information in the context of a calculation or determination of data or information.
  • a collection of comparison information or comparison images can be used, which show the object or several objects in their respective stable positions.
  • Corresponding further data can then be derived on the basis of information assigned to the respective images.
  • information assigned to the respective images can, for example, be information about the object shown on it, the stable position it has assumed, or also a spatial position of the object in the image.
  • a machine learning model can be trained with the corresponding data mentioned above, for example with images that show one or more objects in their respective stable positions, with the images each having further information such as an identifier about the assumed stable position, an identifier of the object shown on it and/or also information about a real spatial position assumed on the image.
  • ML model trained in this way, for example, the image of a recorded object can then be evaluated.
  • image evaluation is also an example of using information regarding the at least one possible stable position of an object.
  • each determination of the at least one object parameter for the detected object can be made using information regarding the at least one possible stable position of the object.
  • the at least one object parameter for the recorded object can be determined in such a way that an identifier for its stable position, a distance angle to a fixed zero point and/or a rotation angle with respect to a surface normal to the placement surface of the object is determined.
  • transformation data for a transformation of the 3D model, including the model grip points specified there can then be specified in the real object. This transformation data can then be used, for example, to determine the at least one gripping point of the object.
  • the control data for the gripping device can then also be determined in a comparable manner from the transformation data and other information with regard to, for example, the accessibility of the object.
  • this refinement of the method has the advantage that the use of the model gripping point from the 3D model of the identified object enables a simplified determination of the at least one gripping point of the detected object.
  • gripping points that were already defined during the construction of the object can later be used to grip a corresponding real object.
  • one or more model gripping points for the 3D model can be determined automatically. In this configuration, for example, no user intervention is necessary, which further simplifies the determination of the at least one gripping point.
  • a suitable 3D model of this object can be selected, for example after capturing an image and, for example, identifying an object.
  • Data required for gripping this object can then be determined, for example, in such a way that a corresponding model grip point is taken from the 3D model and a grip point on the real object is then determined from this.
  • This can, for example, be designed and set up in such a way that by comparing a detected position of the object with the 3D model, control data for a gripping device are determined in such a way that transformation data can be used to convert the coordinates of a model gripping point into those of a corresponding gripping point on the object .
  • the method described has the advantage, for example, that by using the model gripping point from the 3D model, a method for gripping the corresponding object can be simplified compared to the prior art. For example, by comparing the recognized or identified object with the 3D model of the object, real gripping points of the object can be determined. In this way, the appropriate gripping point can be determined in each case for the most varied of positions and positions of the object in space using only one template, namely the 3D model for the object.
  • the method has the further advantage that it also enables automated determination of gripping points for a detected object. Due to the fact that corresponding gripping points can already be provided or defined in the 3D model of the object, it is possible by comparing the 3D model with a detected position of the object to convert the intended model gripping points into real gripping points of the detected object without a further user intervention, for example a marking or input of possible gripping points by a user, is necessary.
  • a 3D model can be any digital depiction or representation of the object that at least substantially represents the external form.
  • the 3D model represents the external shape of the object.
  • the 3D model can also contain information about the internal structure of the object, mobility of components of the object or information about the functionalities of the object.
  • the 3D model can, for example, be stored in a 3D file format, for example created with a 3D CAD software tool.
  • 3D CAD software tool examples include SolidWorks (file format: .sldprt), Autodesk Inventor (file format: .ipt), AutoCAD (file format: .dwg), PTC ProE/Creo (file format: .prt), CATIA (file format: .catpart) , SpaceClaim (file format: .scdoc) or SketchUp (file format: .skp).
  • file formats can be, for example: .blend (Blender file), .dxf (Drawing Interchange Format), .igs (Initial Graphics Exchange Specification), .st1 (stereolithography format), .stp (Standard for the Exchange of Product Model Data), .sat (ACIS text file) or .wrl, .wrz (Virtual Reality Modeling Language).
  • file formats can be used in which material properties of the object such as specific weight, color, material and/or the like of the object or its components are also stored. By using such 3D models, for example, physically correct simulations can be carried out for the object, e.g. to determine one or more stable positions of the object on a surface.
  • the 3D model for the object can be selected, for example, using ID information for the object, with the at least one object parameter comprising this ID information.
  • the 3D model of the object can also be selected, for example, using information relating to the at least one possible stable position of the object.
  • ID information for the object can be determined by means of one of the method sequences explained above using information relating to the at least one possible stable position of the object and based on this ID information, a corresponding 3D model can then be selected, for example, from a corresponding database.
  • the 3D model can be taken from a database for 3D models of various objects, for example, it being possible for the selection from this database to be made using the ID information identified above, for example.
  • the 3D model can also be selected by a user. The user can, for example, select the 3D model from a number of available 3D models.
  • a model grip point can be given, for example, by the coordinates of a specific point or area on an outer surface of the 3D model. Furthermore, the model gripping point can also be given by a gripping surface. Such a gripping area can be defined, for example, by a description of a boundary line of the gripping area on an outside of the object.
  • the at least one model gripping point it can be provided, for example, that one or more gripping points or gripping areas are already marked in the 3D model of the object and the coordinates of these one or more gripping points or gripping areas are then taken from the 3D model. In an advantageous embodiment, this determination of the at least one model gripping point can take place automatically. Furthermore, the at least one model gripping point can also be determined by a user or partially automatically, supported by a user.
  • the model gripping point in the 3D model can be determined automatically, for example, by using a mechanical simulation or model analysis of the 3D model to determine corresponding gripping points. These can then, for example, continue to be recorded or marked directly in the 3D model.
  • model grip points can also already be provided and/or identified as part of a construction of the 3D model.
  • Corresponding model grip points can also be added to a 3D model later, for example, by marking and/or identifying corresponding areas of the 3D model as grip points, for example. This can be done manually, for example. Alternatively, this can also be done automatically, in that corresponding gripping points or gripping areas are determined, e.g. by a mechanical simulation of the object or also predetermined criteria for gripping areas. Such predetermined criteria can be, for example, the position of a center of gravity of the object, the presence of planar areas on the object, and mechanical strength values of different areas of the object.
  • a method according to the present description can be designed and set up in such a way that the use of information regarding at least one possible stable position is designed and set up as the use of an ML model, the ML model being determined using a machine learning method information regarding the at least one possible stable position has been trained and/or set up.
  • a machine learning process is understood to mean, for example, an automated (“machine”) process that does not generate results using rules that have been defined in advance, but in which regularities are (automatically) identified from many examples by means of a machine learning algorithm or learning process, on the basis of which statements are then made are generated via data to be analyzed.
  • machine automated
  • Such machine learning methods can be designed and set up, for example, as a supervised learning method, a partially supervised learning method, an unsupervised learning method or also a reinforcement learning method.
  • machine learning methods are e.g. regression algorithms (e.g. linear regression algorithms), generation or optimization of decision trees (so-called “decision trees"), learning methods or training methods for neural networks, clustering methods (e.g. so-called “k-means clustering "), methods of learning or generation of support vector machines (“Support Vector Machines” (SVM)), methods of learning or generation of sequential decision models, or methods of learning or generation of Bayesian models or networks.
  • machine learning model The result of such an application of such a machine learning algorithm or learning method to specific data is referred to as a “machine learning” model or ML model, in particular in the present description.
  • Such an ML model represents the digitally stored or storable result of the application of the machine learning algorithm or learning method to the analyzed data.
  • the generation of the ML model can be designed and set up in such a way that the ML model is newly created by using the machine learning method or an already existing ML model is changed or adapted by using the machine learning method.
  • ML models are results of regression algorithms (e.g. a linear regression algorithm), neural networks ("neural networks”), decision trees (“decision tree”), the results of clustering methods (including e.g. the clusters obtained). or Cluster Categories, Definitions and/or Parameters), Support Vector Machines (SVM)), Sequential Decision Models or Bayesian Models or Networks.
  • regression algorithms e.g. a linear regression algorithm
  • neural networks e.g. a linear regression algorithm
  • decision trees decision trees
  • clustering methods including e.g. the clusters obtained.
  • SVM Support Vector Machines
  • Sequential Decision Models or Bayesian Models or Networks e.g., Bayesian Models or Networks.
  • Neural networks can be, for example, so-called “deep neural networks”, “feed forward neural networks”, “recurrent neural networks”; “Convolutional Neural Networks” or “Autoencoder Neural Networks”.
  • deep neural networks feed forward neural networks
  • recurrent neural networks recurrent neural networks
  • Convolutional Neural Networks or “Autoencoder Neural Networks”.
  • the application of corresponding machine learning methods to neural networks is often also referred to as “training” the corresponding neural network.
  • Decision trees can be designed and set up, for example, as so-called “iterative dichotomizers 3” (ID3), classification or regression trees (CART) or also so-called “random forests”.
  • ID3 iterative dichotomizers 3
  • CART classification or regression trees
  • random forests random forests
  • a neural network is understood, at least in connection with the present description, as an electronic device which comprises a network of so-called nodes, with each node usually being connected to a number of other nodes.
  • the nodes are also referred to, for example, as neurons, units or units.
  • Everyone has Nodes have at least one input and one output connection.
  • Input nodes for a neural network are nodes that can receive signals (data, stimuli, patterns or the like) from the outside world.
  • Output nodes of a neural network are understood to mean those nodes which can pass on signals, data or the like to the outside world.
  • So-called “hidden nodes” are understood to mean those nodes of a neural network which are designed neither as input nor as output nodes.
  • the neural network can be embodied, for example, as a so-called deep neural network (DNN).
  • DNN deep neural network
  • Such a “deep neural network” is a neural network in which the network nodes are arranged in layers (whereby the layers themselves can be one-, two- or even higher-dimensional).
  • a deep neural network includes at least one or two so-called hidden layers, which only include nodes that are not input nodes or output nodes. That is, the hidden layers have no connections to input signals or output signals.
  • Deep learning is understood to mean, for example, a class of machine learning techniques that utilizes many layers of nonlinear information processing for monitored or unsupervised feature extraction and transformation as well as for pattern analysis and classification.
  • the neural network can also have what is known as an auto-encoder structure, for example.
  • an auto-encoder structure can be suitable, for example, for a dimensionality of the data and, for example, to recognize similarities and similarities.
  • a neural network can also be in the form of a so-called classification network, for example, which is particularly suitable for dividing data into categories.
  • classification networks are used, for example, in connection with handwriting recognition.
  • a further possible structure of a neural network can be, for example, the embodiment as a so-called “deep belief network”.
  • a neural network can, for example, also have a combination of several of the structures mentioned above.
  • the architecture of the neural network can include an auto-encoder structure in order to reduce the dimensionality of the input data, which can then be combined with another network structure in order, for example, to recognize or correct peculiarities and/or anomalies within the data-reduced dimensionality. to classify the data-reduced dimensionality.
  • the values describing the individual nodes and their connections can be stored, for example, in a value set describing the neural network.
  • a value set then represents, for example, an embodiment of the neural network. If such a value set is stored after the neural network has been trained, then, for example, an embodiment of a trained neural network is thereby stored. For example, it is possible to train the neural network in a first computer system with appropriate training data, the appropriate To then store the set of values which is assigned to this neural network and to transfer it to a second system as an embodiment of the trained neural network.
  • a neural network can usually be trained by determining parameter values for the individual nodes or for their connections using a wide variety of known learning methods by entering input data into the neural network and analyzing the then corresponding output data from the neural network.
  • a neural network can be trained with known data, patterns, stimuli or signals in a manner that is known per se today, in order to then be able to use the network trained in this way subsequently, for example, to analyze further data.
  • the training of the neural network is understood to mean that the data with which the neural network is trained is processed in the neural network using one or more training algorithms in order to calculate so-called bias values ("bias”), weighting values (“weights”). ) and/or transfer functions (“Transfer Functions”) of the individual nodes of the neural network or of the connections between two nodes within the neural network.
  • bias bias
  • weight weighting values
  • Transfer Functions transfer functions
  • supervised learning To train a neural network, for example according to the present description, one of the methods of what is known as “supervised learning” can be used, for example. In this case, results or capabilities associated with this data are trained on a network by training with corresponding training data.
  • a supervised learning method can be used, for example, in order to train a neural network, for example, in the stable positions of one or more objects. This can be done, for example, by “training” an identifier for the assumed stable position (the above-mentioned “result”) into an image of an object in a stable position.
  • unsupervised learning can also be used to train the neural network.
  • Such an algorithm creates a model for a given set of inputs, for example, which describes the inputs and enables predictions to be made from them.
  • clustering methods with which the data can be divided into different categories if they differ from one another, for example by characteristic patterns.
  • Supervised and unsupervised learning methods can also be combined when training a neural network, for example when parts of the data are assigned trainable properties or skills, while another part of the data is not.
  • methods of so-called reinforcement learning can also be used for training the neural network, at least among other things.
  • training that requires a relatively high computing power of a corresponding computer can take place on a high-performance system, while further work or data analysis with the trained neural network can then definitely be carried out on a low-performance system.
  • Such further work and/or data analysis with the trained neural network can take place, for example, on an edge device and/or on a control device, a programmable logic controller or a modular programmable logic controller or other corresponding devices according to the present description.
  • a collection of images can be used, for example, which shows a specific object in a stable position on a flat surface, with each of the images being assigned an identifier for the stable position it assumes.
  • the ML model is then trained using this collection of images.
  • a stable position of this object can then subsequently be determined using the trained ML model on a captured image of the object.
  • each of the images can show the object in one of its stable positions on a given or definable surface, in particular on a flat surface or on a substantially horizontal, flat surface.
  • the collection of images then contains, for example, a plurality of images of the object, each in one of its stable positions and also at different angles of rotation in relation to a defined or definable initial position on a surface.
  • the rotation can be defined, for example, to a surface normal of a surface on which the object lies in one of its stable positions.
  • the ML model can be designed, for example, as a neural network, with the machine learning method in this Case can be, for example, a supervised learning method for neural networks.
  • the collection of images used to train the ML model can show different objects in different stable positions, with each of the images being assigned both ID information relating to the object depicted and an identifier relating to the stable position assumed on it can be.
  • the collection of images can be designed and set up such that each of the images shows one of the objects in one of its stable positions on a given or definable surface, in particular on a flat surface or on a substantially horizontal, flat surface.
  • the collection of images can then contain, for example, a plurality of images of the various objects, each in one of its stable positions and in each case at different angles of rotation with respect to a defined or definable initial position.
  • the rotation can be defined with respect to a surface normal of a surface on which the object lies in one of its stable positions.
  • the ML model can be embodied as a neural network, for example, with the associated machine learning method also being able to be a method of monitored learning for neural networks here, for example.
  • the at least one gripping point of the object is determined using the model gripping point using a further ML model, with the further ML model using a machine learning method on transformation data with regard to possible transformations a predetermined or predeterminable initial position was trained or set up in possible positions of the object.
  • the determination of the at least one gripping point of the object takes place with the aid of the application of an image evaluation method to the captured image of the object.
  • the further ML model can be designed and set up in accordance with an ML model according to the present description.
  • the further ML model can be designed and set up, for example, as a transformation ML model which is designed and set up to determine transformation data from a defined or definable initial position of the object to the position of the detected object in the real world .
  • the further ML model can then also be designed and set up as a neural network or else a "random forest" model.
  • the further ML model can also be designed and set up as a so-called "deep learning" neural network, for example.
  • the use of the additional ML model to determine the at least one gripping point can be used, for example, as an application of the captured image of the object to the further ML model.
  • the result of such an application can be, for example, transformation data for a transformation of the predefined or predefinable starting position of the object into the assumed position of the object captured by the camera.
  • input data for the application of the further ML model can also be the following data, for example: recognition data relating to the object, an identifier relating to the stable position in which the object is located and/or a rotation angle with respect to the bearing surface with respect to a fixed one or definable starting position.
  • Identification data for the object can be, for example, ID information relating to the object, description data for a virtual frame around the object and/or scaling data.
  • Output data of such a further ML model can then be, for example, transformation data for the above-mentioned transformation of the object from the predefined or predefinable initial position into the real position of the object on the storage area.
  • the predefined or predefinable starting position can be, for example, a predefined or predefinable position of a 3D model of the object in a corresponding 3D software.
  • Such software can be, for example, a corresponding CAD program or a so-called 3D modeling program.
  • the further ML model can be trained, for example, in that, in a collection of images, each image has either the transformation data of a predefined or predefinable position of a 3D model represented on the image object is assigned to the position of the object in the picture.
  • the general positioning data relating to a selected coordinate system can be assigned to each image for an object represented on it. The assignment of said data to the images of the image collection, for example as explained elsewhere in the present description, can take place, for example, automatically in a simulation environment or also manually.
  • a method commonly used for such applications can be used as the above-mentioned image evaluation method for determining the at least one gripping point of the object.
  • Examples of such methods are, for example, the so-called SURF, SIFT or BRISK method.
  • a method according to the present description can also be designed and set up in such a way that the determination of the at least one object parameter also includes the determination of position data of the object.
  • the position data also includes information regarding a stable position assumed by the object.
  • the determination of the control data for the gripping device is further simplified if the determination of the at least one object parameter for the detected object already includes the determination of the position data of the object. As a result, additional data is determined in this context, which can possibly accelerate and/or simplify the determination of the control data.
  • the position data also includes information regarding a stable position assumed by the object, this further simplifies the determination of the control data for the gripping device.
  • knowledge of the stable position of an object facilitates, accelerates and/or simplifies the analysis of the data, for example with regard to an identity, a location or position in space, a rotation angle and/or a Center of gravity of the object.
  • Position data of the object can include, for example, data relating to a position of the object in space.
  • data relating to a position of the object can include, for example, coordinates of one or more reference points of the object in space.
  • the data relating to a position may include at least coordinates relating to at least three reference points of the object.
  • data relating to a position can also include, for example, coordinates of a reference point of the object and one or more angles of rotation or angles of rotation with respect to one or more axes.
  • the position data of the object can include, for example, coordinates of a reference point of the object, information regarding a stable position of the object and at least one angle of rotation or rotation of the object, in particular exactly one angle of rotation or rotation.
  • a turning or rotation angle can be defined, for example, with respect to a surface on which the object is located.
  • the angle of rotation or rotation can be defined, for example, with respect to an axis perpendicular to this surface.
  • the position data of the object include data that describe a virtual frame around the object.
  • the position data can then additionally include information regarding a stable position of the object and/or at least one angle of rotation or rotation of the object, in particular exactly one angle of rotation or rotation.
  • the position data of the object can consist of exactly the data mentioned above.
  • Such a virtual frame around the object can be defined, for example, as a rectangular contour which encloses at least a predetermined part of the object, in particular encloses the entire object.
  • a rectangular frame for example, any polygonal or also a generally shaped, in particular a regularly shaped, contour can be used (e.g. also a circle or an ellipse).
  • the position data can be determined, for example, using the image of the object and a corresponding image evaluation method. In this case, for example, a position of the object on the image is determined with the aid of the image. A position of the object on a real shelf can then be calculated or determined, e.g. by calibrating the position of the camera and the viewing angle of the camera.
  • the position data of the object can be given, for example, by the data of a virtual frame enclosing the object, for example together with a rotation angle of the object and an identifier of a stable position of the object.
  • the determination of the at least one object parameter, the determination of ID information, the determination of the position data, the determination of a position of the object, the determination of a virtual boundary frame around the object and/or the determination of a dated Subject occupied stable position using the information regarding at least one possible stable position.
  • the use of information regarding the at least one possible stable position of the object can determine the at least one object parameter, ID information, position data, a position of the object, a virtual bounding box of the object and/or or simplify the determination of a stable position occupied by the object.
  • the method of gripping the object is further simplified. Furthermore, the method can also be accelerated in this way, since possible arithmetic processes for calculating the variables mentioned above can be accelerated as a result.
  • the at least one object parameter or also the respective at least one further object parameter can be, for example, an identifier relating to the respective object or ID information relating to the respective object.
  • This refinement permits a further simplification of the method, since in this way a specific object can also be grasped when other objects are also located in the image field of the camera.
  • the method can, for example, be developed in such a way that an image of the object and other objects is captured, and then at least one object parameter is determined for each of the detected objects. Based on this item parameter For example, each of the objects can then be identified and the object can be selected in a subsequent selection step.
  • the item parameter determined for each of the items can then be or include ID information for the item. This then makes the selection of the item particularly easy.
  • the fact that other objects are captured when capturing the image of the object can, for example, be designed and set up in such a way that the other objects are located on the captured image of the object, for example because they are in the immediate vicinity of the object.
  • the acquisition of the image of the object can also be embodied as a video recording of the object, from which, for example, a still image of the object is then extracted or can be extracted in a further step.
  • the other objects can then also be recorded, for example.
  • the video can be generated, for example, in such a way that the storage area moves relative to the camera, for example, is designed as a transport or conveyor belt.
  • the camera can also move in a linear or rotary motion and in this way capture the object and the other objects.
  • determining the respective at least one further object parameter can include ID information for each of the further objects. Furthermore, the determination of the respective at least one further object parameter for each of the further objects can be descriptive data for a bounding box each of the other items. Furthermore, the determination of the respective at least one further object parameter for each of the further objects can also include position data for each of these objects and/or a stable position assumed by the respective object.
  • the item can be selected by a user, for example. This can be done, for example, by displaying the captured image of the object and the other objects on a display device and the user then selecting the object to be gripped there. Furthermore, the object can also be selected automatically. For example, a specific object to be gripped can be specified, for example, by its ID information, a name or a shape. The object to be gripped can then be automatically selected by the system on the basis of the determined object parameters for the detected objects.
  • At least one gripping point of the object is determined using a method according to the present description and then the object is subsequently gripped by a gripping device, wherein the gripping device for gripping the object acts on the at least one gripping point.
  • the gripping device can grip the object, for example, in such a way that, for example, a pincer-like gripper creates a frictional connection to the object when gripping one or more of the gripping points, so that the object can be moved and/or lifted by the gripping device .
  • a frictional connection can, for example also be created by means of one or more suction devices which engage the one or more gripping points.
  • Such a frictional connection can also be produced by magnetic forces, for example, which then enables the object to be transported with the aid of the gripping device.
  • a system for gripping an object comprising an optical detection device for capturing an image of the object, a data processing device for determining the at least one object parameter of the object and/or for determining control data for a gripping device for gripping the object , wherein the system for carrying out a method is designed and set up according to the present description.
  • the optical detection device, the image of the object, the at least one object parameter of the object, the control data and the gripping device can be designed and set up, for example, in accordance with the present description.
  • the data processing device can be, for example, a computer, a PC, a controller, a control device, a programmable logic controller, a modular programmable logic controller, an edge device or a comparable device.
  • the data processing devices and their elements and/or components can also be designed and set up according to the present description.
  • the data processing device can include, for example, an ML model according to the present description.
  • the data processing device be designed and set up as a programmable logic controller, it being possible for the ML model to be provided, for example, in a central module of the programmable logic controller.
  • the ML model can also be provided in a functional module, which is connected to an aforementioned central module of the programmable logic controller via a backplane bus of the programmable logic controller.
  • the data processing device can, for example, comprise a corresponding execution environment which is designed and set up, for example, to run software, during which a method according to the present description is carried out.
  • the data processing device can also include multiple components or modules (e.g. including one or more controllers, edge devices, PLC modules, computers and/or comparable devices). Such components or modules can then be connected, for example, via a corresponding communication connection, e.g. an Ethernet, an industrial Ethernet, a fieldbus, a backplane bus and/or comparable devices. In a further refinement, this communication link can also be designed and set up for real-time communication, for example.
  • a corresponding communication connection e.g. an Ethernet, an industrial Ethernet, a fieldbus, a backplane bus and/or comparable devices.
  • this communication link can also be designed and set up for real-time communication, for example.
  • the system according to the present description comprises a gripping device and the system is also designed and set up to carry out a method according to the present description.
  • the gripping device can be designed and set up, for example, according to the present description.
  • the data processing device is designed and set up as a modular programmable logic controller with a central module and a further module, and that the at least one item parameter of the item is further determined using the further module.
  • the data processing device comprises a modular programmable logic controller with a central module and a further module, and that the at least one item parameter of the item is further determined using the further module.
  • a programmable logic controller is a control device that is programmed and used to regulate or control a system or machine. Specific functions, such as sequence control, can be implemented in such a PLC, so that both the input and output signals of processes or machines can be controlled in this way.
  • the programmable logic controller is defined, for example, in the EN 61131 standard.
  • both actuators of the system or machine which are generally connected to the outputs of the programmable logic controller, and sensors of the system or machine are used.
  • the sensors are located at the PLC inputs, with through the programmable logic controller receives information about what is happening in the system or machine.
  • sensors are: light barriers, limit switches, buttons, incremental encoders, level sensors, temperature sensors.
  • actuators are: contactors for switching on electric motors, electric valves for compressed air or hydraulics, drive control modules, motors, drives.
  • a PLC can be implemented in different ways. This means that it can be implemented as a single electronic device, as a software emulation, as a so-called soft PLC (also called “virtual PLC” or also called PLC application or PLC app), as a PC plug-in card, etc. Modular solutions are also often found, in which the PLC is assembled from several plug-in modules.
  • a modular programmable logic controller can be designed and set up in such a way that several modules can be or are provided, with usually, in addition to a so-called central module, which is used to run a control program, e.g. for controlling a component, machine or system (or part of it ) is designed and set up, one or more expansion modules can be provided.
  • expansion modules can be designed and set up, for example, as a current/voltage supply or also for the input and/or output of signals or data.
  • an extension module can also be used as a functional module for taking over special tasks (e.g. a counter, a converter, data processing with artificial intelligence methods (e.g. includes a neural network or another ML model)).
  • a function module can also be used as an AI module for executing actions using artificial intelligence methods be trained and set up.
  • a function module can include, for example, a neural network or an ML model according to the present description or a further ML model according to the present description.
  • the additional module can then be provided, for example, to carry out special tasks as part of the implementation of the method, e.g. computationally expensive subtasks or computationally expensive special tasks (such as a transformation, the use of AI methods, or the like).
  • special tasks e.g. computationally expensive subtasks or computationally expensive special tasks (such as a transformation, the use of AI methods, or the like).
  • the additional module can be specially designed and set up, for example, and/or also include an additional program runtime environment for corresponding software.
  • the additional module can include the ML model or the additional ML model, for example.
  • the system for gripping the object is further simplified since the data processing device can be specially adapted to an intended gripping task.
  • this is possible without having to change a central process sequence, which can run, for example, in a central module of the programmable logic controller. Special sub-tasks can then run in the further module, which can then be designed and set up differently depending on the exact gripping task.
  • system described above can also be designed and set up in such a way that the determination of the at least one object parameter for the object is done using an ML model and the further module comprises the ML model.
  • control data for the gripping device is determined using a further ML model and the further module comprises the further ML model (162).
  • the ML model can be designed and set up, for example, as an ML model in accordance with the present description.
  • the further ML model can be designed and set up, for example, as a further ML model in accordance with the present description.
  • the at least one object parameter, the object, the control data, the gripping device and the determination of the control data for the gripping device can be designed and set up according to the present description.
  • the ML model can be designed and set up as a so-called “recognition ML model”, for example.
  • a recognition ML model can be designed and set up, for example, to recognize a location of the object and/or a virtual frame around the object, a type or ID information relating to the object and/or a stable position of the object.
  • an ML model according to the present description can include such a recognition ML model.
  • Such a recognition ML model can be designed and set up, for example, as what is known as a “deep neural network”.
  • the captured image of the object for example, can be provided or used as input data for such a recognition ML model.
  • Output data of such a recognition ML model can then be, for example, one, several or all of the parameters mentioned above.
  • the recognition ML model can also be designed and set up to recognize a location, a virtual frame, a type and/or ID information for a plurality or all of the objects depicted on a captured image.
  • a recognition ML model designed in this way can advantageously be used, for example, when there are other objects on the captured image of the object.
  • Output data from a recognition ML model configured in this way can then be, for example, the above-mentioned information relating to the object for each of the detected objects: data relating to a location and/or virtual frame and/or ID information.
  • This information can then be used in a further method step, for example, to select the object to be gripped from all the objects detected - for example using the ID information determined.
  • the object parameters then already determined by this recognition ML model can then be used within the scope of a method according to the present description in order to determine the control data for the gripping device for gripping the object.
  • the ML model can be designed and set up, for example, as a so-called "angle detection ML model” which is designed and set up at least, among other things, to detect a rotation angle of the object on a surface with respect to a defined or definable starting position.
  • An ML model according to the present description can also be such an angle detection ML model include.
  • Such an angle recognition ML model can be designed and set up, for example, as a so-called regression AI model or also a so-called classification AI model.
  • output data can in turn be, for example, a corresponding angle of rotation of the object on the storage surface in relation to a defined or definable starting position—or comprise such an angle of rotation.
  • output data of an angle detection ML model may also include the rotation angle mentioned above, plus the data exemplified above from output data of a detection ML model.
  • the ML model can be designed and set up, for example, as a so-called “transformation ML model", which is used to determine transformation data from a fixed or definable starting position of the object to the position of the detected object on the storage area in the real world is trained and set up.
  • Input data for such a transformation ML model can be, for example, identification data for the object, a stable position of the object and/or a rotation angle of the object on the storage area in relation to a fixed or definable starting position.
  • Identification data for the object can be, for example, ID information, description data for a virtual frame around the object, information regarding a stable position and/or scaling data.
  • input data for such a transformation ML model can also be captured image data of an object lying on a flat surface.
  • the input data mentioned above, such as the identification data for the object, a stable position of the object and/or a rotation angle of the object can then be obtained from this image data in a first step, with the procedure then proceeding according to the above explanation.
  • the recorded image data mentioned of the object lying on the flat surface can also be used directly as input data for a corresponding transformation ML model.
  • Output data of such a transformation ML model can then be, for example, transformation data for the aforementioned transformation of the object from the defined or definable starting position to the real position of the object on the shelf.
  • a fixed or definable starting position of the object can be, for example, the position of a 3D model of the object in a corresponding 3D modeling program (e.g. 3D CAD software). This also applies, for example, to the starting position used in relation to the rotation angle.
  • Such a transformation ML model can be designed and set up, for example, as a so-called “deep neural network” or also a so-called “random forest” model.
  • An ML model according to the present description can comprise, for example, a recognition ML model and/or an angle recognition ML model and/or a transformation ML model.
  • another ML model according to the present description can, for example, be a recognition ML model and/or an angle detection ML model and/or a transformation ML model.
  • an ML model according to the present description can also include, for example, a recognition ML model and/or an angle recognition ML model, or such an ML model can be designed and set up.
  • a further ML model according to the present description can, for example, comprise a transformation ML model or be designed and set up as such a transformation ML model.
  • a system according to the present description can be designed and set up in such a way that the data processing device comprises an edge device or is designed and set up as an edge device (190), and that the at least one item parameter of the item is further determined using the edge device.
  • an edge device often has higher computing power compared to a more conventional industrial control device, such as a controller or a PLC, such an embodiment further simplifies and/or speeds up a method according to the present description.
  • a method according to the present description is executed completely on such an edge device.
  • computationally intensive and/or complex method steps are carried out on the edge device are carried out while other method steps are carried out on a further component of the data processing device, for example a controller or a programmable logic controller.
  • Such computationally intensive and/or complex method steps can be, for example, method steps using machine learning techniques or artificial intelligence, such as the application of one or more ML models according to the present description.
  • An edge device can, for example, include an application for controlling devices or systems.
  • an application can be designed and set up as an application with the functionality of a programmable logic controller.
  • the edge device can be connected, for example, to a further control device of a device or system or also directly to a device or system to be controlled.
  • the edge device can be designed and set up in such a way that it is additionally connected to a data network or a cloud or is designed and set up for connection to a corresponding data network or a corresponding cloud.
  • Such a system with an edge device can also be designed and set up in such a way that the at least one object parameter of the object is determined using an ML model and the edge device includes the ML model.
  • such a system with an edge device can also be designed and set up in such a way that the determination of the control data for the gripping device includes using an additional ML model and the edge device includes the additional ML model.
  • the ML model can be designed and set up, for example, as an ML model in accordance with the present description.
  • the further ML model can also be designed and set up, for example, as a further ML model in accordance with the present description.
  • the ML model can be designed and set up according to the present description, for example. Furthermore, the described method for generating training data for an ML model can be designed and set up according to the present description.
  • the method mentioned is carried out several times—for example each time with different start data for the object. In this way, a larger number of images with an associated identifier can be generated for training the ML model.
  • the method can be repeated so often, for example, that several—advantageously also all—of the possible stable positions of the object on the flat surface are shown on at least one of the images.
  • the method can be repeated, for example, so often that as many as possible—advantageously also all—of the possible stable positions of the object of the flat surface are represented on at least two of the images or at least ten of the images.
  • the ML model, the object, the acquisition of the image and the ID information for the stable position assumed by the object can be designed and set up according to the present description.
  • the starting data can be given, for example, by a height of the object, for example a center of gravity of the object, above the flat surface, an orientation of the object in space, and a vector for an initial speed of the object.
  • the falling movement can be, for example, a movement under the influence of the gravitational force. Furthermore, additional forces such as frictional forces (e.g. in air or in a liquid) and electromagnetic forces can still influence the movement. In an advantageous embodiment, the movement is dominated by the gravitational force, for example. The case movement begins according to the start data.
  • the ML model can, for example, be designed and set up as a recognition ML model according to the present description or can include such a model.
  • the identifier assigned to the captured image can include, for example, further object parameters in accordance with the present description.
  • Such further object parameters can be, for example, information regarding the location and/or position of the object, information regarding the position and/or shape of a virtual frame around the object, a type of item and/or ID information related to the item.
  • the ML model can, for example, also be embodied and set up as an angle recognition ML model according to the present description or can comprise such.
  • the identifier assigned to the captured image can include, for example, further object parameters in accordance with the present description.
  • additional object parameters can include, for example, a rotation angle of the object on the flat surface with respect to a fixed or definable starting position.
  • the ML model can also be embodied and set up as a transformation ML model according to the present description or can include such a model.
  • the identifier assigned to the captured image can include, for example, further object parameters in accordance with the present description.
  • additional object parameters can include, for example, transformation data for the above-mentioned transformation of the object from the defined or definable starting position to the real position of the object on the flat surface.
  • such a fixed or definable starting position of the object can be, for example, the position of a 3D model of the object in a corresponding 3D modeling program (e.g. 3D CAD software).
  • identification parameters and/or object parameters mentioned above can be determined at least in part manually by a user, for example - for example manually by means of a measurement or with the aid of at least partially automated measuring system. Furthermore, such identification parameters can be determined at least partially automatically, for example by image evaluation methods or additional automatic measuring systems, such as an optical measuring system, a laser measuring system and/or an acoustic measuring system.
  • the real position of the object is described, for example, by the identifier data for the object, the position of the object, the information relating to a virtual frame around the object, the identifier for a stable position of the object and/or a rotation angle of the object.
  • identification data for the object can be or include, for example, ID information, description data for a virtual frame around the object, ID information regarding a stable position and/or scaling data.
  • the transformation data, the defined or definable starting position, the rotation angle of the object, the identifier for a stable position of the object and the at least one object parameter can be designed and set up according to the present description. Furthermore, the location or position of the object and/or the information relating to a virtual frame around the object can also be designed and set up according to the present description.
  • the training data can be stored in a storage device and/or, for example, in a database or data collection for corresponding training data.
  • the ML model can be designed and set up according to the present description, for example. Furthermore, the described method for generating training data for an ML model can be designed and set up according to the present description.
  • the method mentioned can be carried out multiple times - e.g. each time with different start data for the object - in order to generate, for example, a plurality of images with an assigned identifier for training the ML model.
  • the method can be repeated, for example, so often that several—advantageously also all—of the possible stable positions of the digital model of the object on the virtual flat surface are displayed on at least one of the images.
  • the method can, for example, so frequently be repeated that as many as possible - advantageously also all - of the possible stable positions of the digital model of the object on the virtual flat surface are shown on at least two of the images or at least ten of the images.
  • the ML model, the object, the acquisition of the image and the ID information for the stable position assumed by the object can be designed and set up according to the present description.
  • the start data can be given, for example, by a height of the object (e.g. a height of a center of gravity of the object) above the flat surface, an orientation of the object in space, and a vector for an initial velocity of the object.
  • a height of the object e.g. a height of a center of gravity of the object
  • an orientation of the object in space e.g. a vector for an initial velocity of the object.
  • the falling movement can be simulated, for example, as a movement under the influence of the gravitational force. Furthermore, additional forces such as frictional forces (e.g. in air or in a liquid) and electromagnetic forces can be taken into account in the simulation. In an advantageous embodiment, the movement is simulated, for example, only taking into account the gravitational force.
  • the simulation of the fall movement then begins according to the start data.
  • the ML model can, for example, be embodied and set up as a recognition ML model according to the present description or can comprise such.
  • the identifier for the created image in addition to the ID information for the stable position assumed by the 3D model of the object, can include, for example, further object parameters in accordance with the present description.
  • Such further Item parameters can be information about a location and/or position of the 3D model of the item, information about a location and/or shape of a virtual frame around the 3D model of the item, a type of item and/or an ID -Include information related to the item.
  • the ML model can, for example, also be embodied and set up as an angle recognition ML model according to the present description or can comprise such.
  • the identifier assigned to the captured image can include, for example, further object parameters in accordance with the present description.
  • additional object parameters can include, for example, a rotation angle of the 3D model of the object on the virtual flat surface with respect to a fixed or definable starting position.
  • the ML model can, for example, also be embodied and set up as a transformation ML model according to the present description or can include such a model.
  • the identifier assigned to the captured image can include, for example, further object parameters in accordance with the present description.
  • additional object parameters can include, for example, transformation data for the aforementioned transformation of the 3D model of the object from a fixed or definable starting position to a real position of the object on the storage area.
  • a fixed or definable starting position of the 3D model of the object can be, for example, the position of the 3D model of the object in an appropriate 3D modeling program (e.g. 3D CAD software).
  • the identification parameters and/or object parameters mentioned above can be determined automatically, for example. Since all size data, position data and other data describing a location and/or position of the object are known in the digital simulation environment (otherwise a simulation of the object, in particular a physical simulation, would not be possible), a position of the object, a location of the object , a rotation angle of the object with respect to the virtual planar surface, transformation data according to the present description and other comparable object parameters with respect to the 3D model of the object can be taken directly from the simulation system. It is therefore possible for an above-described method for generating training data using a 3D model of the object to run automatically, and training data for an ML model can thus be generated or are generated automatically for an ML model in accordance with the present description.
  • the respective identification parameters mentioned above can also be determined at least partially manually by a user—for example manually by means of a measurement or with the aid of an at least partially automated measurement system.
  • identification parameters can be determined at least partially automatically, for example by image evaluation methods or also additional automatic digital measuring systems in a simulation environment for carrying out the method described here.
  • the training data can be stored in a storage device and/or, for example, in a database or data collection for corresponding training data.
  • the determined position of the object is, for example, by the identifier data for the 3D model of the object, a location or position of the 3D model of the object, information regarding a virtual frame around the 3D model of the object, the identifier for a stable position of the 3D model of the object and/or a rotation angle of the 3D model of the object.
  • identification data for the 3D model of the object can be or include, for example, ID information, description data for a virtual frame around the 3D model of the object, ID information for a stable position and/or scaling data.
  • the transformation data, the defined or definable starting position, the rotation angle of the 3D model of the object, the ID information or identifier for a stable position of the 3D model of the object and the at least one object parameter can be designed and set up according to the present description. Furthermore, the attitude or position of the 3D model of the object and/or the information regarding a virtual frame around the 3D model of the object can also be designed and set up according to the present description.
  • the training data can be stored in a storage device and/or, for example, in a database or data collection for corresponding training data.
  • the ML model can be designed and set up according to the present description, for example. Furthermore, the described method for generating training data for an ML model can be designed and set up according to the present description.
  • the method mentioned can also be carried out several times here, for example in order to generate as large a number of images as possible with an associated identifier for training the ML model.
  • the method can be repeated, for example, so often that several—advantageously also all—of the possible stable positions of the digital model of the object on the virtual flat surface are displayed on at least one of the images.
  • the method can be repeated, for example, so often that as many as possible—advantageously also all—of the possible stable positions of the digital model of the object on the virtual flat surface are displayed in at least two of the images or at least ten of the images.
  • the above-mentioned methods for generating training data can also be developed in such a way that the respective methods are designed and set up for training an ML model according to the present description, or for training a further ML model according to the present description, that the ML model or the further ML model is trained for the stable position occupied by the object or the 3D model of the object using the recorded or determined image and at least the ID information associated therewith.
  • the ML model and/or the further ML model can be designed and set up, for example, as a recognition ML model and/or an angle recognition ML model and/or a transformation ML model or such ML models include.
  • the ML model and/or the further ML model can therefore include the function of one, two or also all three of the ML models mentioned.
  • the ML model can be embodied and set up as a recognition ML model and/or an angle recognition ML model, for example, while the further ML model can be embodied and set up as a transformation ML model, for example .
  • the method can be used, for example, to train a recognition ML model according to the present description, an angle recognition ML model according to the present description and/or a transformation ML model according to the present description.
  • the training of the ML model and / or the further ML model can continue, for example, using the captured image of the object, a position of the object, ID information of the object, a rotation angle of the object and / or an identifier with respect to a dated Subject occupied stable position done.
  • To train the ML model and/or the additional ML model for example, the position of the object, the ID information of the object, the rotation angle of the object and/or the identifier regarding the stable position occupied by the object are assigned to the captured image of the assigned to the subject.
  • the captured image can be labeled with a position of the object, ID information for the object and/or an identifier with regard to a stable position occupied by the object.
  • the captured image of the object can be labeled with a position of the object, ID information for the object, a rotation angle of the object and/or an identifier for a stable position occupied by the object will.
  • the captured image can be labeled, for example, with corresponding transformation data for transforming an initial position of the object into the position assumed on the captured image.
  • At least one object parameter determined using the captured or created image can be labeled according to the present description, for example with corresponding transformation data for transforming an initial position of the object into the position assumed on the captured or created image will.
  • the object mentioned above is also achieved by using training data generated by means of a method for generating training data according to the present description for training an ML model, in particular an ML model according to the present description.
  • an ML model in particular ML model according to the present description, wherein the ML model was trained using training data that was generated using a method for generating training data according to the present description.
  • a method or a system for determining control data for a gripping device according to the present description can be designed and set up in such a way that an ML model used in the implementation of the method in the system mentioned was trained using training data, which using a Method for generating training data according to the present description.
  • This exemplary embodiment is based on the problem that in many manufacturing processes, parts are made available via so-called "chutes" as a transport system. Such parts can come from external suppliers, for example, or from an upstream internal production process. For the further manufacturing process it is necessary, for example, that these parts are isolated and individually manipulated or transported in a specific way. Precise information regarding the position and orientation of the isolated parts is necessary, especially for manufacturing processes in which this further treatment is carried out by robotic arms. In such chutes, however, the location and position of the parts is completely random and cannot be fixed in a predetermined way. Therefore, this data must be determined dynamically in order to be able to successfully grip and transport these parts with a robot arm, for example.
  • An exemplary method and system for gripping an object can, for example, be designed and set up within the scope of this exemplary embodiment such that the system can localize objects or parts for which a 3D model of the object or part is available.
  • a 3D model can, for example, have been created by 3D CAD software.
  • Such a method can be executed, for example, on different hardware devices, for example a programmable logic controller, a modular programmable logic controller, an EDGE device or using computing capacity in a cloud, for example to do the appropriate image processing.
  • a programmable logic controller can be designed, for example, in such a way that method steps are carried out using artificial intelligence or machine learning techniques in a special functional module for the programmable logic controller for carrying out artificial intelligence methods.
  • modules can include a neural network, for example.
  • the exemplary system described below can recognize the 6D orientation of any object using, for example, a corresponding 3D model of the object so that the object can be reliably grasped at a gripping point specified in the 3D model. This allows the system to feed corresponding feed parts to a specific production step, for example, with a high level of repeatable accuracy.
  • Such an exemplary system can thus include, for example: the PLC, the camera controller, the camera, software that runs on the respective components, and additional software that generates input values for the software mentioned.
  • a 3D model of the part to be gripped is provided as an input, with possible gripping points for the part being specified or marked in the 3D model.
  • possible stable positions of the part on a flat surface are then determined in a first step.
  • a possible stable position is a position in which the object is in equilibrium and does not tip over. In the case of a coin, for example, this is also a position in which the coin stands on its edge.
  • These possible stable positions can be determined, for example, by dropping the objects with a wide variety of initial conditions onto a flat surface. This can be done in reality or in a corresponding physical simulation using a 3D model of the part. Then, both in the simulation and in reality, you wait until the part stops moving. The position then reached is regarded as a stable position and recorded as such.
  • Another option for determining possible stable positions is to determine those positions in which the selected part touches a flat surface in (at least) three points, with the object then not penetrating the surface in any other point. The one determined in one of the ways described stable positions are then each assigned a unique identifier.
  • training data for the segmentation ML model (M-Seg) is generated.
  • the training data consists of a set of images with detected objects, annotated or labeled with the respective location of the object, ID information or a type of object, as well as a correspondingly assumed stable position.
  • This data can be generated, for example, by positioning various objects in corresponding stable positions in the real world.
  • 3D models of the objects can also be arranged virtually in a raytracer software or game engine in stable positions, with corresponding images of these objects then being generated artificially.
  • the label for each of the objects consists of a rectangular virtual bounding box (xl, y1, x2, y2), the object type and an identifier for the stable position occupied.
  • the rotation angle of the selected part with respect to a surface normal of the storage area is still assigned as a label.
  • this data can be automatically generated for labeling the captured images.
  • All of the data generated in this way can then be used, for example, to train a deep neural network, with standard architectures such as YOLO for the M-Seg model and a convolutional neural network can be used for the regression and the regression model, respectively.
  • standard architectures such as YOLO for the M-Seg model
  • a convolutional neural network can be used for the regression and the regression model, respectively.
  • reference images of the respective parts are then generated in the respective stable positions.
  • This can in turn be done using real objects or also be generated by the virtual simulation mentioned.
  • the disadvantage of using real objects is that labeling has to be done manually.
  • the data required for labeling can be generated automatically and the labeling can therefore also run automatically.
  • the transformation data can also be determined more precisely if the images are generated on the basis of a physical simulation with 3D models.
  • the generated transformation data then make it possible to transform gripping points marked in the 3D model of a specific part into the coordinates of corresponding gripping points of a real detected part using the methods described above, so that a gripping device can grip the real part at the corresponding gripping points using these coordinates.
  • FIG 1 shows a gripping system 100 as an embodiment for a system according to the present description.
  • This exemplary gripping system 100 is designed and set up for detecting, selecting and gripping and transporting objects 200 on a transport device 110 .
  • FIG. 1 a cuboid gripping object 200 is shown, which is transported within a transport device 110 to a flat storage area 112 and deposited there. Furthermore, a camera 130 is provided for capturing the storage area 112 with the object 200 to be gripped, which is connected to an industrial PC 140 .
  • the industrial PC 140 includes image evaluation software that includes a neural network 142 . Using this image evaluation software with neural network 142, the images captured by camera 130 are evaluated in such a way that grasping object 200 and other possible objects are recognized and data for grasping points for grasping this object are determined using a method described in more detail below.
  • figure 2 shows a 3D model 250 of the cuboid gripping object 200.
  • the 3D model 250 of the cuboid gripping object 200 is in figure 2 both in perspective view (far left in figure 2 ) as well as in its three stable positions according to the present description.
  • figure 2 are still three of the six sides of the 3D model 250 of the cuboid 200 marked with corresponding numbers.
  • the stable positions 1, 2 and 3 of the 3D model 250 are each shown in a view from above.
  • gripping points 255 provided for gripping the corresponding object are shown as black squares in the 3D model 250 .
  • the gripping points 255 are points at which a corresponding cuboid 200 can advantageously be gripped by a gripping device 120, 122.
  • the 3D model 250 was thereby from a corresponding 3D CAD program created. Within this program, the appropriate model grips 255 in the 3D model 250 have been identified.
  • figure 3 shows three exemplary methods 410, 420, 430 for determining one or more stable positions of an object or item.
  • a specific object type is selected in a first step 412, for which one or more stable positions are to be determined.
  • this object is dropped onto a flat surface with random initial conditions.
  • the random initial conditions include a randomly determined height above the flat surface and an in Direction and speed any initial speed for the selected object.
  • a step 416 it is awaited until the dropped object stops moving. Once this has come to rest, an image of the object on the flat surface is captured, for example with a camera.
  • a next step 418 the stable position occupied by the object on the level surface is then identified and an identifier unique to the occupied stable position is determined. This identifier, which is unique to the assumed stable position, is then assigned to the captured image.
  • a combination of a captured image determined in this way with a unique identifier for the stable position occupied by the object in the image can then be used, for example, for later comparable measurements in order to assign correspondingly unique identifiers to stable positions.
  • a database relating to the stable positions of objects can be set up with the aid of such image-identification combinations.
  • image identifier combination can be used to train an ML model according to the present description.
  • method step 418 is then carried out in turn in method step 414, for example with the same object and different initial conditions.
  • a new image-identification combination is then generated in turn, which can then be handled as already described above. This is in figure 3 marked with an arrow between method steps 418 and 414.
  • the method can be carried out until sufficient image-identifier combinations are available, for example for a database or also for training a corresponding ML model.
  • FIG. 4 also shows a first automatic method 420 for determining stable positions of one or more objects.
  • a first step 422 an object is selected for which correspondingly stable positions are to be determined.
  • a corresponding 3D model is then selected for this object.
  • Such 3D models can be created or have been created, for example, with corresponding 3D CAD programs.
  • a next step 424 the fall of such an object onto a flat surface is then simulated using the 3D model of the object onto a virtual surface using a simulation environment with a physical simulation (for example with a so-called "game engine”).
  • the initial conditions can be chosen randomly, for example with regard to speed and direction.
  • a step 426 the simulation is then continued until the simulated object no longer moves within the scope of normal measurement accuracy.
  • An image of the 3D model of the object that has come to rest on the virtual flat surface is then generated with the aid of the simulation environment. The image is generated in such a way that it corresponds to a camera recording of a real object corresponding to the 3D model on a real flat surface corresponding to the virtual surface.
  • a unique identifier for the stable position assumed by the 3D model on the image is assigned to this created or generated image.
  • this image-identifier combination can then be used to set up a corresponding database or to train a corresponding ML model.
  • the named method can then be carried out multiple times, in that method step 428 is then followed by method step 424 again.
  • this subsequent step 424 the case of a 3D model is then simulated, for example with different initial conditions. this is in figure 3 represented by a corresponding connecting arrow between method step 428 and method step 424.
  • FIG. 3 Furthermore shows figure 3 a second automatic method 430 for determining stable object positions.
  • an object is selected in a first step 432, for which stable object positions are subsequently determined.
  • this second automatic method works using a 3D model of the selected object type.
  • those positions of the selected 3D model on a virtual surface are determined in which the 3D model touches the virtual flat surface at three or more points without the 3D -model penetrates this flat surface in further points.
  • a next step 436 comparable to the first automatic method 420, one or more images of the 3D model on the virtual flat surface are then generated from each of these determined positions of the 3D model on the virtual flat surface.
  • different virtual camera positions can be used for each of the images.
  • a unique identifier for the stable position assumed by the object on the respective image is then assigned to the corresponding created images.
  • identifier-image combinations can then in turn be used to set up a corresponding database for stable positions of objects and/or to train one or more corresponding ML models.
  • FIG. 5 shows two examples of methods 510, 520 for generating training data for a recognition ML model and/or an angle recognition ML model.
  • first method 510 shown on the left is provided for the manual or partially automated determination of training data for a recognition ML model and/or an angle recognition ML model.
  • a specific object type is selected for which corresponding training data is to be generated.
  • stable object positions for this object type are determined on a flat surface.
  • methods according to the present description can be used.
  • a plurality of images are generated using the selected object in different positions, different stable attitudes and different angles of rotation about a surface normal of the flat surface or selected from a database or image collection, for example.
  • identification data for the object is assigned to the respective images, for example data relating to a virtual frame around the object, an object type, an identification for the assumed stable position and/or a location. Furthermore, when training data is generated for an angle detection ML model, the identifier data also includes a rotation angle.
  • the automatic, simulation-based method 520 illustrated begins in turn in a first method step 522 with a selection of an object type and a correspondingly associated 3D model for this object type.
  • the 3D model can be designed and set up according to the present description, for example.
  • the stable object positions are again automatically determined using the 3D model of the selected object type.
  • This automatic determination can take place, for example, according to the present description.
  • a set of images is automatically generated using the 3D model of the selected object type in different positions, stable attitudes and rotation angles. These images can, for example, in turn be generated according to the present description, for example using a corresponding ray tracer engine.
  • the generated images are then automatically annotated or labeled with appropriate characteristic data.
  • characteristic data is, for example, information regarding a virtual frame around the displayed object, an object type, an identifier regarding a stable position of the object and/or a position of the object. If the training data is intended for training an angle detection ML model, then the characteristic data includes also a rotation angle. The characteristic data mentioned can be automatically annotated or labeled, since these data are already known when the image is generated due to the virtual generation of the images using a simulation environment and a corresponding raytracer engine.
  • the method steps are then carried out, beginning with method step 522 for a further object type. This loop is carried out until training data has been generated for all those objects which are required for an application of the corresponding ML model.
  • two methods 610, 620 are shown, which are examples of methods for generating training data for a transformation ML model.
  • a specific object type is again selected, for which corresponding training data is then generated.
  • an image is then generated using the selected object type, e.g., after the selected object has been dropped onto a flat surface with arbitrary initial conditions (e.g. regarding altitude and starting velocity vector).
  • object position data are then determined from the generated image.
  • object location data can, for example, a position of the object, an identifier for the object, information regarding a virtual Bounding box around the object, a rotation angle and / or an assumed stable position or include.
  • transformation data for the transformation of a 3D model of the selected object into the position of the model on the generated image are determined. This can be achieved, for example, by overlaying the captured image with a representation of the 3D model on a computer screen and transforming or rotating and shifting and escalating the 3D model image of the object by manual transformation actions of a user such that it matches the object shown on the generated image. From the transformation operations used, the desired transformation data can then be determined in a manner known to those skilled in the art.
  • this transformation data is then assigned, for example, to the generated image or also to the determined object position data.
  • These annotated or labeled images or annotated or labeled location data can then be used to train a corresponding transformation ML model.
  • the manual method 610 starts again in the first method step 612 for selecting a new object type, after which corresponding training data are determined for this further object type. This loop is indicated by a dashed arrow to the left of the in figure 5 illustrated manual method 610 symbolized.
  • the sequence of the manual method 610 presented above is carried out until sufficient training data has been determined for all relevant object types.
  • an exemplary automatic method 620 is shown, with which training data for a transformation ML model can be generated in an automated and simulation-based manner.
  • a specific object type and a corresponding 3D model for it are also determined here.
  • an image of the selected 3D model is automatically generated in any position, with any angle of rotation and in any stable position.
  • This can be done, for example, by means of a physical simulation in which the fall of a corresponding object with any starting conditions (e.g. in terms of height and velocity vector) is simulated onto a flat surface, and then an image of the object is generated using a corresponding raytracer engine after it has been has come to rest again in the simulation.
  • This generation of an image can be designed and set up, for example, in accordance with the present description.
  • the images can also be generated, for example, by displaying or rendering the 3D model of the object with different positions, rotation angles and stable positions on an image - e.g. by means of a corresponding 3D modeling or 3D CAD tool.
  • object position data are automatically taken from the generated image or directly from the corresponding simulation environment or the corresponding 3D modeling or 3D CAD tool.
  • object position data can in turn include, for example, a position, information regarding a virtual bounding box around the object, a rotation angle and/or an identifier for an assumed stable position of the object.
  • transformation data of the 3D model of the object are then automatically generated into the object located in the simulation environment or the object represented on the generated image. This can be done, for example, by importing the 3D model of the object into the simulation environment and subsequent automatically determined or indirect transformation operations in such a way that the imported 3D model of the object is transferred to the object on the flat surface in the assumed stable position.
  • This sequence of transformation operations can then already represent the corresponding transformation data.
  • this sequence of transformation operations can be converted into the transformation data in a manner known to those skilled in the art.
  • the generated image or the position data determined for it is annotated or labeled with these corresponding transformation data.
  • the so labeled images or location data can then be used as training data for a corresponding transformation ML model.
  • a first procedural loop back to procedural step 624 can also be used to generate further images for the selected object type and thus also further training data for this object type for a transformation ML model.
  • This first loop is in figure 5 denoted by an arrow at the left side of the process flow of automatic process 620.
  • a new object type is selected in a second superimposed process loop, beginning again with the first process step 622, and the process explained above is then carried out for this further object type.
  • This second superimposed procedural loop is in figure 5 on the right side of the representation of the automatic method 620 by a corresponding dashed arrow from the last method step 628 to the first method step 622.
  • the entire automated process 620 as outlined above is then executed until sufficient training data is available for all required object types to train a corresponding transformation ML model.
  • FIG. 12 shows an example of a method flow for grabbing an object from a surface using a recognition ML model or an angle recognition ML model and a transformation ML model according to the present description.
  • the camera 130 takes a camera picture of the cuboid 200 located on the storage area 112.
  • this camera image is transmitted to the industrial PC 140, on which a corresponding image evaluation software including a corresponding recognition ML model or also a corresponding angle recognition ML model is implemented.
  • a corresponding image evaluation software including a corresponding recognition ML model or also a corresponding angle recognition ML model is implemented.
  • the neural network 142 shown is an example of such a recognition ML model or such an angle recognition ML model.
  • a virtual boundary frame (a so-called "bounding box") around the imaged cuboid 200 is determined using the recognition ML model, as well as an object type for the detected cuboid 200 and its position and scaling on the recorded image as well as a thereby occupied stable position determined.
  • This determination of the parameters mentioned can be designed and set up, for example, as explained in more detail in the present description.
  • an angle recognition ML model can also be used, in which case a rotation angle around a surface normal of the storage area 112 is then determined as an additional parameter. This determination can also be designed and set up according to the present description, for example.
  • method step 711 is carried out for each of the objects displayed.
  • step 712 that virtual bounding box is selected in which the object that is to be gripped by the robot 120 is located.
  • the selected bounding box is the one around box 200.
  • transformation data for a transformation of a 3D model 250 for the cuboid 200 into the cuboid 200 located on the storage area 112 are generated using a transformation ML model trained accordingly for this application.
  • characteristic position data of the cuboid 200 such as its position, information on the virtual boundary frame around the cuboid, the stable position assumed, a rotation angle with respect to a surface normal of the storage area 112 or comparable position data are entered into the transformation ML model.
  • This supplies the corresponding transformation data for the transformation of the 3D model 250 of the cuboid 200 into the cuboid 200 located on the storage area 112.
  • the transformation data generated in method step 713 are then applied in a further method step 715 to the coordinates of the model gripping points 255, which were determined in method step 714, in order to then derive specific robot gripping coordinates for gripping the Cuboid 200 to be determined on the shelf 112.
  • the corresponding robot gripping coordinates are designed and set up in such a way that gripper 122 of robot 120 almost cuboid 200 at one or more gripping points, these gripping points corresponding to model gripping points 255 in 3D model 250 of cuboid 200.
  • method steps 711 to 715 are running in industrial PC 140, for example, the robot gripping coordinates generated in method step 715 are then transmitted from industrial PC 140 to PLC 150 in a next method step 716.
  • this data is subsequently converted by the PLC 150 into corresponding control data for the robot 120 and transmitted to the robot 120. This then grips the cuboid 200 at the calculated gripping points in order to subsequently transport it to a desired storage location.
  • figure 7 shows on the left two 3D models 250, 350, with next to the one already in figure 2 illustrated 3D model 250 of the cuboid 200, a 3D model 350 of a pyramid is also illustrated.
  • corresponding model gripping points 255 are identified in the 3D model 250 of the cuboid 200 , which designate suitable points on the corresponding cuboid 200 at which a gripper can advantageously attach to grip the cuboid 200 .
  • model grip points 355 for gripping a corresponding pyramid are identified.
  • a camera image 132 is shown as an example, on which cuboids 200, 210, 220 and pyramids located on a corresponding flat surface 300, 310 were detected.
  • the three cuboids 200, 210, 220 shown correspond to the 3D model 250 of the cuboid, which is on the left side of figure 7 is shown.
  • the two pyramids 300, 310 correspond to the 3D model 350 of the pyramid, which is on the left side of FIG figure 7 is shown.
  • a first cuboid 200 shown on the camera image 132 is in the second stable position for this cuboid 200, as is explained in the context of the explanations relating to figure 2 was presented.
  • the large long side which in figure 2 marked with the number 2, upwards.
  • a corresponding gripping point 205 is marked on the camera image 122 on the cuboid 200, which corresponds to the in figure 2 corresponds to the illustrated model grip point 255 in the second stable position.
  • transformation data can then be calculated, for example, with which the parameters for the corresponding model grip point 255 can be converted into coordinates for the grip point 205 of the cuboid 200 on the camera image 132 .
  • This transformation data can then be used, for example, to determine control data for a robot so that it can grasp the cuboid 200 at the gripping point 205 with a suction gripper, for example, and thus transport it.
  • the camera image 132 shows a virtual boundary frame 202 around the cuboid 200 shown.
  • a corresponding “region of interest” (ROI) for this cuboid 200 can be defined with this virtual boundary frame 202, for example.
  • ROI region of interest
  • from the data on this virtual bounding box 202 for the shown Cuboid 200 determine further parameters for the cuboid, such as a position, a scaling factor or also an estimate for a rotation angle and/or a stable position.
  • FIG 7 a second cuboid 210 is shown, which is in a first stable position, as shown in figure 2 was explained.
  • the short, long side of the cuboid 210, which is in figure 2 denoted by the number 1, upwards.
  • a corresponding cuboid gripping point 215 is again shown on the camera image 132, which corresponds to the model gripping point 255, which is shown in figure 2 in the figure corresponds to the stable position 1 view of the 3D model 250 of the parallelepiped.
  • a virtual boundary frame 212 around the second cuboid 210 is also shown here.
  • FIG. 7 Furthermore shows figure 7 on the camera image 132 a third cuboid 220, which in turn is in the second stable position according to figure 2 located.
  • a corresponding gripping point 225 is also shown here, which corresponds to that in figure 2 corresponds to the illustrated model grip point 255 in the second stable position.
  • a corresponding virtual boundary frame 222 is also assigned to the third cuboid 220 and is shown on the camera image 132 .
  • the camera image also shows a first pyramid 300 with a grip point 305 visible in the corresponding stable position, which grip point corresponds to one of the pyramid model grip points 355 .
  • a corresponding virtual boundary frame 302 is also drawn in around this first pyramid 300, which can be used, for example, to select the pyramid for a subsequent gripping at the gripping point 305.
  • the camera image 132 also shows a second pyramid 310 in a stable position for a pyramid 300, 310 of this type.
  • a corresponding virtual boundary frame 312 is also drawn into the camera image 132 for this second pyramid 310 .
  • Such a camera image 132 could be captured, for example, when there is something on the storage surface 112 of the transport device 110, as shown in figure 1 is shown, three cuboids 200, 210, 220 are located according to the 3D model 250 of these cuboids 200, 210, 220, and two pyramids 300, 310 according to the 3D model 350 of these pyramids 300, 310.
  • the virtual bounding boxes 202, 212, 222, 302, 312, for example, as well as the respective positions, stable locations and rotation angles of the objects 200, 210, 220, 300, 310 shown can then be determined by an image evaluation method according to the present description the first cuboid 200 can be selected in a corresponding selection step.
  • transformation data can then be determined from the determined position data and parameters of the first cuboid 200 on the camera image 132, with which the 3D model 250 of the cuboid 200 is transformed into the cuboid located on the camera image 132 in reality leaves.
  • This transformation data can then be applied to convert the cuboid model grip points 255 into coordinates of the cuboid 200 grip points 205 on the camera image 132 .
  • These coordinates for the gripping points 205 of the selected cuboid 200 can then be used to determine robot data for controlling a robot, which has a suction gripper, for example. With this, the cuboid 200 can then be grasped at the cuboid gripping points 205 and transported.
  • figure 8 shows a modification of the already in figure 1 shown gripping system 100.
  • gripping system 100 shown is the one shown in figure 1 illustrated industrial PC 140 for image evaluation replaced by an edge device 190, which is also designed and set up at least among other things for evaluating images of the camera 130.
  • the edge device can, for example, be designed and set up according to the present description and, for example, in addition to being coupled to the PLC 150, can also be equipped with a non-in figure 8 shown cloud to be connected. Provision can furthermore be made for the image evaluation method for evaluating the images captured by the camera 130, or at least parts thereof, to run on this cloud.
  • figure 9 represents a further modification of the in figure 1 and 8th gripping systems 100 shown, whereby in figure 9 gripping system 100 shown, the images captured by camera 130 are evaluated in PLC 150.
  • the PLC 150 includes a central control assembly 152, which has a runtime environment 154 for running a corresponding control program, including for controlling the transport device 110 and the robot 120.
  • the PLC 150 includes an input/output assembly 158 via which the PLC 150 communicates with the transport device 110 and the robot 120 .
  • the PLC 150 also includes a function module 160 for running an image evaluation method for evaluating images from the camera 130, wherein the function module 160 comprises a neural network 162, which is an embodiment example for a recognition ML model according to the present description, an angle recognition ML model according to the present description and/or a transformation ML model according to the present description represents.
  • the central module 152, the input/output module 158 and the function module 160 of the PLC 150 are coupled to one another via an internal backplane bus 156.
  • the communication between these modules 152, 158, 160 takes place, for example, via this backplane bus 156.
  • the PLC 150 can be designed and set up in such a way that, as part of a method according to the present description, all those work steps that use an ML model run in the function module 160 of the PLC 150, while all other work steps in the context of the named method are carried out a control program running in the runtime environment 154 of the central module 152 can be executed.
  • PLC 150 can also be designed and set up in such a way that, as part of a method according to the present description, all work steps associated with the evaluation of images, in particular images from camera 130, take place in function module 160, while the work steps for The transport device 110 and the robot 120 are controlled by a control program running in the runtime environment 154 of the central control assembly 152 .
  • the PLC 150 can be set up very effectively to run a method according to the present description because calculation-intensive special tasks, such as the handling of the mentioned ML models or the evaluation of images, are outsourced to the special function module 160, and all other method steps can run in the central module 152.

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Automation & Control Theory (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Physics & Mathematics (AREA)
  • Fuzzy Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Orthopedic Medicine & Surgery (AREA)
  • Image Analysis (AREA)
EP21163930.7A 2021-03-22 2021-03-22 Verfahren zum ermitteln von steuerungsdaten für eine greifeinrichtung zum greifen eines gegenstands Withdrawn EP4063081A1 (de)

Priority Applications (5)

Application Number Priority Date Filing Date Title
EP21163930.7A EP4063081A1 (de) 2021-03-22 2021-03-22 Verfahren zum ermitteln von steuerungsdaten für eine greifeinrichtung zum greifen eines gegenstands
PCT/EP2022/055162 WO2022199994A1 (de) 2021-03-22 2022-03-01 Verfahren zum ermitteln von steuerungsdaten für eine greifeinrichtung zum greifen eines gegenstands
CN202280036828.5A CN117355394A (zh) 2021-03-22 2022-03-01 用于得出用于抓取物体的抓取装置的控制数据的方法
EP22712899.8A EP4284604A1 (de) 2021-03-22 2022-03-01 Verfahren zum ermitteln von steuerungsdaten für eine greifeinrichtung zum greifen eines gegenstands
US18/283,226 US20240253232A1 (en) 2021-03-22 2022-03-01 Method for Ascertaining Control Data for a Gripping Device for Gripping an Object

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP21163930.7A EP4063081A1 (de) 2021-03-22 2021-03-22 Verfahren zum ermitteln von steuerungsdaten für eine greifeinrichtung zum greifen eines gegenstands

Publications (1)

Publication Number Publication Date
EP4063081A1 true EP4063081A1 (de) 2022-09-28

Family

ID=75143503

Family Applications (2)

Application Number Title Priority Date Filing Date
EP21163930.7A Withdrawn EP4063081A1 (de) 2021-03-22 2021-03-22 Verfahren zum ermitteln von steuerungsdaten für eine greifeinrichtung zum greifen eines gegenstands
EP22712899.8A Pending EP4284604A1 (de) 2021-03-22 2022-03-01 Verfahren zum ermitteln von steuerungsdaten für eine greifeinrichtung zum greifen eines gegenstands

Family Applications After (1)

Application Number Title Priority Date Filing Date
EP22712899.8A Pending EP4284604A1 (de) 2021-03-22 2022-03-01 Verfahren zum ermitteln von steuerungsdaten für eine greifeinrichtung zum greifen eines gegenstands

Country Status (4)

Country Link
US (1) US20240253232A1 (zh)
EP (2) EP4063081A1 (zh)
CN (1) CN117355394A (zh)
WO (1) WO2022199994A1 (zh)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10625432B2 (en) * 2015-11-13 2020-04-21 Berkshire Grey, Inc. Processing systems and methods for providing processing of a variety of objects

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10625432B2 (en) * 2015-11-13 2020-04-21 Berkshire Grey, Inc. Processing systems and methods for providing processing of a variety of objects
US20200164531A1 (en) 2015-11-13 2020-05-28 Berkshire Grey, Inc. Processing systems and methods for providing processing of a variety of objects

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
KLANK U ET AL: "Real-time CAD model matching for mobile manipulation and grasping", HUMANOID ROBOTS, 2009. HUMANOIDS 2009. 9TH IEEE-RAS INTERNATIONAL CONFERENCE ON, IEEE, PISCATAWAY, NJ, USA, 7 December 2009 (2009-12-07), pages 290 - 296, XP031609240, ISBN: 978-1-4244-4597-4 *

Also Published As

Publication number Publication date
WO2022199994A1 (de) 2022-09-29
US20240253232A1 (en) 2024-08-01
EP4284604A1 (de) 2023-12-06
CN117355394A (zh) 2024-01-05

Similar Documents

Publication Publication Date Title
DE102018215057B4 (de) Maschinelles-Lernen-Vorrichtung, Robotersystem und maschinelles-Lernen-Verfahren
DE102014108287B4 (de) Schnelles Erlernen durch Nachahmung von Kraftdrehmoment-Aufgaben durch Roboter
DE102014102943B4 (de) Robotersystem mit Funktionalität zur Ortsbestimmung einer 3D- Kiste
DE102019109624B4 (de) Roboterbewegungseinlernvorrichtung, Robotersystem und Robotersteuerung
DE102015111080B4 (de) Robotervorrichtung mit maschinellem Sehen
DE202017106506U1 (de) Einrichtung für tiefes Maschinenlernen zum Robotergreifen
DE112019000097B4 (de) Steuervorrichtung, Arbeitsroboter, Programm und Steuerverfahren
DE112017007398B4 (de) Steuervorrichtung, Greifsystem, Verteilersystem, Programm und Steuerverfahren
DE102020101767B4 (de) Steuerverfahren und steuerung für ein robotersystem
DE102021107333A1 (de) 3d-stellungsabschätzung mit einer 2d-kamera
DE102022107227A1 (de) Greifenlernen unter verwendung modularisierter neuronaler netze
DE102022107249A1 (de) Modularisierung von Netzen zum Lernen von hochdimensionalen Roboteraufgaben
WO2023078884A1 (de) Ansteuerung eines industrieroboters für eine greifaufgabe
DE102021212859A1 (de) Vorrichtung und Verfahren zum Trainieren eines maschinellen Lernmodells zum Erzeugen von Deskriptorbildern zu Bildern von Objekten
DE102021109036A1 (de) Vorrichtung und verfahren zum lokalisieren von stellen von objekten aus kamerabildern der objekte
EP4063081A1 (de) Verfahren zum ermitteln von steuerungsdaten für eine greifeinrichtung zum greifen eines gegenstands
EP4063080A1 (de) Verfahren zum ermitteln von steuerungsdaten für eine grei-feinrichtung zum greifen eines gegenstands
DE102021209646B4 (de) Robotervorrichtung, Verfahren zum computerimplementierten Trainieren eines Roboter-Steuerungsmodells und Verfahren zum Steuern einer Robotervorrichtung
EP4064106A1 (de) Verfahren zum erzeugen von trainingsdaten für ein ml-modell
EP4164841A1 (de) Sensoreinrichtung für ein greifsystem, verfahren zur erzeugung von optimalen greifposen zur ansteuerung einer greifvorrichtung und zugehöriges greifsystem
DE102018126310B3 (de) Verfahren zum Erstellen eines Objektmodells zum Greifen eines Objekts, computerlesbares Speichermedium und Robotersystem
EP3582140A1 (de) System zur automatischen erkennung von laborarbeitsgegenständen sowie verfahren zum betrieb eines systems zur automatischen erkennung von laborgegenständen
DE102022128961A1 (de) Verwendung eines synthetischen datensatzes zum trainieren der robotergestützten depalettierung
DE102023203021A1 (de) Verfahren zum Ermitteln eines Deskriptorbilds für ein Bild eines Objekts
EP4141805A1 (de) Verfahren zum erzeugen, automatisierten annotieren und bereitstellen von trainings-bilddaten

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20230329