DE102020214301A1 - DEVICE AND METHOD FOR CONTROLLING A ROBOT TO PICK AN OBJECT IN DIFFERENT POSITIONS - Google Patents

Abstract

According to various embodiments, a method for controlling a robot to pick up an object in different positions is described, comprising: defining a plurality of reference points on the object, mapping a first camera image of the object in a known position onto a first descriptor image, identifying the descriptors of the reference points the first descriptor image, mapping a second camera image of the object in an unknown position onto a second descriptor image, searching for the identified descriptors of the reference points in the second descriptor image, determining the positions of the reference points in three-dimensional space in the unknown position from the positions found and determining a recording pose of the object for the unknown position from the determined positions of the reference points.

Description

The present disclosure relates to devices and methods for controlling a robot to pick up an object in different positions.

In order to enable flexible production or processing of objects by a robot, it is desirable for the robot to be able to handle an object regardless of the posture with which the object is placed in the working space of the robot. Therefore, the robot should be able to recognize in which position the object is, at least insofar as it is relevant to picking up (e.g. grasping) the object, i.e. it should be able to assume a picking up pose (e.g. a grasping pose) of the object determine in order to be able to correctly align its end effector (e.g. with a gripper) and to be able to move it to the correct position. Corresponding control methods for a robotic device for picking up an object in different positions are desirable.

According to various embodiments, a method for controlling a robot for recording an object in different positions is provided, comprising: defining a plurality of reference points on the object, so that the position of the reference points in three-dimensional space define a recording pose of the object, receiving a first camera image of the object in a known position of the object, so that the positions of the reference points in the first camera image are known, mapping the first camera image onto a first descriptor image using a machine learning model that is trained to assign descriptors to object points visible in camera images, regardless of their position in the camera images, identifying the descriptors of the reference points from the first descriptor image by reading out the first descriptor image at the known positions of the reference points, receiving a second camera image of the object in an unknown position in which the object is to be recorded, Mapping the second camera image onto a second descriptor image using the machine learning model, searching for the identified descriptors of the reference points in the second descriptor image, determining the positions of the reference points in three-dimensional space in the unknown location from the positions found for the descriptors of the reference points in the descriptor image, determining a recording pose of the object for the unknown position from the determined positions of the reference points in three-dimensional space and controlling the robotic device to pick up the object according to the determined pick-up pose.

The method described above enables an object to be reliably picked up (e.g. grasped) for any position of the object. A camera image of the object in an unknown position is sufficient to determine a recording pose of the object. In particular, an arrangement with multiple cameras covering multiple perspectives is not required. The machine learning model can be trained flexibly so that it can be used for different objects. It is then only necessary for reference points to be defined for each object and for the descriptors of the reference points to be determined.

Various examples are given below.

Example 1 is the method of controlling a robot to pick up an object in different postures as described above.

Example 2 is the method of example 1, wherein the positions of the reference points in the three-dimensional space in the unknown location are determined from the found positions of the descriptors of the reference points in the second descriptor image by depth information for the positions of the reference points corresponding to the positions of the reference points in the second descriptor image is projected into three-dimensional space in the second camera image.

This makes it possible to determine the recording pose using a single camera image and the associated depth information (e.g. an RGBD image).

Example 3 is the method of Example 1 or 2, wherein the reference points include at least two reference points determined to be located along an elongate portion of the object, and the shooting pose of the object for the unknown location is determined by determining an axis, which extends through the reference points is determined.

In this way, a pick-up pose can be determined that allows the robot to be controlled to grip the elongated section. For example, a gripper on the end effector of the The robot is controlled so that the gripping tongs are in a plane perpendicular to the axis that extends through the reference points, and then the gripper is closed.

Example 4 is the method of any one of Examples 1 to 3, wherein the reference points include at least three reference points set to be located on a flat surface of the object, and the object's shooting pose for the unknown posture is determined by finding a plane , which extends through the reference points, is determined.

In this way, a pick-up pose can be determined that allows the robot to be controlled to grasp the surface (e.g. for sucking or lifting). For example, a suction device on the end effector of the robot is controlled so that the suction opening or the suction nozzle of the suction device is perpendicular to the plane that extends through the reference points, and then activated.

Example 5 is the method of any one of Examples 1 to 4, wherein training the machine learning model comprises: obtaining a 3D model of the object, the 3D model comprising a grid of vertices, determining a descriptor for each vertex of the grid, generating pairs of training data images, each pair of training data images comprising a training input image showing the object and a target image, and wherein generating the target image comprises determining the vertex positions of vertices of the object model of the object which have the vertices in the training input image and assigning, for each determined vertex position in the training input image, the descriptor determined for the vertex at the vertex position to the position in the target image; and training the machine learning model through supervised learning using the training data image pairs as training data.

This training method enables a machine learning model (such as a neural network) to be trained to perform more accurate prediction (i.e. descriptor determination) compared to using self-supervised learning (i.e. allows for a greater variety of applications of the network). Furthermore, it provides greater flexibility for customizing the machine learning model so that it can be applied to different problems and reduces the training data requirements (e.g. the amount of training data required).

Example 6 is the method of Example 5, wherein generating the training data image pairs includes obtaining multiple images of the object with different poses and generating a training data image pair from each obtained image by generating a target image for the obtained image.

This allows the machine learning model (e.g. a robot with a robot controller that implements the machine learning model) to be trained to understand the topology of an object regardless of the object's pose, e.g. B. to recognize in the workspace of the robot.

Example 7 is the method of Example 5 or 6, comprising determining the vertex positions of vertices of the object model of the object that the vertices have in the training input images from the respective poses that the object has in the training input images (e.g., in the camera coordinate system ).

This enables accurate determination of vertex positions, which in turn enables accurate target images for supervised training.

Example 8 is the method of any of Examples 5 to 7, wherein the vertices of the 3D model are connected by edges, each edge having a weight specifying the proximity of two vertices connected by the edge in the object, and determining a descriptor for each vertex of the trellis by finding descriptors for the vertices that minimize the sum, across pairs of connected vertices, of distances between the descriptors of the pair of descriptors weighted by the weight of the edge between the pair of vertices.

The descriptors are clearly determined in an optimal manner. This achieves a high consistency of the descriptors. In addition, a good explainability of the descriptor mapping is achieved.

Example 9 is a robot control device configured to perform a method according to any one of Examples 1 to 8.

Example 10 is a computer program comprising instructions that, when executed by a processor, cause the processor to perform a method according to any one of Examples 1-8.

Example 11 is a computer-readable medium that stores instructions that, when executed by a processor, cause the processor to perform a method of any of Examples 1-8.

• 1 zeigt einen Roboter.
• 2 veranschaulicht das Trainieren eines neuronalen Netzes gemäß einer Ausführungsform.
• 3 zeigt eine beispielhafte Einbettung eines 4-Knoten-Graphen zur Deskriptorbestimmung.
• 4 veranschaulicht die Definition von Winkeln an Vertices eines 3D-Modells zur Kantengewichtsbestimmung gemäß dem Laplace-Beltrami-Operator.
• 5 zeigt ein Verfahren zum Trainieren eines maschinellen Lernmodells zum Erkennen einer Objekttopologie eines Objekts aus einem Bild des Objekts gemäß einer Ausführungsform.
• 6 zeigt ein Verfahren zum Steuern eines Roboters zum Aufnehmen eines Objekts in verschiedenen Lagen.

In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings, where:

• 1 shows a robot.
• 2 12 illustrates training a neural network according to one embodiment.
• 3 shows an example embedding of a 4-node graph for descriptor determination.
• 4 illustrates the definition of angles at vertices of a 3D model for edge weight determination according to the Laplace-Beltrami operator.
• 5 FIG. 10 shows a method for training a machine learning model to recognize an object topology of an object from an image of the object, according to an embodiment.
• 6 shows a method for controlling a robot to pick up an object in different positions.

The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure can be combined with one or more other aspects of this disclosure to form new aspects.

Various examples are described in more detail below.

1 shows a robot 100.

The robot 100 includes a robotic arm 101, for example an industrial robotic arm, for manipulating or assembling a work piece (or other object(s)). The robot arm 101 includes manipulators 102, 103, 104 and a base (or support) 105 by which the manipulators 102, 103, 104 are supported. The term "manipulator" refers to the movable components of the robotic arm 101, the operation of which enables physical interaction with the environment, e.g. B. to perform a task. For control, the robot 100 includes a (robot) controller 106, which is designed to implement the interaction with the environment according to a control program. The final component 104 (farthest from the support 105) of the manipulators 102, 103, 104 is also referred to as the end effector 104 and may include one or more tools, such as a welding torch, gripping instrument, paint gun, or the like.

The other manipulators 102, 103 (which are closer to the support 105) can form a positioning device so that, together with the end effector 104, the robot arm 101 is provided with the end effector 104 at its end. The robotic arm 101 is a mechanical arm that can provide functions similar to a human arm (possibly with a tool at its end).

The robotic arm 101 may include articulation elements 107, 108, 109 that connect the manipulators 102, 103, 104 to each other and to the support 105. An articulation element 107, 108, 109 may comprise one or more articulations, each of which can provide rotational movement (i.e. rotational movement) and/or translational movement (i.e. translation) for associated manipulators relative to one another. The movement of the manipulators 102, 103, 104 can be initiated by means of actuators that are controlled by the control device 106.

The term "actuator" can be understood as a component configured to effect a mechanism or process in response to its impetus. The actuator can implement instructions (the so-called activation) created by the controller 106 into mechanical movements. The actor, e.g. An electromechanical converter, for example, may be configured to convert electrical energy into mechanical energy in response to being driven.

The term "controller" may be understood as any type of logic implementing entity, which may include, for example, circuitry and/or a processor capable of executing software, firmware, or a combination thereof stored in a storage medium, and the instructions, e.g. B. to an actuator in the present example, can issue. For example, the controller can be configured by program code (e.g., software) to control the operation of a system, a robot in the present example.

In the present example, the controller 106 includes one or more processors 110 and a memory 111 storing code and data based on which the processor 110 controls the robotic arm 101 . According to various embodiments, the controller 106 controls the robotic arm 101 based on a machine learning model 112 stored in the memory 111 .

According to various embodiments, the machine learning model 112 is configured and trained to enable the robot 100 to recognize a picking pose of an object 113 placed in a workspace of the robotic arm 101, for example. This means that the robot 100 recognizes how it can pick up the object 113, i.e. how it has to orient its end effector 104 and where to move it in order to pick up the object 113 (e.g. grab it). The pick-up pose is understood to contain sufficient information for picking, i.e. information about the orientation and position 113 of the object, sufficient to determine from it how the object 113 can be gripped. The recording pose does not necessarily have to contain the complete orientation information about the object 113 since, for example, in the case of an object 113 with a rotationally symmetrical part for gripping, it may be irrelevant how the rotationally symmetrical part is rotated about its axis of rotation.

For example, the robot 100 may be equipped with one or more cameras 114 that enable it to capture images of its workspace. For example, the camera 114 is attached to the robot arm 101 so that the robot can take pictures of the object 113 from different perspectives by moving the robot arm 101 around.

According to various embodiments, the controller 106 implements a machine learning model 112 for object recognition, the output of which it uses to determine the shooting pose of the object 113 .

An example of such a machine learning model 112 for object recognition is a dense object network. A dense object mesh maps an image (e.g., an RGB image provided by camera 114) onto an arbitrary dimensional (dimension D) descriptor space image.

The dense object network is a neural network trained using self-supervised learning to output a descriptor space image for an input image of an image. However, the effectiveness of this approach relies heavily on the quality of the training data collected and the sensors involved (e.g., the cameras 114). Furthermore, the interpretation of the network prediction can be difficult.

According to various embodiments, an approach for recognizing an object and its pose is used, assuming that a 3D model (e.g., a CAD (Computer Aided Design) model) of the object is known, which is typically used for industrial assembly or processing tasks is the case. According to various embodiments, non-linear dimensionality reduction techniques are used to compute optimal target images for training input images for a neural network. Thus, instead of using self-supervised neural network training, according to various embodiments, supervised neural network training is used. RGBD images of an object can also be recorded (RGB + depth information) and a 3D model for the object can be determined from them.

According to one embodiment, data collection is first performed to generate training data for training the machine learning model 112 . Specifically, for example, registered RGB (Red-Green-Blue) images are collected. A registered image here means an RGB image with known intrinsic and extrinsic camera values. For example, in a real-world scenario, a robot-mounted camera 114 (e.g., a robot wrist-mounted camera) is used to scan an object as the robot (e.g., robotic arm 101) moves around . Other extrinsic estimation techniques, e.g. B. ChArUco markers can be used, ie an object can be placed in different positions and poses relative to a ChArUco board and images of the arrangement (of ChArUco board and object) are recorded. In a simulated scenario, photorealistic generated RGB images are used using known object poses.

After collecting the RGB images, target images for the RGB images are rendered for supervised neural network training.

It is assumed that the pose of each object in world coordinates in each collected RGB image is known. This is straightforward for a simulated scenario, but requires manual tuning for a real-world scenario, e.g. B. Placing the object at predefined positions. RGBD images can also be used to determine the position of an object.

With this information and using a vertex descriptor computation technique, such as described below, for each RGB image (i.e., training input image), a descriptor image (i.e., training output image, also known as a target image or ground truth image) is rendered.

When a target image has been generated for each RGB image, i.e. pairs of RGB images and target images have been formed, these pairs of training input image and associated target image can be used as training data for training a neural network, as in 2 illustrated.

2 2 illustrates training a neural network 200 according to one embodiment.

The neural network 200 is a fully convolutional network that maps an h×w×3 tensor (input image) to an h×w×D tensor (output image).

It comprises several stages 204 of convolutional layers, followed by a pooling layer, upsampling layers 205 and skip connections 206 to combine the outputs of different layers.

For training, the neural network 200 receives a training input image 201 and outputs an output image 202 with pixel values in descriptor space (e.g. color components according to descriptor vector components). A training loss is calculated between the output image 202 and the target image 203 associated with the training input image. This can take place for a batch of training input images and the training loss can be averaged over the training input images and the weights of the neural network 200 trained using stochastic gradient descent using the training loss. For example, the training loss computed between the output image 202 and the target image 203 is an L2 loss function (to minimize a pixel-by-pixel least squares error between the target image 203 and the output image 202).

The training input image 201 shows an object and the target image and the output image contain vectors in descriptor space. The vectors in the descriptor space can be mapped to colors so that the output image 202 (as well as the target image 203) resembles a heat map of the object.

The vectors in the descriptor space (also called (dense) descriptors) are d-dimensional vectors (e.g. d is 1, 2 or 3) corresponding to each pixel in the respective image (e.g. each pixel of the input image 201, under assuming that the input image 201 and the output image 202 have the same dimension). The dense descriptors implicitly encode the surface topology of the object shown in the input image 201, invariant to its pose or camera position.

Given a 3D model of the object, it is possible to analytically determine an optimal (in the Riemannian sense) and unique descriptor vector for each vertex of the 3D model of the object. According to various embodiments, target images are generated for registered RGB images using these optimal descriptors (or estimates of these descriptors determined by an optimization), resulting in fully supervised neural network 200 training. In addition, the descriptor space becomes explainable and optimal regardless of the chosen descriptor dimension d.

In the following, the 3D models are considered as Riemannian manifolds ℳ embedded in ℝ ³ which evokes computing geodesics (shortest path between vertices) while considering the geometry. An optimal surface topology can be explicitly encoded by embedding the 3D model in a d-dimensional Euclidean descriptor space such that geodesic distances between neighboring vertices are preserved as best as possible. This Euclidean space is regarded as the descriptor space and a search for the optimal mapping f : ℳ → ℝ ^d is performed. According to one embodiment, the Laplacian computation for lattices and its eigenvalue decomposition are used to determine (or at least estimate) the optimal embedding of vertices in the descriptor space. Thus, instead of separating geodesic computation and mapping optimization, the descriptors are extracted in a single framework by computing the Laplacian of the 3D model.

According to the approach described below, the embedding of a 3D object model in Euclidean space in the descriptor space is determined to preserve distances (e.g., geodesic distances) between vertices.

Knoten in einen ungerichteten Graphen entsprechen. W_ij ≥ 0 soll die Verbindungsstärke oder die Nähe zwischen zwei Knoten x_i und x_j bezeichnen, z.B. W_{i j} = exp(- || x_i -x_j ||₂).A set of points is to be used for dimensionality reduction via the Laplace operator ${\left\{x_i\right\}}_{i=1}^N,x\in {ℝ}^D$

Correspond to nodes in an undirected graph. Let W _ij ≥ 0 denote the connection strength or proximity between two nodes x _i and x _j , eg W _ij = exp(- || x _i -x _j || ₂ ).

zu finden (typischerweise gilt d < D), sodass, falls x_i und x_j nahe liegen, ihre Einbettung dann auch nahe sein sollte: $$Y*=\text{arg}\underset{Y}{\text{min}}\frac{1}{2}{\displaystyle \sum _{j=1}^N{\displaystyle \sum _{i=1}^N{\omega }_{ij}}}{\Vert y_i-y_j\Vert }_2,$$

wobei Y = [y₁,...,y_N]_d×N.The goal is a d-dimensional embedding ${\left\{y_i\right\}}_{i=1}^N,y\in {ℝ}^{\mathrm{i.e}}$

(typically d < D), so if x _i and x _j are close, then their embedding should also be close: $$Y*=\text{bad}\underset{Y}{\text{at least}}\frac{1}{2}{\displaystyle \sum _{j=1}^N{\displaystyle \sum _{i=1}^N{\omega }_{ij}}}{\Vert y_i-y_j\Vert }_2,$$

where Y = [y ₁ ,...,y _N ] _d×N .

wobei L_N×N die positiv-semidefinite Laplace-Matrix ist. A ist die Adjazenzmatrix mit Elementen A_ij = A_ji = w_ij und D_ii = Σj A_ij. Es sollte angemerkt werden, dass die optimale Lösung Y* eine beliebige Skalierung und Tendenz aufweisen kann.The optimization problem (1) is equivalent to $$Y*=\text{bad}\underset{Y}{\text{at least}}\text{Tr}\left(YL{Y}^T\right),L=D-A,$$

where L _N×N is the positive semidefinite Laplacian matrix. A is the adjacency matrix with elements A _ij = A _ji = w _ij and D _ii = Σj A _ij . It should be noted that the optimal solution Y* can have any scale and bias.

unter Verwendung beschränkter Optimierung mit Lagrange-Parametern λ ∈ ℝ^d $$\begin{array}{c}\mathcal{J}\left(Y,\lambda \right)=\text{Tr}\left(YL{Y}^T\right)-Tr\left(\text{diag}\left(\lambda \right)\left(YD{Y}^T-I\right)\right),\\ \frac{\partial \mathcal{J}\left(Y,\lambda \right)}{\partial Y}=2L{Y}^T-2\text{diag}\left(\lambda \right)D{Y}^T,\\ L{Y}^T=diag\left(\lambda \right)D{Y}^T,\end{array}$$

was ein generalisiertes Eigenwertproblem ist, das durch standardmäßige Bibliotheken der linearen Algebra gelöst werden kann. Da L und D positiv-(semi)definite Matrizen sind, können die Eigenwerte als λ₀ = 0 ≤ λ₁ ≤ ... . ≤ λ_N-1. geschrieben werden.To remove this randomness, the weighted second moment can be normalized with Y ^T DY = I, which enforces unit variance in different dimensions. The resulting optimization problem then becomes $$\begin{array}{c}Y*=\text{bad}\underset{Y}{\text{at least}}\text{Tr}\left(YL{Y}^T\right),\\ s.t.YD{Y}^T=I\end{array}$$

using restricted optimization with Lagrangian parameters λ ∈ ℝ ^d $$\begin{array}{c}\mathcal{J}\left(Y,\lambda \right)=\text{Tr}\left(YL{Y}^T\right)-T\mathrm{right}\left(\text{diag}\left(\lambda \right)\left(YD{Y}^T-I\right)\right),\\ \frac{\partial \mathcal{J}\left(Y,\lambda \right)}{\partial Y}=2L{Y}^T-2\text{diag}\left(\lambda \right)D{Y}^T,\\ L{Y}^T=\mathrm{i.e}iaG\left(\lambda \right)D{Y}^T,\end{array}$$

which is a generalized eigenvalue problem that can be solved by standard linear algebra libraries. Since L and D are positive (semi)definite matrices, the eigenvalues can be written as λ ₀ = 0 ≤ λ ₁ ≤ ... . ≤ λ _N-1 . to be written.

Furthermore, the first eigenvector (first column of Y ^T ) equals 1 (all ones vector), which is a trivial solution that maps each vertex to a point. In addition, any two are eigenvectors orthogonal to each other. The solution to the eigenvalue problem gives N eigenvalues and corresponding eigenvectors with dimensionality N. In practice, however, only the first d eigenvectors corresponding to the lowest eigenvalues (except for the trivial solution) are used.

Thus, the ith column of Y is the embedding of node i in R ^d , while each row represents the embedding of each point in different orthogonal dimensions.

3 shows an example embedding of a 4-node graph.

das heißt, die Eigenwerte entsprechen den Einbettungsfehlern in verschiedenen Dimensionen. Zur Einfachheit halber soll d = 1 sein, in welchem Fall jedes x auf Punkt y = 1 abgebildet wird. In diesem Fall wird (5) vereinfacht zu: $${\widehat{y}}^{T}L\widehat{y}={\lambda }_0,\widehat{y}={\left[y_1,\dots ,y_N\right]}^T=1^T.$$

The eigenvalues have an important meaning with regard to the optimality of the embedding. In the case of an optimal embedding Y*, the constraint Y*DY* ^T = I is satisfied, and hence holds $$Y*LY{*}^T=\mathrm{i.e}iaG\left(\lambda \right),$$

that is, the eigenvalues correspond to the embedding errors in different dimensions. For simplicity, let d=1, in which case each x maps to point y=1. In this case (5) is simplified to: $${\widehat{y}}^{T}L\widehat{y}={\mathrm{<figure-callout\; id="0"\; label="\lambda "\; filenames="DE102020214301A1\_0020.png"\; state="\{\{state\}\}">\lambda </figure-callout>}}_0,\widehat{y}={\left[{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="DE102020214301A1\_0020.png"\; state="\{\{state\}\}">y</figure-callout>}}_1,...,y_N\right]}^T=1^T.$$

kann die ursprüngliche Zielsetzung übersetzt werden, um den Einbettungsfehler in jeder Dimension zu minimieren. Ungeachtet des gewählten d ist somit der resultierende Deskriptorvektor optimal.Since λ ₀ = 0, that means that if all vertices of the object are mapped to one point, the embedding error is 0 since the distance between all points y is 0. This is not useful for practical purposes and so the first eigenvalue and eigenvector are omitted. Using d = 2 corresponds to mapping each point x onto a line, and λ1 is the corresponding embedding error, and so on. Since the eigenvectors are orthogonal to each other, increasing d will add new dimensions to the embedding, tending to minimize the error in the new orthogonal dimension. The same effect can be seen in (3): da $\text{Tr}\left(YL{Y}^T\right)={\displaystyle {\sum }_{i=1}^{\mathrm{i.e}}{\lambda }_i},$

the original objective can be translated to minimize the embedding error in each dimension. Thus, regardless of the d chosen, the resulting descriptor vector is optimal.

In some cases subsequent eigenvalues are identical, that is, λ _i = λ _i+1 = λ _i+2 = ... (see the example of 3 , where the eigenvalues for d=2 and d=3 are the same). This carries some information regarding the symmetry where there are several orthogonal dimensions with identical embedding error. Indeed, in the 4-node graph example of 3 , if the graph is fully connected, the embedding is symmetric in every dimension and all eigenvalues are identical except for the trivial solution.

The graph embedding approach above can be directly applied to grids, point clouds, etc. For example, the K-Nearest Neighbor (KNN) algorithm can be used to form local connections between vertices and construct the adjacency matrix. This approach is sufficient to construct the graph Laplacian and calculate the embedding of each vertex. Nonetheless, this approach inherently builds on Euclidean distance metrics and heuristics that do not necessarily respect the underlying Riemannian geometry of the 3D object model. For example, some edges may pass through the object, or may not connect adjacent vertices of the grid. Even a few incorrect entries in the adjacency matrix can result in poor embedding performance. According to one embodiment, when working with models, it is therefore guaranteed that geodesic distances between any two vertices are correct or have a minimal approximation error.

In general, object models such as lattices or point clouds can be represented as Riemannian manifolds embedded in ℝ ³ . A Riemannian manifold M with a smoothly changing metric g can be considered “locally Euclidean”, which captures the property of local smoothness of real-world objects. The generalization of the Laplace operator to Riemannian manifolds is the Laplace-Beltrami (LB) operator Δ. Similar to the Laplacian in Euclidean spaces, the LB operator applied to a function is the divergence of the function's gradient. While the Laplace operator is easy to compute for graphs and in Euclidean spaces (either from adjacency information or finite differences), the LB operator in differential geometry relies on exterior calculus, and is not readily available for a manifold in general .

However, for known discrete manifolds, such as lattices, the LB operator can be approximated. This provides a highly efficient and simple calculation framework when working with grids, point clouds, etc. Since the Riemannian equivalent of the Laplace operator is the Laplace Beltrami, the embedding approach described above can be applied directly to Δ. The eigenvectors Y of Δ will represent the optimal d-dimensional Euclidean embedding of the vertices of the lattice.

Δ can be calculated efficiently for lattices as follows. Assuming a lattice with N vertices V, faces F, and edges E is given. In this case, the size of Δ is N × N. The ith row of Δ describes the adjacency information of the ith vertex to its connecting vertices. Let φ be any function on the lattice. Then applying the discrete LB operator to this function maps to Δφ. The ith element of this function can be described by: $${\left(\mathrm{\Delta }\varphi \right)}_i=\frac{1}{2}{\displaystyle \sum _{ij\in E}\left(\text{cotton}\left({\theta }_k^{ij}\right)+\text{cotton}\left({\theta }_l^{ji}\right)\right)}\left({\varphi }_j-{\varphi }_i\right),$$

und ${\theta }_l^{ji}.$

4 illustrates the definition of angles ${\theta }_k^{ij}$

and ${\theta }_l^{ji}.$

agiert als das Verbindungsgewicht w_ij. Diese in (7) auftretenden Gewichte, d. h. die Gewichte des LB-Operators, wenn bei dem Gitter angewendet, werden gemäß einer Ausführungsform als die Gewichte zum Bestimmen von D und A der Gleichung (2) verwendet.The sum of the cotangent expressions $\text{cotton}\left({\theta }_k^{ij}\right)+\text{cotton}\left({\theta }_l^{ji}\right)$

acts as the connection weight w _ij . These weights appearing in (7), ie the weights of the LB operator when applied to the trellis, are used according to one embodiment as the weights for determining D and A of equation (2).

It should be noted that since α > π / 2 => cot(α) < 0,, negative connection weights w _ij can occur, especially if one angle is significantly larger than the others (not good faces). To overcome this problem, the connection weight can be approximated by edge flipping.

The descriptor generation approach described above treats each vertex uniquely. That is, each vertex is assigned a unique descriptor. However, objects can be symmetric, and thus assigning unique descriptors to seemingly identical vertices will result in non-symmetric embeddings.

To address this issue, according to various embodiments, intrinsic symmetries of shapes are detected and symmetric embeddings are compressed such that symmetric vertices map to the same descriptor. A shape can be shown to have intrinsic symmetry if the eigenfunctions of the Laplace-Beltrami operator appear symmetric in Euclidean space. In other words, symmetric features of a grid, point cloud, etc. are detected if their geodesic-preserving Euclidean embedding (descriptor space) shows Euclidean symmetry. A compact manifold exhibits intrinsic symmetry if there exists a homeomorphism T that preserves geodesic distances between each vertex of the manifold.

dann ist ƒ eine GISIF. Beispielsweise soll auf einem Torus dieser Homöomorphismus eine beliebige Drehung um die z-Achse sein. Dies bedeutet, dass, falls ƒ eine GISIF ist, sie dann gegenüber dieser Drehung invariant sein muss.So-called global intrinsic symmetry-invariant functions (GISIFs) can be used for the compression of the symmetric descriptors. Assuming a global intrinsic symmetric homeomorphism T : ℳ → ℳ, st g(p,q) = g(T(p), T(q)) ∀p,q ∈ ℳ ,, where g represents geodesic distance, and a Function on the manifold ƒ, if for every point p on the manifold it holds: $$ƒ\circ T\left(p\right)=ƒ\left(T\left(p\right)\right)=ƒ\left(p\right),$$

then ƒ is a GISIF. For example, on a torus, this homeomorphism should be any rotation around the z-axis. This means that if ƒ is a GISIF, then it must be invariant to this rotation.

Furthermore, it can be shown that in the case of identical eigenvalues λ _i = λ _i+1 = ... = λ _i+N , such a GISIF is the squared sum of the eigenvector components of the point, ie $$ƒ\left(p\right)={\displaystyle {\sum }_i^{i+N}{\varphi }_i}{\left(p\right)}^2.$$

This is consistent with the identical eigenvalue analysis above, which is the necessary condition of symmetric embedding. Since in practice there are rarely identical eigenvalues due to numerical constraints, a heuristic can be used where eigenvalues are considered identical if they lie within the same ε-sphere (with a small ε), i.e. if they differ by less than a predetermined threshold, e.g. 0.1% or 0.01%. Since symmetrical dimensions only need to be found once for a given object, this can be done manually.

For example, let the first 7 eigenvalues of an eigenvalue decomposition for a torus be:
λ _0-6 = [0.0,0.0175,0.0175,0.069,0.069,0.1525,0.1525]

A GISIF embedding in ℝ ³ - without considering the trivial solution - is then given by $$ƒ\left(p\right)=\left[{\varphi }_1{\left(p\right)}^2+{\varphi }_2{\left(p\right)}^2,{\varphi }_3{\left(p\right)}^2+{\varphi }_4{\left(p\right)}^2,{\varphi }_5{\left(p\right)}^2+{\varphi }_6{\left(p\right)}^2\right].$$

In the case of multiple objects, this can be represented as multiple separate connected graphs. In this case, the adjacency matrix is block diagonal. The symmetric positive-definite Laplacian will again have orthogonal eigenvectors. There are two differences in the result of eigenvalue decomposition compared to the single graph embedding case: first, the non-decreasing eigenvalues will be the embedding errors of all unordered objects. Second, since the corresponding eigenvalues remain orthogonal, the eigenvectors will have zero entries. This means that each dimension of the descriptor space will only correspond to one object embedding. Furthermore, the dimensions are ordered with respect to the embedding error of the corresponding object. Thus, if a 3-dimensional embedding of two objects should be created, d=8 is used since there are two trivial solutions corresponding to λ=0.

This straightforward approach handles multiple objects independently, while there may be suboptimal methods that still provide reasonably good lower-d embeddings that exploit correlation between objects.

Given the poses of objects, the target images can be generated by projecting the descriptors onto the image plane. Descriptor space image random noise or a single descriptor mapped to the farthest point in descriptor space can be used as the non-object (background).

In order to improve the robustness of the trained network 200, image augmentation methods such as domain randomization or perturbations such as Gaussian blurring, cropping or dropout can be applied.

If the machine learning model 112, e.g. the neural network 200, has now been trained to map camera images of an object 113 onto descriptor images, the following procedure can be used to determine a recording pose of an object 113 in an unknown position.

First, several reference points p _i , i=1, . . . N, are selected on the object 113 and descriptors of these reference points are determined. This can be done by taking a camera image of the object 113, selecting reference pixels (u _j , v _j ) on the object (and thus corresponding reference points of the object) and mapping the camera image from the neural network 200 to a descriptor image. Then the descriptors at the positions in the descriptor image given by the positions of the reference pixels can be taken as descriptors of the reference points, ie the descriptors of the reference points are _di = I ^d (u _j , v _j ), where I ^d = ƒ (I; θ) is the descriptor image, where ƒ is the mapping (from camera image to descriptor image) implemented by the neural network, I is the camera image and θ is the weights of the machine learning model 200.

If the object 113 is now in an unknown position, a camera image I _new is again recorded and an associated descriptor image I ^d _new =ƒ{I _new ; θ) determined. In this new descriptor image, descriptors are now sought which are as close as possible to the d _i descriptors of the reference images, for example by $$\left({\mathrm{and}}_i,v_i\right)*={\text{argmin}}_{{\mathrm{and}}_i,v_i}{\Vert I^{\mathrm{i.e}}{}_{ne\mathrm{and}}\left({\mathrm{and}}_i,v_i\right)-{\mathrm{i.e}}_i\Vert }_2{}^2\text{f}\ddot{\text{and}}\text{r all}i=\mathrm{1,}...,N.$$

The positions of the reference points in three-dimensional space are determined from the thus determined or estimated positions (u _i , v _j )* of the reference points in the descriptor image I ^d _new (and thus correspondingly in the new camera image I _new ). For example, a depth image is recorded together with the camera image I _new (or the camera image I _new has a depth channel, e.g. it is an RGBD image), so that from (u _i , v _j )* the three-dimensional position of the i-th reference point p _i can be determined (by projecting the depth value at the position (u _i ,v _j )* into the respective workspace coordinate system).

If the positions in space of several reference points are known, a recording pose can be determined from this, as is shown in 5 is shown.

For example, the position of two reference points p ₁ and p ₂ on the object 500 in space is determined and the two are linearly combined, eg their mean value is taken, in order to define an anchor point 504 . In order to define a gripping orientation, a first axis 501 is defined with the direction p ₁ and p ₂ and a second axis 502 is defined arbitrarily by the anchor point 504, e.g. in the z-axis direction of the camera 114 or in the direction of an axis of the work area coordinate system . A third axis 503 through the anchor point 504 can be calculated by the vector product of the direction vector of the first axis 501 and the direction vector of the second axis 502 . The three axes 501-503 and the anchor point 504 define a pick-up pose for the object 500. A robot can then be controlled to grasp the shaft of the object 500 extending in the direction of the first axis. The reference points p ₁ and p ₂ are defined, for example, such that they extend along the shaft, ie along an elongate part of the object suitable for gripping, as shown.

Analogously, three or more reference points can be arranged on a gripping surface of an object, so that a full 6D recording pose of the object or also the orientation of a gripping surface on which an object can be gripped can be determined from their position.

It should be noted that the gripper does not necessarily have to be in the form of pliers, but can also have a suction device, for example, in order to suck the object on a suitable surface and thus pick it up. In this case, for example, in order to bring the suction device into the correct position, it may be desirable to determine a pickup pose that indicates the orientation and position of a surface of the object that is suitable for suction. This can be done, for example, by determining an anchor point and a plane normal vector in the anchor point.

More than three reference points can also be used to determine a recording pose, e.g. to reduce errors through averaging.

In summary, according to various embodiments, a method as in 6 illustrated provided.

6 shows a method for controlling a robot to pick up an object in different positions.

In 601, several reference points are set on the object, so that the position of the reference points in three-dimensional space define a recording pose of the object.

In 602 a first camera image of the object is received in a known position of the object, so that the positions of the reference points in the first camera image are known.

In 603, the first camera image is mapped onto a first descriptor image using a machine learning model that is trained to assign descriptors to object points that are visible in camera images, regardless of their position in the camera images.

In 604 the descriptors of the reference points are identified from the first descriptor image by reading out the first descriptor image at the known positions of the reference points.

In 605, a second camera image of the object is received in an unknown location in which the object is to be photographed.

In 606, the second camera image is mapped to a second descriptor image using the machine learning model.

In 607 the identified descriptors of the reference points are searched in the second descriptor image.

In 608 the positions of the reference points in the three-dimensional space in the unknown position are determined from the positions of the descriptors of the reference points in the descriptor image that were found.

In 609, a recording pose of the object for the unknown position is determined from the determined positions of the reference points in three-dimensional space.

In 610 the robotic device is controlled to pick up the object according to the determined picking pose.

In other words, according to various embodiments, the positions of a plurality of reference points are identified using their descriptors in a camera image that shows the object in an unknown position, and their position in three-dimensional space is determined from their position in the camera image. A recording pose (e.g. the orientation of one or more axes for gripping the object) is then determined from the position in three-dimensional space of the multiple reference points.

"Picking up" means, for example, gripping with a gripper. However, other types of holding mechanisms can also be used, such as a sucker for sucking the object. Also, "picking up" does not necessarily mean moving the object alone, e.g. it can also take a component on a larger structure and bend it without separating it from the larger structure.

For example, the machine learning model is a neural network. However, other machine learning models can also be used, which are trained accordingly.

According to various embodiments, the machine learning model assigns descriptors to pixels of the object (in the image plane of the respective camera image). This can be viewed as indirectly encoding the surface topology of the object. This connection between descriptors and the surface topology can be made explicit by rendering to map the descriptors to the image plane. It should be noted that descriptor values at faces (ie points that are not vertices) of the object model can be determined using interpolation. For example, if a surface is given by 3 vertices of the object model with their respective descriptor values y1, y2, y3, then at any point on the surface the descriptor value y can be given as a weighted sum of these values w ₁ y ₁ + w ₂ y ₂ + w ₃ · y ₃ can be calculated. In other words, the descriptor values are interpolated at the vertices.

For example, to generate image pairs for training data for the machine learning model, an image of the object (e.g. an RGB image) including the object (or several objects) with known 3D (e.g. CAD) model and known pose ( in a global (i.e. world) coordinate system, is mapped to a (dense) descriptor image that is optimal in the sense that it is obtained by a search for descriptors to minimize the deviation of geometric properties (especially the proximity of points of the object) between the object model and of its representation (embedding) in the descriptor space. In practical use, the theoretical optimal solution for minimization will in general not be found, since the search is restricted to a certain search space. Nevertheless, an estimate of the minimum within the constraints of a practical application (available calculation accuracy, maximum number of iterations, etc.).

Thus, descriptors for the vertices are sought by performing a minimization process of the sum, across pairs of connected vertices, of distances between the descriptors of the pair of vertices, weighted by the weight of the edge between the pair of vertices, each descriptor for a respective vertex of the vertices is searched.

Each pair of training data images comprises a training input image of the object and a target image, the target image being formed by projecting the descriptors of the vertices visible in the training input image onto the Training input image plane is generated according to the pose that the object has in the training input image.

The images along with their associated target images are used for supervised training of the machine learning model.

The machine learning model is thus trained to recognize unique features of an object (or objects). This information can be used for various robot control applications by evaluating the machine learning model in real time, e.g. B. Predicting an object gripping pose for assembly. It should be noted that the supervised training approach allows for explicit coding of symmetry information.

The procedure of 6 can be performed by one or more computers that include one or more data processing units. The term "computing unit" can be understood as any type of entity that enables the processing of data or signals. For example, the data or signals may be processed according to at least one (ie one or more than one) specific function performed by the data processing unit. A data processing unit may be an analog circuit, a digital circuit, a mixed-signal circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array (FPGA), include or be formed from an integrated circuit or any combination thereof. Any other way of implementing the respective functions, which will be described in more detail below, can also be understood as a data processing unit or logic circuitry. It is understood that one or more of the method steps detailed herein may be performed (e.g., implemented) by a computing device via one or more specific functions performed by the computing device.

Various embodiments may receive and use sensor signals from various sensors such as a (e.g., RGB) camera, video, radar, LiDAR, ultrasound, thermal imaging, etc. to obtain, for example, sensor data showing an object. Embodiments can be used to generate training data and train a machine learning system, e.g. B. for autonomous control of a robot, z. B. a robot manipulator to achieve different manipulation tasks under different scenarios. In particular, embodiments in the control and monitoring of the execution of manipulation tasks z. B. in assembly lines applicable.

Although specific embodiments have been illustrated and described herein, one of ordinary skill in the art should recognize that a variety of alternative and/or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the present invention. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that this invention be limited only by the claims and their equivalents.

Claims (11)

A method for controlling a robot to pick up an object in different positions, comprising: defining a plurality of reference points on the object, such that the position of the reference points in three-dimensional space define a pick-up pose of the object; receiving a first camera image of the object in a known position of the object, so that the positions of the reference points in the first camera image are known; mapping the first camera image to a first descriptor image using a machine learning model that is trained to assign descriptors to object points visible in camera images, regardless of their position in the camera images; identifying the descriptors of the reference points from the first descriptor image by reading out the first descriptor image at the known positions of the reference points; receiving a second camera image of the object in an unknown location in which the object is to be photographed; mapping the second camera image to a second descriptor image using the machine learning model; searching the identified descriptors of the reference points in the second descriptor image; Determining the positions of the reference points in three-dimensional space in the unknown location from the found positions of the descriptors of the reference points in the descriptor image; determining a recording pose of the object for the unknown position from the determined positions of the reference points in three-dimensional space; and controlling the robotic device to pick up the object according to the determined picking pose. procedure according to claim 1 , wherein the positions of the reference points in three-dimensional space in the unknown location are determined from the found positions of the descriptors of the reference points in the second descriptor image by projecting depth information for the positions of the reference points in the second descriptor image that correspond to the positions of the reference points in the second camera image into the three-dimensional space will. procedure according to claim 1 or 2 wherein the reference points include at least two reference points determined to be located along an elongate portion of the object, and wherein the taking pose of the object for the unknown location is determined by determining an axis extending through the reference points. Method according to one of Claims 1 until 3 wherein the reference points include at least three reference points that are set to be located on a flat surface of the object, and wherein the taking pose of the object for the unknown attitude is determined by determining a plane that extends through the reference points. Method according to one of Claims 1 until 4 , wherein training the machine learning model comprises: obtaining a 3D model of the object, the 3D model comprising a lattice of vertices; determining a descriptor for each vertex of the grid; generating pairs of training data images, each pair of training data images comprising a training input image showing the object and a target image, and wherein generating the target image comprises determining the vertex positions of vertices of the object model of the object that have the vertices in the training input image; and assigning, for each determined vertex position in the training input image, the descriptor determined for the vertex at the vertex position to the position in the target image; and training the machine learning model through supervised learning using the training data image pairs as training data. procedure after claim 5 wherein generating the training data image pairs comprises obtaining a plurality of images of the object in different poses and generating a training data image pair from each obtained image by generating a target image for the obtained image. procedure after claim 5 or 6 , comprising determining the vertex positions of vertices of the object model of the object that the vertices have in the training input images from the respective poses that the object has in the training input images. Procedure according to one of Claims 5 until 7 , wherein the vertices of the 3D model are connected by edges, each edge having a weight specifying the proximity of two vertices connected by the edge in the object, and determining a descriptor for each vertex of the grid by finding descriptors for the vertices , which minimizes the sum, over pairs of connected vertices, of distances between the descriptors of the pair of vertices weighted by the weight of the edge between the pair of vertices. Robot control device that is set up, a method according to any one of Claims 1 until 8th to perform. A computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of Claims 1 until 8th performs. A computer-readable medium storing instructions which, when executed by a processor, cause the processor to perform a method according to any one of Claims 1 until 8th performs.

Images