DE102022202321A1 - Device and method for controlling a robot - Google Patents

Abstract

According to various embodiments, a method for training a neural network to recognize objects in image data is described, comprising, for each image of a plurality of training images, determining, for each of a plurality of object bounding boxes, bounding box coordinates, object hierarchy information, and classification information about objects that are the correspond to object bounding boxes, and training the neural network to reduce a loss between basic knowledge for the hierarchy information, for the bounding box coordinates and for the classification information and the hierarchy information, wherein the bounding box coordinates and the classification information have been determined for the training images, respectively.

Description

State of the art

The present disclosure relates to devices and methods for controlling a robot.

Robotic devices can be trained for object manipulation using machine learning. This typically involves training a machine learning model to perform object detection, such as in N. Carion et al., “End-to-end object detection with transformers,” in European Conference on Computer Vision, Springer, 2020, pages 213-229 , referred to herein as Reference 1, or in X. Zhu et al. “Deformable detr: Deformable transformers for end-to-end object detection,” arXiv preprint arXiv:2010.04159, 2020, referred to herein as reference 2.

In particular, object manipulation with a robotic arm is of great interest and has a wide range of applications, e.g. B. in industrial manufacturing or search & rescue. This can particularly be applications where many objects accumulate and the objects are therefore difficult to detect due to occlusions etc. However, even in such cases it may be necessary for objects to be reliably recognized. Furthermore, their stacking relationships should be determined, e.g. B. to decide which object should be manipulated first.

Accordingly, reliable approaches to detecting objects are desirable when multiple objects are arranged together, e.g. B. in the work area of a robot, so that stacking relationships exist between them (i.e. the objects are at least partially piled up), and to recognize these stacking relationships.

Disclosure of the invention

• • Bestimmen von Merkmalsvektoren aus dem Trainingsbild durch einen Encoder eines Transformer-Encoder-Decoder-Netzes;
• • Bestimmen, für jeden einer Mehrzahl von Objektbegrenzungsrahmen, von Begrenzungsrahmenkoordinaten aus den Merkmalsvektoren durch einen Decoder des Encoder-Decoder-Netzes und einen Begrenzungsrahmenkopf;
• • Bestimmen von Objekthierarchieinformationen aus den Begrenzungsrahmenkoordinaten für die Mehrzahl von Objekten und von Merkmalen für die Begrenzungsrahmen, die aus den Merkmalsvektoren abgeleitet werden, wobei die Objekthierarchieinformationen Stapelbeziehungen der Mehrzahl von Objekten in Bezug zueinander spezifizieren; und
• • Bestimmen von Klassifizierungsinformationen über Objekte, die den Objektbegrenzungsrahmen entsprechen, durch einen Klassifizierungskopf.

According to various embodiments, a method for training a neural network to recognize objects in image data is provided, comprising, for each image of a plurality of training images,

• • Determining feature vectors from the training image by an encoder of a transformer-encoder-decoder network;
• • determining, for each of a plurality of object bounding boxes, bounding box coordinates from the feature vectors by a decoder of the encoder-decoder network and a bounding box header;
• • determining object hierarchy information from the bounding box coordinates for the plurality of objects and features for the bounding boxes derived from the feature vectors, the object hierarchy information specifying stacking relationships of the plurality of objects with respect to one another; and
• • Determining classification information about objects corresponding to the object bounding boxes by a classification head.

The method further includes training the neural network to reduce a loss between basic knowledge for the hierarchy information, for the bounding box coordinates and for the classification information and the hierarchy information, where the bounding box coordinates and the classification information have been determined for the training images, respectively.

By considering bounding boxes, bounding box classification information and adjacency information when training according to the above method, i.e. H. By training the neural network to determine bounding boxes and classifications for the bounding boxes and training the neural network to determine adjacency information according to the above method, high performance of the (trained) neural network can be achieved for these three inference results. Here, for example, performance refers to a combination of classification accuracy, bounding box inference accuracy, and object relation inference accuracy via the adjacency matrix. For example, this is made up of object precision (which describes how many of the detected tuples (detection, classification and relationship) are all correct) and object recall (which describes how many correct tuples were detected from all target tuples in one Image) that can be achieved are derived. For example, performance may refer to image accuracy, which indicates the percentage of images where the OP would be 100% and the OR would be 100%, in other words, the images where all target tuples were correctly detected and no more.

By taking adjacency information into account, the neural network is able to infer the spatial relationships between objects shown in images and can thus take this information into account when recognizing (detecting) objects, such as taking into account shadows cast by other objects , and parts of an object that are obscured. The specific adjacency information (also called object hierarchy information, e.g. in the form of an adjacency matrix) may or may not be further used by downstream components. For example, a robotic device controller may control a robotic device based purely on the output of the visual task (bounding box and classification). In other words, in one embodiment, the object hierarchy information (adjacency information) is used only for training.

Various examples are described below.

Example 1 is the method for training a neural network to recognize objects in image data as described above.

Example 2 is the method of Example 1, comprising transforming the feature vectors into a feature vector for each object bounding box, concatenating, for each object bounding box, the feature vector for the bounding box with the bounding box coordinates for the object bounding box into an input vector, and determining the object hierarchy information from the input vectors.

Therefore, the feature values are transformed through dimension transformation to adapt to the number of bounding boxes. This enables convenient input into layers of a neural network to determine the object hierarchy through concatenation.

Example 3 is the method of Example 2, comprising determining the object hierarchy information from the input vectors through a plurality of processing channels, each processing channel receiving the input vectors for the object bounding boxes and producing an adjacency matrix component, and multiplying the adjacency matrix components generated by the channels to form an adjacency matrix generate that specifies the object hierarchy information.

For example, the adjacency matrix is square and the adjacency matrix components are rectangular and produce the adjacency matrix when multiplied. For example, each channel output has dimensions Q x H, where Q = 300 (bounding box queries) and H = 260 (the hidden size of the encoder plus that of the bounding box coordinates). Multiplying the outputs of the channels results in the adjacency matrix. This matrix multiplication allows the elements of the adjacency matrix to be invariant with respect to object recognition order, for example the relationship between objects 125 and 262 should be identical if they were instead identified as objects 231 and 51, for example. In other words, query order doesn't matter.

Example 4 is the method according to any of Examples 1 to 3, wherein the neural network is formed using a combined loss of a cross-entropy loss for the classification information, a binary element-wise loss for the object hierarchy information, and a linear combination of absolute deviation loss and generalized overlap loss for the bounding box coordinates is trained.

This loss provides robust training while taking all three neural network outputs (classifiers, bounding boxes, and object hierarchy) into account.

Example 5 is a method for controlling a robotic device, comprising training a machine learning model according to any one of Examples 1 to 4, capturing a camera image showing the one or more objects, feeding the camera image to the neural network to display the one or more objects Detect objects and control the robot taking into account the detected one or more objects.

Example 6 is a controller designed to perform a method according to any of Examples 1 to 5.

Example 7 is a computer program that includes instructions that, when executed by a computer, cause the computer to perform a method according to any of Examples 1 to 5.

Example 8 is a computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform a method according to any of Examples 1 to 5.

• 1 einen Roboter zeigt.
• 2 einen Betrieb der Steuerung für eine Bin-Picking-Aufgabe veranschaulicht.
• 3 ein beispielhaftes Szenario mit drei Objekten zeigt.
• 4 ein Grundwissen-Objekthierarchiediagramm und ein Beispiel für ein geschätztes Objekthierarchiediagramm für das Szenario mit drei Objekten aus 3 zeigt.
• 5 ein Flussdiagramm zeigt, das ein Verfahren zur Steuerung einer Robotervorrichtung gemäß einer Ausführungsform veranschaulicht.

In the drawings, similar reference numerals generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, but are generally intended to illustrate the principles of the invention. In the following description various aspects are described with reference to the following drawings, in which:

• 1 shows a robot.
• 2 illustrates operation of the controller for a bin picking task.
• 3 shows an example scenario with three objects.
• 4 a basic knowledge object hierarchy diagram and an example estimated object hierarchy diagram for the three object scenario 3 shows.
• 5 shows a flowchart illustrating a method for controlling a robotic device according to an embodiment.

The following detailed description refers to the accompanying drawings which show, by way of illustration, specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be used and structural, logical and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

Various examples are described in more detail below.

1 shows a robot 100.

The robot 100 includes a robot arm 101, for example an industrial robot arm, for handling or assembling a workpiece (or one or more other objects). The robot arm 101 has manipulators 102, 103, 104 and a base (or holder) 105 through which the manipulators 102, 103, 104 are supported. The term “manipulator” refers to the movable elements of the robot arm 101, the actuation of which enables physical interaction with the environment, e.g. B. the execution of a task. For control, the robot 100 has a (robot) controller 106, which is designed to implement interaction with the environment according to a control program. The last element 104 (farthest from the holder 105) of the manipulators 102, 103, 104 is also referred to as the end effector 104 and has a gripping tool (which can also be a suction cup).

The other manipulators 102, 103 (located closer to the holder 105) can form a positioning device so that together with the end effector 104, the robot arm 101 is provided with the end effector 104 at its end. The robot arm 101 is a mechanical arm that can provide similar functions to a human arm.

The robot arm 101 can have joint elements 107, 108, 109 that connect the manipulators 102, 103, 104 to each other and to the holder 105. A joint element 107, 108, 109 can have one or more joints, which can each enable a rotary movement (i.e. rotational movement) and/or translational movement (i.e. displacement) relative to one another on assigned manipulators. The movement of the manipulators 102, 103, 104 can be initiated by actuators that are controlled by the controller 106.

The term “actuator” can be understood as a component that is capable of influencing a mechanism or process in response to a control. The actuator can convert instructions given by the controller 106 (the so-called activation) into mechanical movements. The actor, e.g. B. an electromechanical converter, can be designed to convert electrical energy into mechanical energy in response to a control.

The term “controller” may be understood as any type of logic implementation unit, which may include, for example, a circuit and/or a processor capable of executing software, firmware, or a combination thereof stored in a storage medium and capable of issuing instructions , e.g. B. to an actuator, in the present example. The controller may be configured, for example, through program code (e.g., software) to control the operation of a system, in the present example a robot.

In the present example, the controller 106 includes one or more processors 110 and a memory 111 that stores code and data based on which the processor 110 controls the robot arm 101. According to various embodiments, the controller 106 controls the robot arm 101 based on a machine learning model 112 (e.g., comprising one or more neural networks) stored in the memory 111.

The task of the robot is, for example, to carry out bin picking, that is, to grab an object from a plurality of objects 113 (grasping also includes picking up the object 113 with a suction cup) and to show the object 113, for example, to a scanner, or the object 113 to move to another container. In order to be able to determine the object 113 to be picked up and a suitable gripping point on the object 113, the controller 106 uses images of the robot work area in which the objects 113 are located. These images can be taken from a camera 114 attached to the robot arm 101 (or in some other way so that the controller can control the angle of view of the camera 114).

Detecting objects can be difficult when there is a stack of objects because in captured images, objects may be at least partially obscured and one object may cast a shadow on another object.

On the other hand, if there is a stack of objects 113, i.e. H. the objects are not isolated and therefore cannot be manipulated without affecting other objects, but may be stacked on top of each other, care must be taken to avoid the stack falling when a supporting object is removed (possibly causing damage to objects or people). can be injured). Also, in terms of efficiency of robot operation, it may be desirable to respect object hierarchy (i.e., the interdependence of objects in terms of the stacking order of objects), as it may be easier and require less force to stack an object from the top of a heap to grab rather than trying to pull an object out from underneath.

Therefore, in some applications, the controller 106 should consider the object hierarchy when deciding which object to include, e.g. B. if there is a stack of objects. This means that the robot arm 101 should not pick up an object that has another object on it that may fall and break when the robot arm 101 picks up the supporting object. The object hierarchy (information) here is intended to describe the relationships between objects 113 that are stacked on top of one another (i.e. have a stacking relationship), e.g. e.g., a first cup stacked on top of a second cup, etc. For applications that require the manipulation of objects in a dense mess, knowledge of the object hierarchy allows for efficient manipulation (and avoidance of damage to the objects).

However, inferring this relationship (i.e., object hierarchy) from a single RGB image is an inherently difficult task, as it may have to deal with multiple occluded objects in different lighting conditions. According to various embodiments, a machine learning model (e.g., corresponding to machine learning model 112) is provided in the form of a neural network called an adjacency network (Adj-Net) that detects all objects in a scene that are in a color (RGB) Image are shown, and the spatial relationship is inferred from the given color image. This actually improves object recognition, and thus can be beneficial even if the adjacency information (the object hierarchy, i.e. spatial relationship) itself is not used in further control.

Figure 2 illustrates operation of controller 106 for a bin picking task using such an adjacency network.

1. 1. Aufnehmen eines Bildes 201 der Szene (d. h. des Roboterarbeitsbereichs, der die Objekte 113 aufweist) mittels der Kamera 114.
2. 2. Durchleiten des Bildes 201 durch das Adjazenznetz 202 (welches insbesondere als Objektdetektor arbeitet) zur gemeinsame Vorherzusage von N Vorschlägen für Begrenzungsrahmen 212 (durch ein Begrenzungsrahmen-MLP (mehrschichtiges Perzeptron) 203 des neuronalen Netzes 202), von Klassenwerten 213 für die Begrenzungsrahmen (durch einen Klassifizierer-MLP 204 des neuronalen Netzes 202), und einer einzelnen Adjazenzmatrix 211 der Größe N x N (durch Adjazenz-MLPs 205 des neuronalen Netzes 202, deren Ausgänge durch eine Kombinationsoperation 206 kombiniert werden).
3. 3. Herausfiltern der „None“-Klassen aus der Ausgabe des Objektdetektors, um eine reduzierte Ausgabe von Begrenzungsrahmen 214, Klassen 215, und einer kleineren Adjazenzmatrix 216 zu erzeugen.
4. 4. Weiterleiten dieser Ausgaben an weitere Komponenten der Steuerung (z. B. Bestimmen der tatsächlichen Greifpose und/oder Auswählen des zu greifenden Objekts unter Verwendung dieser Ausgaben).
5. 5. Zurückkehren zu (1), d. h. Aufnehmen eines neues Bildes 201, falls erforderlich (z. B., um das nächste Objekt zu greifen).

The controller 106 performs the following:

1. 1. Capture an image 201 of the scene (ie, the robot workspace containing the objects 113) using the camera 114.
2. 2. Passing the image 201 through the adjacency network 202 (which in particular works as an object detector) for the joint prediction of N proposals for bounding boxes 212 (by a bounding box MLP (multi-layer perceptron) 203 of the neural network 202), class values 213 for the bounding boxes ( by a classifier MLP 204 of the neural network 202), and a single adjacency matrix 211 of size N x N (by adjacency MLPs 205 of the neural network 202, the outputs of which are combined by a combining operation 206).
3. 3. Filter out the "None" classes from the output of the object detector to produce a reduced output of bounding boxes 214, classes 215, and a smaller adjacency matrix 216.
4. 4. Forwarding these outputs to other components of the controller (e.g., determining the actual grasping pose and/or selecting the object to be grasped using these outputs).
5. 5. Return to (1), ie, capture a new image 201 if necessary (e.g., to grab the next object).

The images 201 are, for example, RGB-D images (i.e., color images with depth information). Other types of image data, e.g. B. Video, radar, LiDAR, ultrasound, motion, thermal images, or sonar can also be used (alternatively or additionally). These can be captured by appropriate image sensors (cameras).

• • Die sicherste Reihenfolge beim Greifen von Objekten, z. B. für Ansammlungen zerbrechlicher Objekte.
• • Das beste Objekt zum Schieben, um die Unordnung für ein effizientes Greifen zu verteilen, z. B. für objektspezifische Greifaufgaben.
• • Der beste nächste Blickwinkel für das neue Bild zur Verbesserung der probabilistischen Darstellung der Szene.

For example, the additional control components 210 (e.g., an action selector) may use the adjacency matrix 209 (generally object hierarchy information about the objects) to determine one or more of the following:

• • The safest order when grasping objects, e.g. B. for accumulations of fragile objects.
• • Best object to push to distribute the mess for efficient grasping e.g. B. for object-specific gripping tasks.
• • The best closest viewing angle for the new image to improve the probabilistic representation of the scene.

The controller can then control the robot accordingly.

The machine learning model 112 is trained by a machine learning (ML) method (either by the controller 106 or by another device from which it is then stored in the controller 106) to generate an adjacency matrix output 211 from an image input, which is a probabilistic represents object hierarchy graphs.

3 shows an example scenario with three objects, a mouse (referred to as object a), a roll of tape (referred to as object b), and a box (referred to as object c).

ist eine Menge von Knoten V, die mit gerichteten Kanten

verbunden sind. Wenn die Objekthierarchie feststeht, reicht diese Darstellung aus, um die Beziehung zwischen den Objekten zu erfassen. Jeder Knoten stellt ein Objekt dar, und jede Kante stellt die Stapelbeziehung zwischen verbundenen Knoten dar. Ist beispielsweise ein Objekt b über einem Objekt a platziert, wird es daher als v_a und v_b mit einer Kante ε_ba dargestellt, die sich von b nach a erstreckt, wobei v ∈ V und ε ∈

Wenn die Hierarchie jedoch unbekannt ist und aus unvollständigen Daten abgeleitet wird, z. B. aus Bildern einer Szene, ist diese Darstellung unzureichend und aufgrund von Unsicherheiten bei der Inferenz fehleranfällig. Daher wird gemäß verschiedenen Ausführungsformen die Objekthierarchie als ein gewichteter gerichteter Graph $\mathcal{G}\triangleq \left\{V,\mathcal{E},\mathcal{W}\right\}$

dargestellt, wobei jeder Kante ε ∈ V ein Gewicht ω_ε ∈ W zugeordnet ist. Das Gewicht ist die Wahrscheinlichkeit, dass die Kante bei einem gegebenen Bild I existiert, also die Wahrscheinlichkeit, dass sich die entsprechenden Objekte in Richtung der Kante stapeln: $${\omega }_{\in }\triangleq \mathbb{P}\left(\in |I\right)$$

The object hierarchy can be represented as a directed graph. A directed graph $\mathcal{G}\triangleq \left\{v,\mathcal{E}\right\}$

is a set of vertices V with directed edges

are connected. If the object hierarchy is fixed, this representation is sufficient to capture the relationship between the objects. Each node represents an object, and each edge represents the stacking relationship between connected nodes. For example, if an object b is placed above an object a, it is therefore represented as v _a and v _b with an edge ε _ba extending from b to a extends, where v ∈ V and ε ∈

However, if the hierarchy is unknown and derived from incomplete data, e.g. B. from images of a scene, this representation is inadequate and prone to errors due to uncertainties in inference. Therefore, according to various embodiments, the object hierarchy is represented as a weighted directed graph $\mathcal{G}\triangleq \left\{v,\mathcal{E},\mathcal{W}\right\}$

shown, where each edge ε ∈ V is assigned a weight ω _ε ∈ W. The weight is the probability that the edge exists for a given image I, i.e. the probability that the corresponding objects stack in the direction of the edge: $${\omega }_{\in }\triangleq \mathbb{P}\left(\in |I\right)$$

It is understood that as a probability of a carrier the A _i,j 's is between 0 and 1. In addition, the diagonal elements are all equal to 0.

4 shows a basic knowledge object hierarchy graph 401 and an example of an estimated object hierarchy graph (with probabilistic, ie, confidence information) 402 for the three-object scenario of 3 . The basic knowledge graph 401 specifies that object a lies directly above both objects b and c, and the estimated object hierarchy graph 402 gives an estimate with probabilistic edge weights that express the confidence of the respective relationships between the objects.

in Form einer Adjazenzmatrix dargestellt, die als A

bezeichnet wird, oder einfach als A, um die Notationen abzukürzen. Die Adjazenzmatrix ist eine quadratische Matrix mit den Dimensionen N_V × N_V, wobei N_V die Anzahl der Knoten, d. h. der in der Szene erkannten Objekte ist. Jedes Element A_i,j entspricht der Kante der Objekte i bis j: $$A_{i,j}={\omega }_{\in }^{ij}\triangleq \mathbb{P}\left({\in }_{ij}|I\right)$$

According to various embodiments, an object hierarchy graph

represented in the form of an adjacency matrix, denoted as A

or simply A to abbreviate the notations. The adjacency matrix is a square matrix with dimensions N _V × N _V , where N _V is the number of nodes, that is, objects detected in the scene. Each element A _i,j corresponds to the edge of objects i to j: $$A_{i,j}={\omega }_{\in }^{ij}\triangleq \mathbb{P}\left({\in }_{ij}|I\right)$$

wohingegen die geschätzte Adjazenzmatrix, die den Graphen der geschätzten Objekthierarchie darstellt, ist: $$A=\left[\begin{array}{ccc}0& 0.9& 0.6\\ 0.2& 0& 0.7\\ 0& 0& 0\end{array}\right].$$

In the example of 4 is the basic knowledge adjacency matrix corresponding to the basic knowledge object hierarchy graph $$A=\left[\begin{array}{ccc}0& 1& 1\\ 0& 0& 1\\ 0& 0& 0\end{array}\right],$$

whereas the estimated adjacency matrix representing the graph of the estimated object hierarchy is: $$A=\left[\begin{array}{ccc}0& 0.9& 0.6\\ 0.2& 0& 0.7\\ 0& 0& 0\end{array}\right].$$

The network architecture of the adjacency network Adj-Net 202 is described in more detail below. According to various embodiments, the adjacency network 202 uses as a backbone a transformer-encoder-decoder (object detector) network (such as described in Reference 1) formed by an encoder 207 and a decoder 208. The encoder 207 may be preceded by a visual feature extractor 217 (e.g., a convolutional neural network such as ResNet) that receives the current image 201 as input. The decoder 208 is provided with queries 218 that are visible to specify areas of the input image 201 where the adjacency network 202 looks for objects. For example, the number of queries is a predefined hyperparameter, but the parameters of the Queries (ie, image locations) may be learned during training of the adjacency network 202.

The adjacency network 202 has an adjacency head (implemented by a projection operation 209, adjacency MLPs 205, and the combination operation 206) which receives from the encoder 207 a matrix of feature vector size E × H, where E is the number of encoder feature vectors, and H is the dimension of a respective feature vector.

Object detection is also an important component of the Adj-Net 202, so it has both classification and bounding box prediction heads (implemented by the classification MLP 204 and the bounding box MLP 203, respectively. The decoder 208 outputs a matrix of feature vectors of size Q × H, where Q is a preset number of possible bounding box candidates. Each feature vector passes through the bounding box MLP 203 and produces a set of predicted bounding box coordinates as a matrix size Q × 4. The classifier MLP 204 receives a concatenated feature vector of both the decoder output and the Bounding box coordinates and outputs a Q probability of vectors 213 for each corresponding bounding box of length C+1, where C corresponds to a respective candidate class, and the additional term corresponds to an "empty" class intended to filter out bounding box candidates that either mark areas without objects or are an overlap with previous object detections.

To predict the adjacency matrix 211, the encoder output is passed through the projection block, which serves a dual purpose; first, it projects the dimension of the encoder output onto a feature matrix size Q × H to match the bounding box proposals; second, it thinks about the connections between encoder feature relationships and tries to connect them to the hierarchy between objects.

The feature matrix is then concatenated with the bounding box predictions 212 and split into two channels. Each channel is passed through a neural network-based adjacency block (i.e., adjacency MLP 205) that maintains dimensions, and then the outputs from each channel are multiplied and passed through a sigmoid layer (i.e., processed by the combining combination 206) to create a preliminary Q × Q adjacency matrix 211.

Each row and column of the neighborhood in the adjacency matrix 211 corresponds to a bounding box proposal, and the classifier filters the bounding box proposals over the “empty” class, i.e. H. Proposals for which the classifier predicts an “empty” class are discarded.

$$A_p=\text{Sigmoid}\left(\frac{{ƒ}_1{\left({\chi }_e^{\text{'}}\right)}^T\cdot {ƒ}_2\left({\chi }_e^{\text{'}}\right)}{H}\right)$$

wobei A_p die vorläufige Adjazenzmatrix 211 ist.Let x _e denote the output of the encoder 207, x _d the bounding box proposal coordinates 212, χ' _e the projected encoder output concatenated with x _d , and the functions g(·), ƒ ₁ (·) and ƒ ₂ (·) describe the Projection block 209 or both neighboring blocks 205. The adjacency head operation can then be described as follows: $${\chi }_e^{\text{'}}=G\left(x_e^T\right)\cup x_b$$

$$A_p=\text{Sigmoid}\left(\frac{{ƒ}_1{\left({\chi }_e^{\text{'}}\right)}^T\cdot {ƒ}_2\left({\chi }_e^{\text{'}}\right)}{H}\right)$$

where A _p is the preliminary adjacency matrix 211.

For example, the dimension of the hidden state is H = 256, where the encoder outputs E = 17821 feature vectors, and Q = 300 bounding box proposals are considered. The size of each class probability vector depends on the situation of the problem. For example, for testing on the Visual Manipulation Relationship Dataset (VMRD), C+1 = 32. Binary classification (i.e. object or no object, C+1 = 2) can be applied to many real-life problems.

As described above, multi-layer perceptrons (MLPs) are used, for example, for the projection block 209, the bounding box, the classification and both channels of the adjacency head.

The Adj-Net 202 can be trained in a supervised manner from a training set consisting of RGB image inputs and labels consisting of the basic knowledge classes, bounding boxes and spatial relationships. Two examples of such datasets available online are VMRD and REGRAD (Relational Grasp Dataset). Given a labeled image, Adj-Net 202 produces raw bounding boxes, classification scores, and an adjacency matrix.

To calculate a loss, the derived bounding boxes are matched to the “nearest” target bounding boxes and classes. For example, the Hungarian algorithm can be used to match the bounding boxes. In addition, the rele vant rows and columns in A _p and a matched adjacency matrix A _m is formed that has the entries corresponding to the matched bounding boxes.

For example, there are many (e.g. 300) proposed bounding boxes, so (in many use cases) most of them do not correspond to any object. The adjacency matrix A _p describes the relationships between all of these 300 objects. The classification header determines whether the proposal should be rejected (via the “no object” class) or not. From the non-discarded bounding boxes, e.g. For example, if there are 5 objects, A _m is the 5x5 adjacency matrix that describes the relationship between these objects. For example, in training, using the Hungarian matching algorithm as a matcher, the matcher compares the “closest” ones from the 300 proposals with the target adjacency matrix. Assuming that the baseline is a 4x4 adjacency matrix, the matcher finds the 4 proposals that are closest to the targets via an internal cost function, and training is performed on these matched proposals. This comparison is not required for inference (i.e. after training).

und die Klasse c beträgt der gewichtete Kreuzentropieverlust: $$L_c=-{\displaystyle \sum _{p\in \mathcal{P}}{\displaystyle \sum _{c=1}^{C+1}{w}_c\cdot {\mathbb{P}}_{gt}}}\left(c_p|I\right)\text{log}\mathbb{P}\left(c_p|I\right)$$

The classifier is trained via weighted cross-entropy loss. For an image I with proposals ρ ∈

and class c is the weighted cross-entropy loss: $$L_c=-{\displaystyle \sum _{p\in \mathcal{P}}{\displaystyle \sum _{c=1}^{C+1}{w}_c\cdot {\mathbb{P}}_{Gt}}}\left(c_p|I\right)\text{log}\mathbb{P}\left(c_p|I\right)$$

und dem von generalisiertem Überlappungs- (GloU) Verlust $L_{GIoU}$

verwendet.Since the number of detections for which the basic knowledge class is "unknown" is generally significantly larger than the number of objects in the scene, the class "unknown" is weighted with a small w _C+1 << 1, while for other classes w _c = 1. For the matched bounding boxes, a linear combination of the 1 ₁ (absolute deviation) loss is used ${\mathrm{<figure-callout\; id="1"\; label="L\; l"\; filenames="DE102022202321A1\_0023.png"\; state="\{\{state\}\}">L</figure-callout>}}_{l_{}}$

and that of generalized overlap (GloU) loss $L_{GIOU}$

used.

wobei A_ij das (i, j)-te Element von A_m ist und $A_{ij}^{gt}$

das Grundwissen dieses Elements ist. Dies führt potenziell zu einem langsameren Lernen, vermeidet jedoch das lokale Minimum einer Adjazenzmatrix aus Nullen.Since the adjacency matrix A _m (and by extension A _p ) is generally a sparse matrix with many 0s and few 1s, there is a risk that the adjacency network 202 will converge to the appropriate local minimum of a matrix of zeros during learning . This is a significant problem when using element-wise l ₁ and l ₂ losses, as they do not significantly penalize a false 0. This problem is exacerbated when trying to learn the entire _Ap , which is padded with zeros or ones for the rows and columns that do not match. Thus, according to various embodiments, a binary element-wise cross-entropy loss over Am is used according to $$L_{adj}=-{\displaystyle \sum _{i,j\in A_m}\left(A_{ij}^{Gt}{\text{logA}}_{ij}+\left(1-A_{ij}^{Gt}\right)\text{log}\left(1-A_{ij}\right)\right),}$$

where A _ij is the (i, j)th element of A _m and $A_{ij}^{Gt}$

is the basic knowledge of this element. This potentially results in slower learning, but avoids the local minimum of an adjacency matrix of zeros.

wobei die γ allesamt Hyperparameter sind.The overall loss function is a weighted sum of all individual loss functions: $$L_{tOtal}={\gamma }_c{L}_c+{\gamma }_{l_1}{L}_{l_1}+{\gamma }_{GIOU}{L}_{GIOU}+{\gamma }_{adj}{L}_{adj},$$

where the γ are all hyperparameters.

The adjacency network 202, for example, is trained in two phases.

vortrainiert, indem γ_adj = 0 gesetzt wird. Beispielsweise wird die Superkonvergenz-Lernraten-Planungs-Technik mit einer maximalen Lernrate von 0,01 in Verbindung mit dem AdmaW-Optimierer mit einem Gewichtsabfall von 1×10-6 verwendet. Die Superkonvergenztechnik ermöglicht ein schnelleres Training und dient als Netzregularisierungstechnik für eine bessere Verallgemeinerung.In the first phase, only the detector (classification head and bounding box head) is used by omitting $L_{adj}$

pre-trained by setting γ _adj = 0. For example, the superconvergence learning rate scheduling technique with a maximum learning rate of 0.01 is used in conjunction with the AdmaW optimizer with a weight decay of 1×10-6. The superconvergence technique enables faster training and serves as a network regularization technique for better generalization.

wieder in die Verlustfunktion eingeführt, indem γ_adj auf einen Wert größer als Null gesetzt wird, und das Training wird fortgesetzt, z. B. über die Superkonvergenz und einem AdamW mit einer maximalen Lernrate von 0,001.In the second phase, $L_{adj}$

reintroduced into the loss function by setting γ _adj to a value greater than zero and training continues, e.g. B. via superconvergence and an AdamW with a maximum learning rate of 0.001.

In summary, according to various embodiments, a method for controlling a robotic device is provided, as in 6 is illustrated.

5 shows a flowchart 500 illustrating a method for controlling a robot device according to an embodiment.

• • Merkmalsvektoren aus dem Trainingsbild durch einen Encoder eines Transformator-Encoder-Decoder-Netzes bestimmt, in 502,
• • für jeden einer Mehrzahl von Objektbegrenzungsrahmen Begrenzungsrahmenkoordinaten aus den Merkmalsvektoren durch einen Decoder des Encoder-Decoder-Netzes und einen Begrenzungsrahmenkopf bestimmt, in 503;
• • Objekthierarchieinformationen aus den Begrenzungsrahmenkoordinaten bestimmt für die Mehrzahl von Objekten und Merkmalen für die Begrenzungsrahmen, die von den Merkmalsvektoren abgeleitet werden, in 504, wobei die Objekthierarchieinformationen Stapelbeziehungen der Mehrzahl von Objekten in Bezug zueinander spezifizieren;
• • Klassifizierungsinformationen über Objekte, die den Objektbegrenzungsrahmen entsprechen, durch einen Klassifizierungskopf bestimmt, in 505.

In 501, for each image of a plurality of training images,

• • Feature vectors from the training image determined by an encoder of a transformer-encoder-decoder network, in 502,
• • for each of a plurality of object bounding boxes, bounding box coordinates from the feature vectors determined by a decoder of the encoder-decoder network and a bounding box header, in 503;
• • Object hierarchy information from the bounding box coordinates determined for the plurality of objects and features for the bounding boxes derived from the feature vectors in 504, the object hierarchy information specifying stacking relationships of the plurality of objects with respect to one another;
• • Classification information about objects corresponding to the object bounding boxes, determined by a classification header, in 505.

In 506, the neural network is trained to reduce a loss between basic knowledge for the hierarchy information, for the bounding box coordinates and for the classification information and the hierarchy information, where the bounding box coordinates and the classification information have been determined for the training images, respectively.

This means that the neural network is trained with the sum of a loss between basic knowledge for the hierarchy information and the derived hierarchy information, a loss between basic knowledge for the bounding box coordinates and the derived bounding box coordinates, and a loss between basic knowledge for the classification information and the derived classification information.

While in the above embodiments the approach of 5 is applied to the control of a robot arm, it can be applied to calculate a control signal for controlling any engineering system in a scenario where object hierarchy plays a role, such as: B. a computer-controlled machine, such as a robot, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system.

According to one embodiment, the method is computer implemented.

Although specific embodiments have been shown and described herein, those of ordinary skill in the art will recognize that a variety of alternative and/or equivalent implementations may replace the specific embodiments shown and described without departing from the scope of the present invention. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, this invention is intended to be limited only by the claims and their equivalents.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Non-patent literature cited

• N. Carion et al., “End-to-end object detection with transformers,” in the European Conference on Computer Vision, Springer, 2020, pages 213-229 [0002]

Claims (8)

Method for training a neural network to recognize objects in image data, comprising: for each image of a plurality of training images determining feature vectors from the training image by an encoder of a transformer-encoder-decoder network; determining, for each of a plurality of object bounding boxes, bounding box coordinates from the feature vectors by a decoder of the encoder-decoder network and a bounding box head; determining object hierarchy information from the bounding box coordinates for the plurality of objects and features for the bounding boxes derived from the feature vectors, the object hierarchy information specifying stacking relationships of the plurality of objects with respect to one another; determining classification information about objects corresponding to the object bounding boxes through a classification head; and Training the neural network to reduce a loss between ground truths for the hierarchy information, for the bounding box coordinates and for the classification information and the hierarchy information, where the bounding box coordinates and the classification information were determined for the training images, respectively. Procedure according to Claim 1 , comprising transforming the feature vectors into a feature vector for each object bounding box, concatenating, for each object bounding box, the feature vector for the bounding boxes with the bounding box coordinates for the object bounding boxes to an input vector, and determining the object hierarchy information from the input vectors. Procedure according to Claim 2 , comprising determining the object hierarchy information from the input vectors by a plurality of processing channels, each processing channel receiving the input vectors for the object bounding boxes and generating an adjacency matrix component, and multiplying the adjacency matrix components generated by the channels to generate an adjacency matrix specifying the object hierarchy information. Procedure according to one of the Claims 1 until 3 , where the neural network is trained using a combined loss of a cross-entropy loss for the classification information, a binary element-wise loss for the object hierarchy information, and a linear combination of absolute deviation loss and generalized overlap loss for the bounding box coordinates. Method for controlling a robotic device, comprising: training a machine learning model according to one of Claims 1 until 4 ; Capturing a camera image showing the one or more objects; Feeding the camera image into the neural network to recognize the one or more objects; and controlling the robot taking into account the detected one or more objects. Control that is designed to implement a method according to one of the Claims 1 until 5 to carry out. Computer program that includes instructions that, when executed by a computer, cause the computer to perform a method according to one of the Claims 1 until 5 carries out. A computer-readable medium containing instructions that, when executed by a computer, cause the computer to perform a method according to one of the Claims 1 until 5 carries out.

Images