DE102021202570A1 - DEVICE AND METHOD FOR LOCATING LOCATIONS OF OBJECTS FROM CAMERA IMAGES OF THE OBJECTS - Google Patents

Abstract

According to various embodiments, a method for locating locations of objects from camera images of the objects is described, comprising, in which a machine learning model for mapping camera images to descriptor images is trained, for each camera image, the locations to be localized on the respective object by means of the descriptors provided by the trained machine learning model are determined and the descriptors used to determine the locations to be located are updated over the course of the sequence of camera images.

Description

The present disclosure relates to devices and methods for locating locations of objects from camera images of the objects.

In order to enable flexible production or processing of objects by a robot, it is desirable for the robot to be able to handle an object regardless of the posture with which the object is placed in the working space of the robot. Therefore, the robot should be able to recognize which parts of the object are in which positions, so that it can, for example, grab the object in the correct place to e.g. B. to attach to another object, or to weld the object at the current location. This means that the robot should be able to recognize the pose (position and orientation) of the object, or even regions of the object like a barcode, from one or more images captured by a camera attached to the robot. One approach to achieve this is to use descriptors, i. H. To determine points (vectors) in a predefined descriptor space, for parts of the object (i.e. pixels of the object represented in a camera image plane), the robot being trained to assign the same descriptors to the same parts of an object, regardless of a current pose of the object, and thus to recognize the topology of the object in the image, so that it is then known, for example, where each corner of the object is located in the image. If the pose of the camera is known, the pose of the object or the position of areas of the object in three-dimensional space can then in turn be inferred. The recognition of the topology can be realized with a machine learning model that is trained accordingly. However, difficulties also arise with this approach, because it can happen, for example, that the machine learning model does not always assign a descriptor to the same part of the object due to different lighting conditions (especially reflections) or insufficient training in different camera images. This can then lead to inaccuracies in the determination of poses or generally in the determination of positions of the object. Accordingly, approaches that can be used to achieve greater accuracy, e.g. when determining poses, are desirable.

• Abbilden des Kamerabilds auf ein Deskriptorbild mittels des trainierten maschinellen Lernmodells
• Identifizieren der zu lokalisierenden Stellen des Objekts für den Referenzsatz von Deskriptoren durch Suchen der Deskriptoren des Referenzsatzes von Deskriptoren in dem Deskriptorbild; und

zumindest für einen Teil der Kamerabilder der Folge von Kamerabildern aufweisend:

• Ermitteln der relativen Lage der lokalisierten Stellen für den Referenzsatz; und Vergleich der Referenz für die relativen Lage der zu lokalisierenden Stellen und der für den Referenzsatz ermittelten relativen Lage der lokalisierten Stellen, Identifizieren der zu lokalisierenden Stellen des Objekts für einen Testsatz von Deskriptoren durch Suchen der Deskriptoren des Testsatzes von Deskriptoren in dem Deskriptorbild,
• Ermitteln der relativen Lage der lokalisierten Stellen für den Testsatz; und Vergleich der Referenz für die relativen Lage der zu lokalisierenden Stellen und der für den Testsatz ermittelten relativen Lage der lokalisierten Stellen und Aktualisieren der Deskriptoren des Referenzsatzes auf die Deskriptoren des Testsatzes falls die Übereinstimmung für ein oder mehrere der Kamerabilder zwischen der Referenz für die relativen Lage der zu lokalisierenden Stellen und der ermittelten relativen Lage der lokalisierten Stellen für den Testsatz besser ist als für den Referenzsatz.

According to various embodiments, a method for locating positions of objects from camera images of the objects is provided, comprising: determining the positions to be localized for an object type of the objects, determining a reference for the relative position of the positions to be localized, training for the object type, one machine learning model for mapping camera images, each camera image showing an object of the object type, onto descriptor images, wherein a descriptor image, onto which a camera image is to be mapped, for a point of the object which shows the camera image at an image position, at the image position a descriptor of the location of the object, setting a reference set of descriptors to an initial set of descriptors, receiving a temporal sequence of camera images, each camera image showing an object of the object type, and locating, for each camera image, the locations to be localized on the respective object by:

• Mapping the camera image to a descriptor image using the trained machine learning model
• identifying the locations of the object to be located for the reference set of descriptors by searching the descriptors of the reference set of descriptors in the descriptor image; and

having at least for a part of the camera images of the sequence of camera images:

• determining the relative location of the located locations for the reference set; and comparing the reference for the relative position of the locations to be located and the relative location of the located locations determined for the reference set, identifying the locations of the object to be located for a test set of descriptors by searching the descriptors of the test set of descriptors in the descriptor image,
• determining the relative location of the located locations for the test set; and comparing the reference for the relative locations of the locations to be located and the relative locations of the located locations determined for the test set and updating the descriptors of the reference set to the descriptors of the test set if the match for one or more of the camera images between the reference for the relative locations of the locations to be located and the determined relative position of the located locations for the test set is better than for the reference set.

The method described above allows for cases where multiple descriptors are used (e.g. tracked), such as for pose detection or region detection (or tracking), a more accurate localization of locations of the object, since the descriptors used over the sequence of Camera images are improved in the sense that they better fulfill their theoretical property, namely that they are always assigned to the same positions in a view-invariant manner. In particular, the method described above increases the robustness to variations in lighting conditions etc. in a sequence of camera images.

In this way, the method described above enables, for example, the safe picking up (e.g. gripping) of an object for any position of the object or the exact determination of certain areas of the object such as the location of a barcode.

Various exemplary embodiments are specified below.

Embodiment 1 is the method described above for locating locations of objects from camera images of the objects.

Exemplary embodiment 2 is the method according to exemplary embodiment 1, with the relative position having the paired distances of the points to be localized or of the localized points in three-dimensional space.

This evaluates the extent to which the localized points are correctly localized in three-dimensional space. In particular, the use of distances in three-dimensional space (instead of just distances in the camera image plane) ensures that detection errors that are only reflected in a deviation perpendicular to the camera image plane are taken into account.

Embodiment 3 is the method according to embodiment 1 or 2, wherein the locations to be localized for the object type are specified on a reference camera image of an object of the object type, the reference camera image is fed to the machine learning model and the reference set of descriptors is based on the descriptors of the to locating locations in the descriptor image output by the machine learning model for the reference camera image.

In this way, the locations to be localized and a good initial value for the reference set of descriptors can easily be determined, since this reference set corresponds to the locations to be located, at least for the reference camera image. Over the course of the sequence of camera images, the reference set used to locate the locations to be located can then be improved.

Embodiment 4 is the method according to any one of embodiments 1 to 3, wherein the test set is chosen within a restricted range of the initial set of descriptors.

For example, the restricted range allows only some relative deviation of the descriptors of the test set from those of the initial set. This can prevent descriptors that deviate too much from the initial set and are therefore suitable for a number of camera images with a low probability from being (unnecessarily) tested or even being included in the reference set due to one (or fewer) camera images. In particular, an unstable behavior of the descriptor adaptation process can be avoided.

Exemplary embodiment 5 is the method according to any one of exemplary embodiments 1 to 4, comprising determining the test set of descriptors using a covariance matrix adaptation evolution strategy method.

This enables the efficient optimization of a function that is not in closed form, in this case the mapping of descriptors to the accuracy achieved in determining the locations of the object, in the case of a large number of evaluation points, as is the case over the course of the camera images.

• Lokalisieren von Stellen eines durch den Roboter zu behandelnden Objekts nach einem der Ausführungsbeispiele 1 bis 5,
• Ermitteln einer Pose des Objekts aus den lokalisierten Stellen und Steuern des Roboters in Abhängigkeit von der ermittelten Pose
• und/oder
• Ermitteln eines (z.B. abzutastenden oder anderweitig für die Verarbeitung bzw. Steuerung relevanten) Bereichs des Objekts aus den lokalisierten Stellen und Steuern des Roboters in Abhängigkeit des ermittelten Bereichs.

Embodiment 6 is the method for controlling a robot, including:

• Locating locations of an object to be treated by the robot according to any one of embodiments 1 to 5,
• Determining a pose of the object from the localized points and controlling the robot depending on the determined pose
• and or
• Determining an area of the object (eg to be scanned or otherwise relevant for the processing or control) from the localized points and controlling the robot depending on the area determined.

• eine Kamera, die zum Bereitstellen von Kamerabildern von Objekten eingerichtet ist;
• eine Steuereinrichtung, die zum Durchführen des Verfahrens nach einem der Ausführungsbeispiele 1 bis 6 eingerichtet ist.

Embodiment 7 is a software or hardware agent, particularly robot, comprising:

• a camera configured to provide camera images of objects;
• a control device that is set up to carry out the method according to one of the exemplary embodiments 1 to 6.

Embodiment 8 is the software or hardware agent of embodiment 7 having at least one actor, wherein the controller is configured to control the at least one actor using the located locations.

Embodiment 9 is a computer program that includes instructions that, when executed by a processor, cause the processor to perform a method according to any one of Embodiments 1-6.

Embodiment 10 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of Embodiments 1-6.

• 1 zeigt einen Roboter.
• 2 veranschaulicht das Trainieren eines neuronalen Netzes gemäß einer Ausführungsform.
• 3 veranschaulicht die Ermittlung einer Objektpose bzw. Greifpose gemäß einer Ausführungsform.
• 4 veranschaulicht das Anpassen von Deskriptoren, die zum Lokalisieren von Stellen von Objekten verwendet werden, im Lauf einer Folge von Kamerabildern.
• 5 zeigt ein Ablaufdiagramm für ein Verfahren zum Lokalisieren von Stellen von Objekten aus Kamerabildern der Objekte.

In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings:

• 1 shows a robot.
• 2 12 illustrates training a neural network according to one embodiment.
• 3 illustrates the determination of an object pose or gripping pose according to an embodiment.
• 4 Figure 12 illustrates the adjustment of descriptors used to locate locations of objects over a sequence of camera images.
• 5 shows a flowchart for a method for locating locations of objects from camera images of the objects.

The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure can be combined with one or more other aspects of this disclosure to form new aspects.

Various examples are described in more detail below.

1 shows a robot 100.

The robot 100 includes a robotic arm 101, for example an industrial robotic arm, for manipulating or assembling a work piece (or other object(s)). The robot arm 101 includes manipulators 102, 103, 104 and a base (or support) 105 by which the manipulators 102, 103, 104 are supported. The term "manipulator" refers to the movable components of the robotic arm 101, the operation of which enables physical interaction with the environment, e.g. B. to perform a task. For control, the robot 100 includes a (robot) controller 106, which is designed to implement the interaction with the environment according to a control program. The final component 104 (farthest from the base 105) of the manipulators 102, 103, 104 is also referred to as the end effector 104 and may include one or more tools, such as a welding torch, gripping instrument, paint gun, or the like.

The other manipulators 102, 103 (which are closer to the base 105) can form a positioning device so that, together with the end effector 104, the robotic arm 101 is provided with the end effector 104 at its end. The robotic arm 101 is a mechanical arm that can provide functions similar to a human arm (possibly with a tool at its end).

The robotic arm 101 may include articulation elements 107, 108, 109 that connect the manipulators 102, 103, 104 to each other and to the base 105. An articulation element 107, 108, 109 may comprise one or more articulations, each of which can provide rotational movement (i.e. rotational movement) and/or translational movement (i.e. translation) for associated manipulators relative to one another. The movement of the manipulators 102, 103, 104 can be initiated by means of actuators that are controlled by the control device 106.

The term "actuator" can be understood as a component configured to effect a mechanism or process in response to its impetus. The actuator can implement instructions (the so-called activation) created by the controller 106 into mechanical movements. The actor, e.g. An electromechanical converter, for example, may be configured to convert electrical energy into mechanical energy in response to being driven.

The term "controller" may be understood as any type of logic implementing entity, which may include, for example, circuitry and/or a processor capable of executing software, firmware, or a combination thereof stored on a storage medium, and the instructions, e.g. B. to an actuator in this example. For example, the controller may be configured by program code (e.g., software) to control operation of a system, a robot in the present example.

In the present example, the controller 106 includes one or more processors 110 and a memory 111 storing code and data based on which the processor 110 controls the robotic arm 101 . According to various embodiments, the controller 106 controls the robotic arm 101 based on a machine learning model 112 stored in the memory 111 .

According to various embodiments, the machine learning model 112 is designed and trained to enable the robot 100 to recognize from camera images a pickup pose of an object 113 that is placed, for example, in a workspace of the robot arm 101, for example for a robot that picks up objects to pick up a crate ("bin-picking"). This means that the robot 100 recognizes how it can pick up the object 113, i.e. how it has to orient its end effector 104 and where to move it in order to pick up the object 113 (e.g. grab it). The pick-up pose is understood to contain sufficient information for picking, i.e. information about the orientation and position 113 of the object, sufficient to determine from it how the object 113 can be gripped. The recording pose does not necessarily have to contain the complete orientation information about the object 113 since, for example, in the case of an object 113 with a rotationally symmetrical part for gripping, it may be irrelevant how the rotationally symmetrical part is rotated about its axis of rotation.

For example, the robot 100 may be equipped with one or more cameras 114 that enable it to capture images of its workspace. For example, a camera 114 is attached to the robot arm 101 so that the robot can take pictures of the object 113 from different perspectives by moving the robot arm 101 around.

An example of a machine learning model 112 for object recognition is a dense object mesh. A dense object mesh maps an image (e.g., an RGB image provided by the camera 114) onto a descriptor space image of some dimension D chosen. However, other machine learning models 112 can also be used, in particular those that do not necessarily generate a "dense" feature map, but merely assign descriptor values to certain points (e.g. corners) of the object.

According to various embodiments, an approach for recognizing an object and its pose is used, assuming that a 3D model (e.g., a CAD (Computer Aided Design) model) of the object is known, which is typically used for industrial assembly or processing tasks is the case. Non-linear dimensionality reduction techniques can be used to calculate optimal target images for training input images for a neural network. Thus, according to various embodiments, supervised neural network training is used. RGBD images of an object can also be recorded (RGB + depth information) and a 3D model for the object can be determined from them. Alternatively, self-supervised training can be performed, in which the machine learning model learns descriptors for locations of the object itself.

According to one embodiment for generating training data for training the machine learning model 112 for a supervised training, a data collection is first carried out. Specifically, for example, registered RGB (Red-Green-Blue) images are collected. A registered image here means an RGB image with known intrinsic and extrinsic camera values. For example, in a real-world scenario, a robot-mounted camera 114 (e.g., a robot wrist-mounted camera) is used to scan an object as the robot (e.g., robotic arm 101) moves around . In a simulated scenario, photorealistic generated RGB images are used using known object poses.

After collecting the RGB images, target images for the RGB images are rendered for supervised neural network training.

It is assumed that the pose of each object in world coordinates in each collected RGB image is known. This is straightforward for a simulated scenario, but requires manual tuning for a real-world scenario, e.g. B. Placing the object at predefined positions. RGBD (RGB plus depth information) images can also be used to determine the position of an object.

With this information and using a vertex descriptor computation technique, such as described below, for each RGB image (i.e., training input image), a descriptor image (i.e., training output image, also known as a target image or ground truth image) is rendered.

If a target image has been generated for each RGB image, i.e. pairs of RGB images and target images have been formed, these pairs of training input image and associated target image can be used as training data for training a neural network, as in 2 illustrated.

2 2 illustrates training a neural network 200 according to one embodiment.

The neural network 200 is a fully convolutional network that maps an h×w×3 tensor (input image) to an h×w×D tensor (output image).

It comprises several stages 204 of convolutional layers, followed by a pooling layer, upsampling layers 205 and skip connections 206 to combine the outputs of different layers.

For training, the neural network 200 receives a training input image 201 and outputs an output image 202 with pixel values in descriptor space (e.g. color components according to descriptor vector components). A training loss is calculated between the output image 202 and the target image 203 associated with the training input image. This can take place for a batch of training input images and the training loss can be averaged over the training input images and the weights of the neural network 200 trained using stochastic gradient descent using the training loss. For example, the training loss computed between the output image 202 and the target image 203 is an L2 loss function (to minimize a pixel-by-pixel least squares error between the target image 203 and the output image 202).

The training input image 201 shows an object and the target image and the output image contain vectors in descriptor space. The vectors in the descriptor space can be mapped to colors so that the output image 202 (as well as the target image 203) resembles a heat map of the object.

The vectors in the descriptor space (also called (dense) descriptors) are d-dimensional vectors (e.g. d is 1, 2 or 3) corresponding to each pixel in the respective image (e.g. each pixel of the input image 201, under assuming that the input image 201 and the output image 202 have the same dimension). The dense descriptors implicitly encode the surface topology of the object shown in the input image 201, invariant to its pose or camera position.

Given a 3D model of the object, it is possible to analytically determine an optimal and unique descriptor for each vertex of the 3D model of the object. According to various embodiments, target images are generated for registered RGB images using these optimal descriptors (or estimates of these descriptors determined by an optimization), resulting in fully supervised neural network 200 training. In addition, the descriptor space becomes explainable and optimal regardless of the chosen descriptor dimension d.

If the machine learning model 112, e.g. the neural network 200, has now been trained to map camera images of an object 113 onto descriptor images, the following procedure can be used to determine a recording pose of an object 113 in an unknown position.

First, several reference points p _i , i=1, . . . , N, are selected on the object 113 and descriptors of these reference points are determined. These reference points are the locations of the object to be located for a later ("new") camera image and the reference point descriptors is the (first, i.e. initial) reference set of descriptors (which can be adjusted over the course of a sequence of camera images as described below). The selection can be made by taking a camera image of the object 113, selecting reference pixels (u _i , v _i ) on the object (and thus corresponding reference points of the object) and mapping the camera image from the neural network 200 to a descriptor image. Then the descriptors at the positions in the descriptor image given by the positions of the reference pixels can be taken as descriptors of the reference points, ie the descriptors of the reference points are di = I ^d (u _{i , vi )} , where I ^d = f (I; θ) is the descriptor image, where f is the mapping (from camera image to descriptor image) implemented by the neural network, / is the camera image and θ is the weights of the machine learning model 200.

If the object 113 is now in an unknown position, a new camera image I _new is recorded and an associated descriptor image I ^d _new =f{I _new ; θ) determined. In this new descriptor image, descriptors are now sought which are as close as possible to the d _i descriptors of the reference images, for example by
(u _i ,v _i )* = argmin u _i ,v _i || I ^d _new (u _i ,v _i ) - d _i || ₂ ² for all i = 1, ..., N.

The positions of the reference points are derived from the thus determined or estimated positions (u _i , v _i )* of the reference points in the descriptor image I ^d _new (and thus correspondingly in the new camera image I _new ). determined in three-dimensional space. For example, a depth image is recorded together with the camera image I _new (or the camera image I _new has a depth channel, e.g. it is an RGBD image), so that from (u _i , v _i )* the three-dimensional position of the i-th reference point p _i can be determined (by projecting the depth value at the position (u _i ,v _i )* into the respective workspace coordinate system).

If the positions in space of several reference points are known, a recording pose can be determined from this, as is shown in 3 is shown.

For example, the position of two reference points p ₁ and p ₂ on the object 300 in space is determined and the two are linearly combined, e.g. their average is taken, in order to define an anchor point 304 . In order to define a gripping orientation, a first axis 301 is defined with the direction p ₁ and p ₂ and a second axis 302 is defined arbitrarily by the anchor point 304, e.g. in the z-axis direction of the camera 114 or in the direction of an axis of the work area coordinate system . A third axis 303 through the anchor point 304 can be calculated by the vector product of the direction vector of the first axis 301 and the direction vector of the second axis 302 . The three axes 301-303 and the anchor point 304 define a pick-up pose for the object 300. A robot can then be controlled to grasp the shaft of the object 300 extending in the direction of the first axis. The reference points p ₁ and p ₂ are defined, for example, such that they extend along the shaft, ie along an elongate part of the object suitable for gripping, as shown.

Similarly, three or more reference points can be arranged on a gripping surface of an object, so that a full 6D recording pose of the object or the orientation of a gripping surface on which an object can be picked up (gripped or sucked) can be determined from their position.

It should be noted that the gripper does not necessarily have to be in the form of pliers, but can also have a suction device, for example, in order to suck the object on a suitable surface and thus pick it up. In this case, for example, in order to bring the suction device into the correct position, it may be desirable to determine a pickup pose that indicates the orientation and position of a surface of the object that is suitable for suction. This can be done, for example, by determining an anchor point and a plane normal vector in the anchor point.

More than three reference points can also be used to determine a recording pose, e.g. to reduce errors through averaging.

Similar to a capture pose, controller 106 may also determine areas on object 300, e.g., a bounding box of a barcode provided on object 300, to identify the object. The control device 106 can then, for example, control the robot arm 101 in such a way that it holds the object 300 in front of a camera in such a way that it can read the barcode.

As explained above, a dense object network assigns a (eg RGB) camera image of an object or a part of it, recorded from any angle, to a descriptor image that assigns a multidimensional descriptor to each pixel (or each position) in the input camera image. The descriptors have the theoretical property that a given point on the object's surface is always associated with the same descriptor, regardless of the viewing angle. This property is useful for various applications, e.g. B. for the detection of areas on the surface of the target object by the detection of multiple descriptors, z. B. from vertices of the area or a recording pose as above with reference to 3 described. With additional depth information, ie RGBD input data instead of just RGB images (ie RGB images plus depth information), the detected points can be projected into 3D space to fully define such an area.

However, in practice, the mapping of points on the object to view-invariant descriptors is usually not perfect and the quality degrades due to factors such as small number of visual features, low dimensionality of the descriptor space, insufficient training data of some parts, ambiguity due to symmetric parts or similar objects, significantly different viewing angles or significantly different external conditions such as ambient lighting, invalid depth information due to reflective surfaces, etc. in the camera images showing the different views of an object.

According to various embodiments, the robustness to such factors is increased by not tracking initially selected descriptors statically, but optimizing them over the course of a sequence of camera images by modifying them (typically slightly) so that improved detection of locations on an object (and thus a pose or an area given by the positions, eg corner points a bounding box). In other words, during operation (“online”) the selection of the descriptors used is improved over time (ie over the course of a received chronological sequence of camera images).

The selection of the descriptors can be performed using a black box optimization method. Examples include Bayesian Optimization (BO) and CMA-ES (Covariance Matrix Adaptation Evolution Strategy). According to one embodiment, CMA-ES is used for a high number of samples due to its better performance, as this is typically a relevant aspect in this application.

To improve the descriptors over the course of a sequence of camera images, a set of descriptors (e.g. three descriptors for pose detection or four descriptors for defining a quadrilateral area) is first set as the first reference (i.e. initial value for the set of descriptors), for example by selecting one corresponding number of locations on an object in a camera image by a user, which are then mapped to descriptors by the machine learning model. This reference set of descriptors is adjusted over time (i.e. over the sequence of camera images). Concretely, it is replaced by a test set of descriptors if this provides higher accuracy. There is therefore a current reference set of descriptors for each camera image, beginning with the descriptor set selected as the first reference. According to one embodiment, several test sets of descriptors are evaluated in parallel, i.e. on the same image or the same subsequence of images, and the one that provides the highest accuracy is selected as the new reference set.

1. (A) Für das Kamerabild und den (aktuellen) Referenzsatz von Deskriptoren wird eine Ermittlungsgenauigkeit ermittelt, die angibt, wie gut die zu ermittelnden Stellen des Objekts ermittelt werden (z.B. wie gut ein Zielbereich ermittelt wird)
2. (B) Ein neuer Satz von Deskriptoren (Testsatz) wird zur Erhöhung der Ermittlungsgenauigkeit für zukünftige Kamerabilder der Folge vorgeschlagen.

The following is then carried out for each camera image of the sequence of camera images:

1. (A) A determination accuracy is determined for the camera image and the (current) reference set of descriptors, which indicates how well the locations of the object to be determined are determined (e.g. how well a target area is determined)
2. (B) A new set of descriptors (test set) is proposed to increase the detection accuracy for future camera images of the sequence.

• (A1) Für jeden Deskriptor wird die Position (Pixel) in dem Kamerabild ausgewählt, der durch das maschinelle Lernmodell auf den Deskriptor abgebildet wird
• (A2) Jede dieser Positionen in der Kamerabildebene wird durch Verwendung von Tiefeninformation (und z.B. intrinsische Kamerakalibrierungsparameter) auf die zugehörige 3D-Position abgebildet
• (A3) Ein Maß wird berechnet, das die relative Lage der 3D-Positionen erfasst, z.B. paarweise (z.B. euklidische) Abstände zwischen den 3D-Positionen.
• (A4) Das berechnete Maß wird mit einem Referenzmaß verglichen. Das Ergebnis des Vergleichs (z.B. ein Mittel über die Unterschiede der paarweisen Abstände zwischen den 3D-Positionen und Referenzabständen) gibt die Ermittlungsgenauigkeit an. Je geringer das Vergleichsergebnis ist, desto höher ist die Ermittlungsgenauigkeit. Beispielsweise wird der Wert der Ermittlungsgenauigkeit als der Kehrwert des Vergleichsergebnisses festgelegt. Das Referenzmaß bzw. Referenzabstände können beispielsweise in einem ersten Kamerabild ermittelt werden oder auch am echten Objekt gemessen werden. Beispielsweise wird gemessen, wie die Abstände der Eckpunkte eines Barcodes (oder anderer prominenter Stellen) am echten Objekt sind.

For (A) a quality measure of a set of descriptors is used for the determination accuracy. Instead of (or in addition to) evaluating descriptors individually (e.g. by calculating a confidence that a identified location is unique), the quality of a descriptor set is assessed based on the relative spatial location of the locations they identified on the object's surface (in three-dimensional space ) rated:

• (A1) For each descriptor, the position (pixel) in the camera image that is mapped to the descriptor by the machine learning model is selected
• (A2) Each of these positions in the camera image plane is mapped to the associated 3D position using depth information (and eg intrinsic camera calibration parameters).
• (A3) A metric is computed that captures the relative location of the 3D positions, eg, pairwise (eg, Euclidean) distances between the 3D positions.
• (A4) The calculated dimension is compared with a reference dimension. The result of the comparison (eg an average of the differences in the paired distances between the 3D positions and reference distances) indicates the determination accuracy. The lower the comparison result, the higher the determination accuracy. For example, the value of the determination accuracy is set as the reciprocal of the comparison result. The reference dimension or reference distances can be determined, for example, in a first camera image or can also be measured on the real object. For example, the distance between the corner points of a barcode (or other prominent points) on the real object is measured.

For (B) an optimization procedure is carried out for a (not known in closed form but evaluable by (A)) objective function, which receives a set of descriptors as input and outputs the accuracy measure for the camera image. (The accuracy measure can also be an average over the accuracy measures of several camera images.) A black-box optimization method can be used as follows to improve the reference set of descriptors over the course of the sequence of camera images (or to changing conditions, e.g. changed lighting). ) to adjust.
(B1) Choose one or more promising test sets of descriptors (e.g. as suggested by the black-box optimization method, e.g. according to an acquisition function)
(B2) Evaluate the test set and the reference set by determining the accuracy measure for the test set and the reference set (for one camera image or across multiple camera images)
(B3) If the measure of accuracy for the test set is better than for the reference set, make the test set the new reference set (ie update the previous reference set to the test set).

The optimization starts with the first reference for the descriptors as the reference set. Provision can be made for the descriptors not to be adjusted completely freely, but only within a limited range around the initial reference set. This ensures that the descriptors do not stray too far from the initial reference set.

The controller 106 can perform the above-described method for adjusting the descriptors fully automatically, without a human user being involved (except possibly initially for setting the first reference). Using camera images from the target application instead of a generic dataset allows for optimal performance. Therefore, according to one embodiment, the method is used online during operation of the application (e.g. while using a robot to pick objects from a box) to automatically improve the quality of the results over time. For each newly received camera image of the sequence of camera images, the controller 106 can automatically select which set of descriptors performs best and should be considered for that particular image. For example, the controller 106 uses the current reference set if it is better than a current test set being evaluated for the current camera image, or vice versa.

4 illustrates the procedure described above.

First, descriptors 401 are set manually as the first reference (represented here by their positions in the first descriptor image 402 belonging to a first camera image). To determine the degree of accuracy according to (A) for the (first) reference set, their pairwise distances 403 are determined.

For a second camera image, the positions of the descriptors in the associated second descriptor image 404 are shifted in this example due to detection errors. For example, a descriptor 405 is mismatched by the machine learning model due to light reflections.

Therefore, according to (B), a new descriptor 406 is selected for the descriptor set by means of which the paired distances between the 3D positions of the locations given by the descriptors better match the determined distances 403, i.e. the shape of the determined area better matches the original shape . This adjustment (or testing for a possible adjustment) is carried out repeatedly for new camera images (e.g. for each received camera image in a sequence of camera images).

In summary, according to various embodiments, a method as in 5 illustrated provided.

5 FIG. 5 shows a flowchart 500 for a method for locating locations of objects from camera images of the objects.

In 501 the locations to be localized for an object type of the objects are specified.

In 502 a reference for the relative position of the points to be located is determined.

In 503, for the object type, a machine learning model for mapping camera images, each camera image showing an object of the object type, is trained on descriptor images, with a descriptor image onto which a camera image is to be mapped for a point of the object that the camera image is at an image position shows at the image position a descriptor of the location of the object.

In 504, a reference set of descriptors is set to an initial set of descriptors.

In 505 a temporal sequence of camera images is received, each camera image showing an object of the object type.

In 506, for each camera image, the locations to be localized on the respective object are localized by mapping the camera image to a descriptor image using the trained machine learning model and identifying the locations of the object to be localized for the reference set of descriptors by searching the descriptors of the reference set of descriptors in the descriptor image (e.g. to search for positions in the descriptor value that have descriptors as close as possible to the descriptors of the reference set, as above in connection with 3 described). The points to be localized are thus localized in the camera image plane and can be localized in three-dimensional space, for example by means of depth information or by solving a PnP (perspective-n-point) problem.

• Ermitteln der relativen Lage der lokalisierten Stellen für den Referenzsatz; und
• Vergleich der Referenz für die relativen Lage der zu lokalisierenden Stellen und der für den Referenzsatz ermittelten relativen Lage der lokalisierten Stellen und
• Identifizieren der zu lokalisierenden Stellen des Objekts für einen Testsatz von Deskriptoren durch Suchen der Deskriptoren des Testsatzes von Deskriptoren in dem Deskriptorbild;
• Ermitteln der relativen Lage der lokalisierten Stellen für den Testsatz; und Vergleich der Referenz für die relativen Lage der zu lokalisierenden Stellen und der für den Testsatz ermittelten relativen Lage der lokalisierten Stellen; und Aktualisieren der Deskriptoren des Referenzsatzes auf die Deskriptoren des Testsatzes falls die Übereinstimmung für ein oder mehrere der Kamerabilder zwischen der Referenz für die relativen Lage der zu lokalisierenden Stellen und der ermittelten relativen Lage der lokalisierten Stellen für den Testsatz besser ist als für den Referenzsatz.

In 507 the following is carried out for at least some of the camera images of the sequence of camera images:

• determining the relative location of the located locations for the reference set; and
• Comparison of the reference for the relative position of the points to be located and that for the Refe limit rate determined relative position of the localized points and
• identifying the locations of the object to be located for a test set of descriptors by searching the descriptors of the test set of descriptors in the descriptor image;
• determining the relative location of the located locations for the test set; and comparing the reference for the relative locations of the locations to be located and the relative locations of the located locations determined for the test set; and updating the reference set descriptors to the test set descriptors if the match for one or more of the camera images between the reference for the relative locations of locations to be located and the determined relative locations of the located locations is better for the test set than for the reference set.

It should be noted that 506 and 507 are performed in parallel or alternately, e.g., 507 is performed for each camera image (or even after a predetermined number of camera images) in the course of the 506 localization process.

In other words, according to various embodiments, a machine learning model for mapping camera images onto descriptor images is trained and for each camera image the points to be localized on the respective object are determined using the descriptors supplied by the trained machine learning model. The descriptors used to determine the locations to be located are updated (i.e. adjusted) over the course of the sequence of camera images, e.g. to follow changing lighting conditions or to compensate for errors in the first reference set of descriptors or a lack of optimality in the first reference set of descriptors .

The objects are instances of the object type, e.g. all have the same form specified by the object type. For example, the objects are parts of one with a specific shape. However, there can also be differences in shape as long as the topology of the objects is the same. For example, the object type can be "shoe", the points to be localized can be edge points of the shoe tongue and the objects can be different shoes.

For example, the machine learning model is a neural network. However, other machine learning models can also be used, which are trained accordingly.

According to various embodiments, the machine learning model assigns descriptors to pixels of the object (in the image plane of the respective camera image). This can be viewed as indirectly encoding the surface topology of the object. This connection between descriptors and the surface topology can be made explicit by rendering to map the descriptors to the image plane. It should be noted that descriptors on faces (ie points that are not vertices) of the object model can be determined using interpolation. For example, if a surface is given by 3 vertices of the object model with their respective descriptors y1, y2, y3, then at any point of the surface the descriptor y can be a weighted sum of these values
w ₁ * y ₁ + w ₂ * y ₂ + w ₃ * y ₃ can be calculated. In other words, the descriptors are interpolated at the vertices.

For example, to generate image pairs for training data for the machine learning model, an image of the object (e.g. an RGB image) including the object (or several objects) with known 3D (e.g. CAD) model and known pose ( in a global (i.e. world) coordinate system, is mapped to a (dense) descriptor image that is optimal in the sense that it is obtained by a search for descriptors to minimize the deviation of geometric properties (especially the proximity of points of the object) between the object model and of its representation (embedding) in the descriptor space. In practical use, the theoretical optimal solution to the minimization will in general not be located, since the search is restricted to a certain search space. Nevertheless, an estimate of the minimum within the constraints of a practical application (available calculation accuracy, maximum number of iterations, etc.).

Each training data image pair comprises a training input image of the object and a target image, the target image being generated by projecting the descriptors of the vertices visible in the training input image onto the training input image plane according to the pose the object has in the training input image. The images along with their associated target images are used for supervised training of the machine learning model.

The machine learning model is thus trained to recognize unique features of an object (or objects). This information can be used for various robot control applications by evaluating the machine learning model in real time, e.g. B. Predicting an object gripping pose for assembly. It should be noted that the supervised training approach allows for explicit coding of symmetry information.

The procedure of 5 can be performed by one or more computers that include one or more data processing units. The term "computing unit" can be understood as any type of entity that enables the processing of data or signals. For example, the data or signals may be processed according to at least one (ie one or more than one) specific function performed by the data processing unit. A data processing unit may be an analog circuit, a digital circuit, a mixed-signal circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array (FPGA), include or be formed from an integrated circuit or any combination thereof. Any other way of implementing the respective functions, which will be described in more detail below, can also be understood as a data processing unit or logic circuitry. It is understood that one or more of the method steps detailed herein may be performed (e.g., implemented) by a computing device via one or more specific functions performed by the computing device.

Various embodiments may receive and use sensor signals from various sensors, such as a (e.g., RGB or RGB-D) camera, video, radar, LiDAR, ultrasonic, thermal imaging sensor, etc. to obtain sensor data, for example , showing an object. Embodiments are used to autonomously control a robot, e.g. B. a robot manipulator to achieve different manipulation tasks under different scenarios. In particular, embodiments in the control and monitoring of the execution of manipulation tasks z. B. in assembly lines applicable.

Although specific embodiments have been illustrated and described herein, one of ordinary skill in the art should recognize that a variety of alternative and/or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the present invention. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that this invention be limited only by the claims and their equivalents.

Claims (10)

Method for locating locations of objects from camera images of the objects, comprising: determining the locations to be localized for an object type of the objects; determining a reference for the relative location of the locations to be located; Training, for the object type, a machine learning model for mapping camera images, each camera image showing an object of the object type, to descriptor images, a descriptor image to which a camera image is to be mapped, for a location of the object showing the camera image at an image location , has at the image position a descriptor of the location of the object; setting a reference set of descriptors to an initial set of descriptors; Receiving a chronological sequence of camera images, each camera image showing an object of the object type, and locating, for each camera image, the points to be localized on the respective object by: Mapping the camera image to a descriptor image using the trained machine learning model Identifying the locations of the object to be located for the reference set of descriptors by looking up the descriptors of the reference set of descriptors in the descriptor image; and having at least for part of the camera images of the sequence of camera images: determining the relative location of the located locations for the reference set; and Comparison of the reference for the relative position of the locations to be located and the relative position of the located locations determined for the reference set identifying the locations of the object to be located for a test set of descriptors by searching the descriptors of the test set of descriptors in the descriptor image; determining the relative location of the located locations for the test set; and comparing the reference for the relative location of the locations to be located and the relative location of the located locations determined for the test set; and updating the descriptors of the reference set to the descriptors of the test set if the match for one or more of the camera images between the reference for the relative positions of the locations to be located and the determined relative positions of the located locations is better for the test set than for the reference set. procedure after claim 1 , the relative position having the paired distances of the points to be localized or the localized points in three-dimensional space. procedure after claim 1 or 2 , wherein the locations to be localized for the object type are specified on a reference camera image of an object of the object type, the reference camera image is fed to the machine learning model and the reference set of descriptors is based on the descriptors of the locations to be localized in the location defined by the machine learning model for the Reference camera image output descriptor image is set. Procedure according to one of Claims 1 until 3 , where the test set is chosen within a restricted range of the initial set of descriptors. Procedure according to one of Claims 1 until 4 , comprising determining the test set of descriptors using a covariance matrix adaptation evolution strategy method. A method of controlling a robot, comprising: locating locations of an object to be treated by the robot according to any one of Claims 1 until 5 , determining a pose of the object from the localized points and controlling the robot depending on the determined pose and/or determining an area of the object from the localized points and controlling the robot depending on the determined area. Software or hardware agent, in particular a robot, having the following: a camera set up to provide camera images of objects; a control device for carrying out the method according to one of Claims 1 until 6 is set up. software or hardware agent claim 7 , which has at least one actuator, wherein the control device is configured to control the at least one actuator using the localized locations. A computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of Claims 1 until 6 performs. A computer-readable medium storing instructions which, when executed by a processor, cause the processor to perform a method according to any one of Claims 1 until 6 performs.

Images