DE102018209220A1 - Method and system for handling objects using a robot - Google Patents

Abstract

A method according to the invention for handling objects (51, 52) with the aid of at least one robot (10) comprises the steps: - detecting (S20) pre-selected (S10) sample objects (31-33) to be handled robotically; - classifying (S40) the Sample objects based on different combinations of different features of the captured sample objects; - Selecting (S60) features based on this classification; - Capturing (S70) target objects (51, 52); - Clusters (S90) of the target objects based on the selected features; and - manipulating (S100) the target objects using the robot based on this clustering.

Description

The present invention relates to a method and system for handling objects using at least one robot and to a computer program product for carrying out the method.

Robots are often designed to handle objects, such as transport or edit.

For this purpose, it is known from in-house practice to visually detect workspaces, to identify objects therein and to handle them with robots. The disadvantage of this, the objects to be identified must be known in advance.

In particular, for a visual object recognition the so-called machine learning is basically known from the literature.

The object of the present invention is to improve the handling of objects using robots.

This object is achieved by a method having the features of claim 1. Claims 11, 12 protect a system or computer program product for carrying out a method described herein. The subclaims relate to advantageous developments.

• - Erfassen, insbesondere paralleles oder sequentielles Erfassen, von als robotisch handzuhabend vorausgewählten Beispielobjekten;
• - Klassifizieren der Beispielobjekte auf Basis unterschiedlicher Kombinationen verschiedener Merkmale der erfassten Beispielobjekte;
• - Auswählen von Merkmalen auf Basis dieses Klassifizierens;
• - Erfassen, insbesondere paralleles oder sequentielles Erfassen, von Zielobjekten;
• - Clustern der Zielobjekte auf Basis der ausgewählten Merkmale; und
• - Handhaben der Zielobjekte mithilfe des Roboters auf Basis dieses Clusterns. Somit wird in einer Ausführung ein zweiphasiges bzw. -stufiges maschinelles Lernen für das Handhaben von Objekten mithilfe von Robotern vorgeschlagen, bei dem
• - in einer ersten Phase bzw. Stufe anhand von als robotisch handzuhabend vorausgewählten Beispielobjekten spezifisch diejenige Kombination von Merkmalen ausgewählt wird, die zum Identifzieren robotisch handzuhabender Objekte (voraussichtlich) besonders geeignet sind; und
• - in einer zweiten Phase bzw. Stufe anhand dieser ausgewählten Merkmale dann Objekte, die mithilfe des bzw. der Roboter(s) handgehabt werden sollen, geclustert werden.

According to an embodiment of the present invention, a method of handling objects using one or more robots includes the steps of:

• Detecting, in particular parallel or sequential detection, of example objects preselected to be robatically operable;
• Classifying the example objects based on different combinations of different features of the captured example objects;
• - selecting features based on this classifying;
• Detecting, in particular parallel or sequential detection, of target objects;
• Clustering the target objects based on the selected features; and
• - Handling the target objects using the robot based on this clustering. Thus, in one embodiment, two-stage machine learning is proposed for handling objects using robots, in which
• in a first phase or stage, the combination of features which are (presumably) particularly suitable for identifying robotic objects is selected specifically on the basis of example objects preselected as being robatically operable; and
• in a second phase or stage, on the basis of these selected features, then objects which are to be handled by the robot (s) are clustered.

By selecting suitable features in the first phase, the cluster (i) n (g) in the second phase can be improved over non-specific or generally pre-trained object identifications. The target objects are not yet known (specifically) in one execution before or before the clustering.

The concept can be illustrated with a simple example: assuming new control cabinet components that are not yet exactly known or specific (target objects) are to be gripped by a robot from a conveyor belt and transported away. Then it can be determined from known control cabinet components (example objects) that the color is not a suitable feature for the identification of control cabinet components or for distinguishing between control cabinet components and other products, but their shape is very specific. Accordingly, the outer contour can be selected as a feature. Now, if the new cabinet components are promoted together with other products, they can be recognized as belonging to the same group or object class belonging to their outer contour and handled as such in a uniform manner by the robot, for example, gripped with the same gripper and / or transported to the same delivery location.

The or one or more of the robots in one embodiment (at least) at least three, in particular at least six, in one embodiment at least seven, joints or axes, in particular hinges or axes, in one embodiment, a robot arm with at least three, in particular at least six, in one embodiment at least seven, joints or axes, in particular rotary joints or axes, and / or one, in particular mobile, stationary or movable on a linear axis, basis. For such robots, the present invention is particularly advantageous because of their variability.

The example objects are or are preselected in one embodiment on the basis of a prediction of the target objects as robotic, in particular for the (prospective) target objects typical objects. As a result, the cluster (i) n (g) can be (further) improved.

Classifying the example objects may in one embodiment comprise supervised learning, in particular. In the present case, adaptation or adaptation of a model, for example a neural network, a Support Vector Machine, Decision Trees, Random Forrests, Markov Systems, in particular Markov Process, Hidden Markov Model or Markov Random Fields, Naive Bayes, Belief networks, or the like understood, the (input) features on (predetermined) object classes maps or assigns different values of (input) features different (predetermined) object classes.

• - Klassifizieren einer Testmenge der Beispielobjekte auf Basis einer ersten Kombination von Merkmalen einer Trainingsmenge der erfassten Beispielobjekte;
• - Klassifizieren einer, insbesondere derselben, Testmenge der Beispielobjekte auf Basis einer oder mehrerer weiterer Kombinationen von Merkmalen einer, insbesondere derselben, Trainingsmenge der erfassten Beispielobjekte;
• - Bewerten dieser Klassifizierungen; und
• - Auswählen der Merkmale auf Basis dieser Bewertung, insbesondere derjenigen Merkmalskombination mit der besten Bewertung, insbesondere der besten Klassifizierung der Trainings(menge der erfassten Beispiel)objekte.

Accordingly, the method or classification in one embodiment comprises the steps:

• Classifying a test set of the example objects based on a first combination of features of a training set of the captured example objects;
• Classifying one, in particular the same, test set of the example objects on the basis of one or more further combinations of features of, in particular the same, training set of the captured example objects;
• - evaluating these classifications; and
• Selecting the features on the basis of this evaluation, in particular that combination of features with the best rating, in particular the best classification of the training (amount of captured example) objects.

The number of features selected may be predetermined, which in one embodiment may improve convergence and / or predictability. In another embodiment, the number of selected features is dynamically determined or changed on the basis of the classification, increased in one embodiment until a predetermined classification accuracy or minimum rating has been achieved or reduced until a predetermined classification accuracy or minimum rating is no longer achieved.

The clustering of the target objects may in one embodiment comprise unsupervised learning, in particular. In the present case, this is understood to mean the assignment of (different values) of (input) features to or in (different, in one embodiment, previously unknown) object classes or groups, in a manner without reward (by the environment).

• - Ermitteln eines ein- oder mehrdimensionalen Identifikationsparameters eines ersten Zielobjekts auf Basis bzw. in Abhängigkeit von ausgewählten Merkmalen dieses Zielobjekts;
• - Ermitteln (je) eines ein- oder mehrdimensionalen Identifikationsparameters eines oder mehrerer weiterer Zielobjekte auf Basis bzw. in Abhängigkeit von deren ausgewählten Merkmalen; und
• - Zuordnen des ersten Zielobjekts und des (jeweiligen) weiteren Zielobjekts zu derselben Gruppe bzw. Objektklasse, falls eine Abweichung ihrer Identifikationsparameter, insbesondere voneinander und/oder einem Identifikationsparameter der Gruppe bzw. Objektklasse, beispielsweise einem Mittelwert oder dergleichen, einen vorgegebenen Schwellwert unterschreitet.

Accordingly, the method or clusters in one embodiment comprises the steps:

• Determining a one-dimensional or multi-dimensional identification parameter of a first target object on the basis of or depending on selected features of this target object;
• Determining (each) a one-dimensional or multi-dimensional identification parameter of one or more further target objects based on or depending on their selected features; and
• - Assign the first target object and the (respective) further target object to the same group or object class, if a deviation of their identification parameters, in particular from each other and / or an identification parameter of the group or object class, such as an average value or the like, below a predetermined threshold.

In one embodiment, the groups or object classes are generated dynamically, in particular only during clustering. Thus, for example, a new group or object class can be generated for each new target object whose identification parameter deviates from the identification parameters of the already existing groups or object classes or those already assigned to target objects by more than the threshold value.

In one embodiment, the example objects and / or target objects are detected optically or visually, in particular by means of one or more RGB cameras, in particular one or more RGBD cameras.

By means of the optical or visual detection, the object classification or the (object) cluster (i) n (g) ("object recognition") can take place advantageously, in particular without interaction, quickly and / or precisely. By RGB (D) cameras can be used advantageously additional features.

In one embodiment, the example objects and / or target objects are captured from different perspectives. As a result, in one embodiment, features that are not determinable from a perspective can advantageously also be used for object recognition and / or their reliability can be increased.

Additionally or alternatively, or at least one of the cameras is guided by the robot or one of the robots. In this way, in one embodiment, objects from different perspectives and / or from close proximity and / or immediately before handling by the camera-guiding robot can advantageously be detected.

In one embodiment, the handling of target objects may include picking up, in particular magnetic, pneumatic and / or mechanical, in particular positive and / or frictional, gripping, transporting, dispensing, in particular settling or laying, and / or processing, in particular coating, in particular painting, welding, soldering and / or gluing, and / or a machining Processing of the target objects include, in particular be, which is carried out or carried out in an embodiment based on or in dependence on the cluster (i) n (g).

• - Clustern eines ersten Zielobjekts sowie eines oder mehrerer weiterer Zielobjekte auf Basis bzw. in Abhängigkeit von deren ausgewählten Merkmalen;
• - gleichartiges Handhaben von Zielobjekten, falls diese derselben Gruppe bzw. Objektklasse zugeordnet worden sind; und
• - unterschiedliches Handhaben von Zielobjekten, falls diese verschiedenen Gruppen bzw. Objektklassen zugeordnet worden sind.

Accordingly, in one embodiment, the method comprises the steps:

• Clustering a first target object and one or more further target objects based on or depending on their selected features;
• - similar handling of target objects, if they have been assigned to the same group or object class; and
• - Different handling of target objects, if they have been assigned to different groups or object classes.

Thus, in one embodiment, target objects that have been assigned to clustering the same group or class of objects are similarly handled, for example, similarly gripped and / or delivered at the same location, and target objects that are different in clusters or object classes have been assigned, handled differently, for example, different gripped and / or delivered at different locations.

• - Extrahieren verschiedener Merkmale der erfassten Beispielobjekte; und
• - Klassifizieren der Beispielobjekte auf Basis unterschiedlicher Kombinationen dieser extrahierten Merkmale.

In one embodiment, the method comprises the steps:

• Extracting various features of the captured example objects; and
• Classify the example objects based on different combinations of these extracted features.

• - Extrahieren der ausgewählten Merkmale der erfassten Zielobjekte; und
• - Clustern der Zielobjekte auf Basis dieser extrahierten Merkmale.

Additionally or alternatively, in one embodiment, the method comprises the steps of:

• - extracting the selected features of the detected target objects; and
• Clusters the target objects based on these extracted features.

In particular, extracting features of captured (example or target) objects may segment, in one embodiment a color segmentation, a plane segmentation, a hull segmentation, an object segmentation ("Object segmentation") and / or masking, in particular.

Features on the basis of which (example or target) objects are classified or clustered can in one embodiment have color and / or brightness values, in particular red, green and / or blue values of the RGB color space, color value (" hue "), saturation, brightness of the HSV color space, or the like, dimensional values, in particular height, width, length, volume, edge length, and / or distance values, in particular a distance to a center or and the like, as well as mean values and / or standard deviations thereof, in particular. As a result, the cluster (i) n (g) can be (further) improved.

• - Mittel zum Erfassen von als robotisch handzuhabend vorausgewählten Beispielobjekten;
• - Mittel zum Klassifizieren der Beispielobjekte auf Basis unterschiedlicher Kombinationen verschiedener Merkmale der erfassten Beispielobjekte;
• - Mittel zum Auswählen von Merkmalen auf Basis dieses Klassifizierens;
• - Mittel zum Erfassen von Zielobjekten;
• - Mittel zum Clustern der Zielobjekte auf Basis der ausgewählten Merkmale; und
• - Mittel zum Handhaben der Zielobjekte mithilfe des Roboters auf Basis dieses Clusterns.

According to one embodiment of the present invention, a system, in particular hardware and / or software, in particular program technology, is set up to carry out a method described here and / or has:

• Means for detecting example objects preselected to be robotically manipulable;
• Means for classifying the example objects based on different combinations of different features of the captured example objects;
• - means for selecting features based on this classifying;
• - means for detecting target objects;
• Means for clustering the target objects based on the selected features; and
• - Means for handling the target objects using the robot based on this clustering.

• Mittel zum optischen Erfassen der Beispielobjekte und/oder Zielobjekte, insbesondere mittels wenigstens einer, in einer Ausführung robotergeführten, RGB-Kamera; und/oder
• Mittel zum Erfassen der Beispielobjekte und/oder Zielobjekte aus verschiedenen Perspektiven; und/oder
• Mittel zum Aufnehmen, Transportieren, Abgeben und/oder Bearbeiten der Zielobjekte; und/oder
• Mittel zum Vorauswählen der Beispielobjekte auf Basis einer Prognose der Zielobjekte; und/oder
• Mittel zum überwachten Lernen zum Klassifizieren der Beispielobjekte und/oder Mittel zum unüberwachten Lernen zum Clustern der Zielobjekte; und/oder Mittel zum Extrahieren verschiedener Merkmale der erfassten Beispielobjekte und Klassifizieren der Beispielobjekte auf Basis unterschiedlicher Kombinationen dieser extrahierten Merkmale; und/oder
• Mittel zum Extrahieren der ausgewählten Merkmale der erfassten Zielobjekte und Clustern der Zielobjekte auf Basis dieser extrahierten Merkmale; und/oder
• Mittel zum Klassifizieren einer Testmenge der Beispielobjekte auf Basis einer ersten Kombination von Merkmalen einer Trainingsmenge der erfassten Beispielobjekte, Klassifizieren einer, insbesondere derselben, Testmenge der Beispielobjekte auf Basis wenigstens einer weiteren Kombination von Merkmalen einer, insbesondere derselben, Trainingsmenge der erfassten Beispielobjekte, Bewerten dieser Klassifizierungen und Auswählen der Merkmale auf Basis dieser Bewertung; und/oder Mittel zum Ermitteln eines Identifikationsparameters eines ersten Zielobjekts auf Basis seiner ausgewählten Merkmale, Ermitteln eines Identifikationsparameters wenigstens eines weiteren Zielobjekts auf Basis seiner ausgewählten Merkmale und Zuordnen dieses weiteren Zielobjekts und des ersten Zielobjekts zu derselben Gruppe, falls eine Abweichung ihrer Identifikationsparameter einen vorgegebenen Schwellwert unterschreitet; und/oder
• Mittel zum Clustern eines ersten Zielobjekts sowie wenigstens eines weiteren Zielobjekts auf Basis von deren ausgewählten Merkmalen, gleichartigen Handhaben dieser Zielobjekte, falls diese derselben Gruppe zugeordnet worden sind, und unterschiedlichen Handhaben dieser Zielobjekte, falls diese verschiedenen Gruppen zugeordnet worden sind.

In one embodiment, the system or its agent has:

• Means for optically detecting the example objects and / or target objects, in particular by means of at least one robot-guided RGB camera in one embodiment; and or
• Means for collecting the example objects and / or target objects from different perspectives; and or
• Means for capturing, transporting, delivering and / or editing the target objects; and or
• Means for preselecting the example objects based on a prediction of the target objects; and or
• Supervised learning means for classifying the example objects and / or unsupervised learning means for clustering the target objects; and / or means for extracting various features of the captured example objects and classifying the example objects Base different combinations of these extracted features; and or
• Means for extracting the selected features of the detected target objects and clusters the target objects based on those extracted features; and or
• Means for classifying a test set of the example objects based on a first combination of features of a training set of the acquired example objects, classifying one, in particular the same, test set of the example objects on the basis of at least one further combination of features of, in particular, training set of the acquired example objects, evaluating these classifications and selecting the features based on that score; and / or means for determining an identification parameter of a first target object based on its selected features, determining an identification parameter of at least one further target object based on its selected features, and assigning that further target object and the first target object to the same group if a deviation of their identification parameters exceeds a predetermined threshold below; and or
• Means for clustering a first target object and at least one further target object based on their selected features, similar handling of those target objects if they have been assigned to the same group, and different handling of those target objects if they have been assigned to different groups.

A means in the sense of the present invention may be formed by hardware and / or software technology, in particular a data or signal-connected, preferably digital, processing, in particular microprocessor unit (CPU, GPU or the like), preferably with a memory and / or bus system. and / or one or more programs or program modules. The CPU or GPU may be configured to execute instructions implemented as a program stored in a memory system, to capture input signals from a data bus, and / or to output output signals to a data bus. A storage system may comprise one or more, in particular different, storage media, in particular optical, magnetic, solid state and / or other non-volatile media. The program may be such that it is capable of embodying or executing the methods described herein so that the CPU or GPU may perform the steps of such methods and thus, in particular, control the robot (s). In one embodiment, a computer program product may include, in particular, a non-volatile storage medium for storing a program or a program stored thereon, wherein execution of this program causes a system or a controller, in particular a computer, to do so method described herein or one or more of its steps. GPUs can be used in one embodiment for, in particular, high-level, parallel computation.

In one embodiment, one or more, in particular all, steps of the method are completely or partially automated, in particular by the system or its (e) means. In one embodiment, the system has the robot (s).

• 1: ein System zum Handhaben von Objekten mithilfe eines Roboters nach einer Ausführung der vorliegenden Erfindung; und
• 2: ein Verfahren zum Handhaben von Objekten mithilfe des Roboters nach einer Ausführung der vorliegenden Erfindung.

Further advantages and features emerge from the subclaims and the exemplary embodiments. This shows, partially schematized:

• 1 : a system for handling objects using a robot according to an embodiment of the present invention; and
• 2 : A method of manipulating objects using the robot according to an embodiment of the present invention.

1 shows a system for handling target objects 51 . 52 using a robot (arm) s 10 according to an embodiment of the present invention by a robot controller 2 which executes a method described below according to an embodiment of the present invention.

The robot (poor) 10 leads a gripper 11 and an RGBD camera 12 with the objects on a conveyor belt 4 be captured optically.

In a first step S10 become example objects 31 - 33 based on a forecast of the target objects 51 . 52 , which is expected later with the robot 10 to be handled, preselected.

For example, if the robot later wants to sort different electronic components, then in step S10 typical representatives of the various components, such as multiple connectors, multiple boards and the like, preselected as example objects and on the conveyor belt 4 arranged.

In one step S20 become the example objects 31 - 33 parallel or sequential with the RGBD camera 12 detected.

In one step S30 become different features of the captured example objects 31 - 33 extracted, for example, color values, dimensional values or the like.

In one step S40 For example, a test set of the example objects is classified based on a first combination of the extracted features of a training set of the acquired example objects, and the same test set is classified based on another combination of the extracted features of the same training set of the acquired example objects; for example, the example object greatly simplified in the embodiment 33 (Test set) once based on color values and once based on dimension values of the example objects 31 . 32 (Training set).

The respective classification will be in step S50 evaluated and in step S60 selected the combination of features for which the classification was best.

Are again purely exemplary the example objects 31 . 33 (typical) plug and the example object 32 a (typical) board, and could the example object 33 On the basis of color values, they are best classified as connectors, so they are selected as feature (s). Could the example object 33 however, on the basis of dimensional values, they are best classified as connectors, so these are selected as feature (s). Of course, this is just a highly simplified example, usually combinations of several different features will be selected.

In one step S70 become the target objects 51 . 52 parallel or sequential with the RGBD camera 12 captured and in one step S80 the in step S60 selected features of the detected target objects 51 . 52 extracted, in the simplistic example so the color or dimensional values.

In one step S80 become the target objects 51 . 52 clustered on the basis of these selected, extracted features, in the exemplary embodiment combined into object classes or groups of similar color or dimension values.

Then the robot takes 10 in step S100 the target objects 51 . 52 sequentially and place them at one of several storage locations 61 - 63 where it deposits destination objects of the same object class or group at the same storage location.

Although exemplary embodiments have been explained in the foregoing description, it should be understood that a variety of modifications are possible. It should also be noted that the exemplary embodiments are merely examples that are not intended to limit the scope, applications and construction in any way. Rather, the expert is given by the preceding description, a guide for the implementation of at least one exemplary embodiment, with various changes, in particular with regard to the function and arrangement of the components described, can be made without departing from the scope, as it turns out according to the claims and these equivalent combinations of features.

LIST OF REFERENCE NUMBERS

10

robot

11

grab

12

RGBB camera

2

control

31-33

example objects

4

conveyor belt

51, 52

targets

61-63

storage spaces

Claims (12)

Method for handling objects (51, 52) using at least one robot (10), comprising the steps: - acquiring (S20) example objects (31-33) preselected as robotically operable (S10); - classifying (S40) the example objects based on different combinations of different features of the captured example objects; - selecting (S60) features on the basis of this classifying; - detecting (S70) target objects (51, 52); Clustering (S90) the target objects based on the selected features; and - Handling (S100) the target objects using the robot based on this clustering. Method according to Claim 1 , characterized in that the example objects and / or target objects are detected optically, in particular by means of at least one RGB camera (12). Method according to one of the preceding claims, characterized in that the camera is guided by a robot and / or the objects are detected from different perspectives. Method according to one of the preceding claims, characterized in that the handling comprises picking up, transporting, delivering and / or processing the target objects. Method according to one of the preceding claims, characterized in that the Example objects are preselected based on a prediction of the target objects (S10). Method according to one of the preceding claims, characterized in that the classification of the sample objects comprises a supervised learning (S40) and / or the clustering of the target objects an unsupervised learning (S90). Method according to one of the preceding claims, characterized by the steps of: - extracting (S30) various features of the captured example objects; and - classifying (S40) the example objects based on different combinations of these extracted features; and / or the steps of: - extracting (S80) the selected features of the detected target objects; and clusters (S90) the target objects based on these extracted features. Method according to one of the preceding claims, characterized by the steps: - classifying (S40) a test set (33) of the example objects on the basis of a first combination of features of a training set (31, 32) of the captured example objects; - classifying (S40) one, in particular the same, test set (33) of the example objects on the basis of at least one further combination of features of, in particular, training set (31, 32) of the captured example objects; - evaluating (S50) these classifications; and - selecting (S60) the features based on that score. Method according to one of the preceding claims, characterized by the steps: - determining an identification parameter of a first target object on the basis of its selected features; Determining an identification parameter of at least one further target object on the basis of its selected features; and - assigning this further target object and the first target object to the same group if a deviation of their identification parameters falls short of a predetermined threshold value. Method according to one of the preceding claims, characterized by the steps: - clusters of a first target object and at least one further target object on the basis of their selected features; - similar handling of these target objects, if they have been assigned to the same group; and - different handling of these target objects if they have been assigned to different groups. System for handling objects by means of at least one robot (10), which is set up to carry out a method according to one of the preceding claims and / or comprises: - means for detecting example objects (31-33) preselected to be robotic to handle; Means for classifying the example objects based on different combinations of different features of the captured example objects; - means for selecting features based on this classifying; - means for detecting target objects (51, 52); Means for clustering the target objects based on the selected features; and - Means for handling the target objects using the robot based on this clustering. A computer program product having a program code stored on a computer-readable medium for performing a method according to any one of the preceding claims.

Images