DE102020206403A1 - Configuring, executing and / or analyzing an application of a mobile and / or collaborative robot - Google Patents

Abstract

A method according to the invention for configuring, performing and / or analyzing an application of a robot arrangement (1, 1.1) with at least one mobile and / or collaborative robot (1), with the aid of a user interface (2; 2 ') for inputting and / or outputting Data has the step: superimposing (S20) a view, in particular a representation (10; 20; 30), of an environment with at least one virtual object (110-210); this overlay for configuring, executing and / or analyzing the application is made available by the, in particular on the, user interface, in particular the view of the environment and the virtual object by which, in particular on the, user interface is overlaid and / or the application is configured, performed and / or analyzed on the basis of this superimposition, in particular with the aid of the user interface (S30).

Description

The present invention relates to a method and a system for configuring, executing and / or analyzing an application of a robot arrangement with at least one mobile and / or collaborative robot using a user interface for inputting and / or outputting data and a computer program product for executing the method.

The so-called mixed reality ("Mixed Reality", MR) includes in particular the so-called augmented reality ("Augmented Reality", AR) and the extended virtuality ("Augmented Virtuality", AR). In one embodiment, real and virtual objects interact in the mixed reality.

In contrast, a basic idea of the present invention is the use of a mixed reality (MR or AR) that is expanded in one embodiment for configuring, executing and / or analyzing applications of mobile and / or collaborative robots.

One object of an embodiment of the present invention is accordingly to improve the use of mobile and / or collaborative robots.

This object is achieved by a method with the features of claim 1. Claims 13, 14 provide a system or computer program product for performing a method described here under protection. The subclaims relate to advantageous developments.

According to one embodiment of the present invention, a robot arrangement has one or more mobile and / or collaborative robots. In one embodiment, the or one or more of the mobile robots, in one embodiment collaborative robots (each) have a mobile base, in particular a chassis, in one embodiment not rail-bound, in particular one or more, in particular steered and / or driven wheels and / or at least one robot arm, which in one embodiment has at least six joints or axes, in particular swivel joints or axes, and / or is arranged on, in particular on the mobile base, and / or one Robot-side energy supply, in particular an electrical energy store, for at least one drive of the mobile base and / or the robot arm. In one embodiment, the or one or more of the mobile robots are AGVs and / or collaborative robots. The robot arrangement can additionally or alternatively have one or more stationary robots, in particular one or more (stationary) robot arms, which in one embodiment each have at least six joints or axes, in particular swivel joints or axes, and / or one external to the robot in one Execution have central, robot arrangement control, which controls one or more of the mobile and / or one or more of the stationary robots or is provided for this purpose, in particular is set up or is used, in particular a fleet management (system).

Collaborated in one embodiment, human-collaborated in one embodiment, the or at least one of the collaborative robots at least temporarily ("human-robot collaboration"), in one embodiment it works, at least temporarily, (also) (fully) automated or . without (human or human-robot) collaboration, alternates in one version between (human) collaborative operation and (fully) automated operation or operation without (human or human-robot) collaboration, or is the Robot or the robot arrangement capable of this, in particular provided, set up in one embodiment, or is used for this purpose. In one embodiment, the or one or more of the collaborative robots are cobots and / or the robot arrangement comprises at least one hybrid cell. In one embodiment, the or at least one of the collaborative robots is a mobile or stationary robot and / or has a robot arm, which in one embodiment has at least six joints or axes, in particular swivel joints or axes. In one embodiment, the application comprises one or no human-robot collaboration, in one embodiment a change between (human) collaborative operation and (fully) automated operation or operation without (human or human-robot) collaboration. Correspondingly, a collaborative robot in one embodiment is a robot which, in one embodiment, is provided, in particular set up, or is used, at least temporarily to carry out a human-robot collaboration or, in one embodiment, the robot arrangement is a robot arrangement that is capable of doing this capable of being provided, in particular set up, or used in an embodiment that (at least one) of your robot (s) carries out a human-robot collaboration at least temporarily, in one embodiment the collaborative robot at least temporarily carries out a human-robot Collaboration through.

The present invention can be used with particular advantage for such robots (arrangements), in particular because of their fields of application, complexity and / or variability, without being restricted thereto.

1. A) Überlagern
   1. a) einer Ansicht einer Umgebung, in einer Ausführung einer Umgebung der Benutzerschnittstelle und/oder der Roboteranordnung, insbesondere des bzw. eines oder mehrerer der mobilen und/oder kollaborativen Roboter(s), mit
   2. b) einem oder mehreren virtuellen Objekten, in einer Ausführung durch die, in einer Ausführung an, insbesondere auf oder in, der, Benutzerschnittstelle, in einer Ausführung Überlagern in der Ansicht der Umgebung; und
2. B) zur Verfügung stellen dieser Überlagerung durch die, in einer Ausführung an, insbesondere auf oder in, der, Benutzerschnittstelle, insbesondere dem Benutzer, zum Konfigurieren, Durchführen und/oder Analysieren der Applikation, in einer Ausführung Konfigurieren, Durchführen und/oder Analysieren der Applikation auf Basis, insbesondere mithilfe, dieser Überlagerung, in einer Ausführung mithilfe der, insbesondere durch, die Benutzerschnittstelle.

According to one embodiment of the present invention, a method for configuring, performing and / or analyzing an application of the robot arrangement with the aid of a single or Multipart user interface via which a, in particular human, user inputs data and / or outputs the data to a user or is provided for this purpose, in particular is set up and / or used, the steps of:

1. A) Overlay
   1. a) a view of an environment, in one embodiment of an environment of the user interface and / or the robot arrangement, in particular of the or one or more of the mobile and / or collaborative robot (s)
   2. b) one or more virtual objects, in one embodiment by, in one embodiment on, in particular on or in, the user interface, in one embodiment overlaying in the view of the environment; and
2. B) make this overlay available through the, in one embodiment at, in particular on or in the, user interface, in particular the user, for configuring, performing and / or analyzing the application, in one embodiment configuring, performing and / or analyzing the Application based on, in particular with the aid of, this overlay, in one embodiment with the aid of, in particular through, the user interface.

As a result, the time required and / or the effort required for configuring, performing or analyzing the application can be reduced in one embodiment and / or the reliability and / or precision achieved thereby can be increased.

In one embodiment, the configuration includes planning or specifying movements and / or activities or work of the robot arrangement and / or an environment interacting with it, in particular storage, loading and / or unloading stations, conveying, loading or processing, Testing and / or refueling and / or electrical (charging) charging stations or the like and / or design, in particular parameterization and / or changing of the robot arrangement and / or the environment interacting with it. In one embodiment, the configuration takes place before the application is carried out.

In one embodiment, performing an application of the robot arrangement includes commanding, in particular controlling or regulating, and / or monitoring movements and / or activities of the robot arrangement and / or an environment interacting with it, in particular storage, loading and / or unloading stations, conveyors -, processing, testing and / or refueling and / or electrical (charging) charging stations or the like. In one embodiment, the implementation according to the invention takes place online. In one embodiment, the implementation of an application of the robot arrangement includes, in particular, the control of the or one or more of the mobile and / or collaborative robot (s), in one embodiment online and / or remotely by a user of the user interface.

In one embodiment, the analyzing includes collecting, in particular storing, and / or evaluating data from the robot arrangement and / or an environment interacting with it, in particular storage, loading and / or unloading stations, conveying, handling or processing, testing and / or refueling and / or electrical (charging) charging stations or the like, in particular an evaluation during and / or after the application has been carried out.

• - wenigstens eine, in einer Ausführung zwei- oder dreidimensionale, Darstellung, der Umgebung und/oder
• - einen Blick bzw. ein Sichtfeld eines Benutzers der Benutzerschnittstelle.

The view of the environment includes in one implementation

• - At least one, in one embodiment two- or three-dimensional, representation of the environment and / or
• a view or a field of view of a user of the user interface.

• - wenigstens ein aufgenommenes, insbesondere aktuelles, in einer Ausführung aktualisiertes, oder gespeichertes, Bild, insbesondere Kamerabild, der Umgebung und/oder
• - eine künstliche Rekonstruktion der Umgebung, in einer Ausführung an, insbesondere auf bzw. in, der Benutzerschnittstelle.

This representation of the environment in turn includes in one embodiment

• - At least one recorded, in particular current, updated or stored in an execution, image, in particular camera image, of the environment and / or
• an artificial reconstruction of the environment, in one embodiment on, in particular on or in, the user interface.

• - mithilfe der, insbesondere durch die, Benutzerschnittstelle bzw. wenigstens eine(r) benutzerschnittstellenseitige(n) Kamera,
• - mithilfe der, insbesondere durch die, Roboteranordnung, insbesondere des bzw. den mobilen und/oder kollaborativen Roboter(s), bzw. wenigstens eine(r) roboter(anordnungs)seitige(n) Kamera und/oder
• - mithilfe, insbesondere durch, wenigstens eine(r) benutzerschnittstellen- und roboteranordnungsexterne(n) Kamera aufgenommen, insbesondere aktualisiert.

The camera image is or will be in one version

• - with the help of, in particular through the, user interface or at least one camera (s) on the user interface,
• - with the aid of the, in particular through the, robot arrangement, in particular the mobile and / or collaborative robot (s), or at least one robot (arrangement) camera and / or
• - Recorded, in particular updated, with the aid of, in particular by, at least one camera external to the user interface and robot arrangement.

• - Überlagern eines aufgenommenen (Ab-, insbesondere Kamera)Bildes der Umgebung und des wenigstens einen virtuellen Objekts, in einer Ausführung auf einem Bildschirm der Benutzerschnittstelle oder durch Projektion;
• - Überlagern einer künstlichen Rekonstruktion der Umgebung, beispielsweise einer Darstellung der Umgebung in Form von Gittern, vereinfachten geometrischen Flächen und/oder Körpern, virtuellen Wänden, Böden, Decken, Körpern und dergleichen, und des wenigstens einen virtuellen Objekts, in einer Ausführung auf einem Bildschirm der Benutzerschnittstelle oder durch Projektion;
• - Überlagern des wenigstens einen virtuellen Objekts mit einem Blick bzw. Sichtfeld bzw. einer Sicht des Benutzers auf die Umgebung, insbesondere auf einem Brillen-, Linsen- oder Maskenglas der Benutzerschnittstelle oder durch Projektion.

Thus, the superimposition includes in particular the following, non-exhaustive variants:

• - Superimposing a recorded (image, in particular camera) image of the environment and the at least one virtual object, in one embodiment on a screen of the user interface or by projection;
• - Superimposing an artificial reconstruction of the environment, for example a representation of the environment in the form of grids, simplified geometric surfaces and / or bodies, virtual walls, floors, ceilings, bodies and the like, and the at least one virtual object, in one embodiment on a screen the user interface or by projection;
• - Overlaying the at least one virtual object with a view or field of view or a view of the user on the surroundings, in particular on a spectacle, lens or mask lens of the user interface or by projection.

• - auf Basis bzw. in Abhängigkeit von einer ermittelten, in einer Ausführung aktuellen, in einer Ausführung aktualisierten, Perspektive der Ansicht;
• - auf Basis bzw. in Abhängigkeit von (einer) ermittelten, in einer Ausführung aktuellen, in einer Ausführung aktualisierten, Pose(n)
  • - der Benutzerschnittstelle,
  • - der Roboteranordnung, insbesondere des bzw. eines oder mehrerer der mobilen und/oder kollaborativen Roboter(s), und/oder
  • - wenigstens eines, in einer Ausführung statischen oder bewegten, Umgebungsobjekts, in einer Ausführung wenigstens eines mit dem bzw. einem der virtuellen Objekt(e) verknüpften Umgebungsobjekts,
  relativ zu der Umgebung; und/oder
• - auf Basis bzw. in Abhängigkeit von (einer) ermittelten, in einer Ausführung aktuellen, in einer Ausführung aktualisierten, Blickrichtung eines Trägers der Benutzerschnittstelle, insbesondere relativ zu der Umgebung und/oder Benutzerschnittstelle

überlagert.In one embodiment, the view of the environment and the virtual object (s) are

• - on the basis of or as a function of a determined perspective of the view that is current in one version and updated in one version;
• - on the basis of or as a function of (a) determined pose (s) that are current in one version and updated in one version
  • - the user interface,
  • - The robot arrangement, in particular the or one or more of the mobile and / or collaborative robot (s), and / or
  • - At least one, in one embodiment static or moving, surrounding object, in one embodiment at least one surrounding object linked to the or one of the virtual object (s),
  relative to the environment; and or
• - on the basis of or as a function of (a) determined, current in an embodiment, updated in an embodiment, viewing direction of a wearer of the user interface, in particular relative to the environment and / or user interface

superimposed.

Correspondingly, in one embodiment, the method includes determining, in particular recording and / or updating, this perspective, this pose (s) and / or viewing direction. In one embodiment, a determined pose or viewing direction is a captured and / or updated pose or viewing direction.

In one embodiment, a pose comprises at least one one, two or three-dimensional position and / or at least one one, two or three-dimensional orientation, in particular a position and / or orientation relative to or in the environment.

In one embodiment, the view of the environment is updated on the basis of the determined pose of the user interface, in one embodiment in real time and / or in such a way that a viewing direction of the view changes (accordingly) with or when the pose of the user interface changes , in one embodiment corresponding to a change in a user interface-fixed viewing direction (relative to the environment).

If, for example, the user interface is rotated around an axis, the viewing direction of the view of the environment also rotates in one embodiment (only) if the viewing direction of the user is constant relative to the user interface or also independently of this.

In one embodiment, the view of the environment is updated on the basis of the determined pose of the robot arrangement, in one embodiment of the mobile and / or collaborative robot, in one embodiment in real time and / or in such a way that a viewing direction of the view with or in a Changing the pose of the robot (correspondingly) (with) changes, in one embodiment corresponding to a change in a robot-fixed viewing direction (relative to the environment).

For example, if the robot rotates around an axis, in one version the direction of view of the view of the environment also rotates (accordingly with).

Thus, in addition to or as an alternative to a view of the environment of the user interface, a view of the environment of the robot arrangement, in an embodiment of the mobile and / or collaborative robot, can be overlaid with one or more virtual objects, for example remote controls, remote sensing or other applications of the To configure, perform and / or analyze robot assembly, for example activities in manufacturing facilities, warehouses or the like.

In one embodiment, a pose, in particular position and / or orientation, of the or at least one of the virtual object (s) in the overlay, in one embodiment in the view of the surroundings, in particular in the representation of the surroundings or the Field of view of the user Based on the determined pose of the robot arrangement, updated in an embodiment of the mobile and / or collaborative robot, in an embodiment in real time and / or in such a way that a pose, in particular position and / or orientation, of the, in particular with the robot arrangement, is in an embodiment of the mobile and / or collaborative robot, linked, virtual object in the overlay or view of the environment with or when the pose of the robot arrangement or the robot is changed relative to the environment (accordingly) (with) changes in a Execution corresponds to the change in the pose, in particular position and / or orientation, of the virtual object in the superimposition or view of the change in pose, in particular position and / or orientation, of the robot arrangement or the robot relative to the environment in terms of direction and size.

If, for example, the robot moves, the virtual object linked to it also shifts in the overlay or view of the environment (accordingly with) in one embodiment.

Likewise, the robot arrangement can also not be linked to virtual objects and / or not (real) present in the (view of) the environment and / or (only) symbolized by virtual objects.

In one embodiment, a pose, in particular position and / or orientation, of the or at least one of the virtual object (s) in the overlay, in one embodiment in the view of the surroundings, in particular in the representation of the surroundings or the Field of view of the user, based on a forecast or only planned but not (yet) approached or planning pose of the robot arrangement, in an embodiment of the mobile and / or collaborative robot, updated, in an execution in real time and / or during planning (phase) of the application and / or such that a pose, in particular position and / or orientation, of the, in particular with the robot arrangement, in one embodiment the mobile and / or collaborative robot, linked, virtual object is in the overlay or view the environment with or in the event of a change in the predicted or planning pose of the robot arrangement or the robot relative to the environment (accordingly) (with) changes rt, in one embodiment, the change in the pose, in particular position and / or orientation, of this virtual object in the overlay or view corresponds to the change in the forecast or planning pose, in particular position and / or orientation, of the robot arrangement or the robot relative to the environment in direction and size.

If, for example, a planning pose, for example a node of a navigation path or the like, is shifted, in particular as a result of a user input, in one embodiment the virtual object linked to it is also shifted in the overlay or view of the environment (correspondingly with).

In one embodiment, a user can thereby advantageously identify the effects of planned or future robots (arrangements), in particular potential injuries to protective spaces, collisions with obstacles or the like.

In one embodiment, a pose, in particular the position and / or orientation, of the or one or more of the virtual objects in the overlay is displayed, in one embodiment in the view of the surroundings, in particular in the representation of the surroundings or the field of view of the user Based on the determined pose of a surrounding object linked to the (respective) virtual object updated, in an execution in real time and / or in such a way that a pose, in particular position and / or orientation, of the (respective) virtual object is in the overlay or view the environment with or when the pose of the associated environment object changes relative to the environment (accordingly) (with), in one embodiment the change corresponds to the pose, in particular the position and / or orientation, of the virtual object in the overlay or View of the change in pose, in particular position and / or orientation, of the surrounding object relative to the surroundings in Ric respect and size.

If, for example, a workpiece, transported goods, tool or conveyor table or another surrounding object is shifted, in one embodiment the virtual object linked to it is also shifted in the overlay or view of the environment (accordingly with).

Likewise, a pose, in particular the position and / or orientation, of the or one or more of the virtual objects relative to an outer contour of the view of the surroundings, in particular relative to an outer contour of the representation of the surroundings or an outer contour of the user's field of vision, can also be constant or . be stationary relative to the outer contour.

For example, if a perspective of the view pivots around a vertical axis, such a virtual object can be relative to an outer contour of a (image, especially camera) image of the surroundings, an outer contour of an artificial reconstruction of the surroundings or an outer contour of a view (field) of the user or an outer contour of a Spectacles, lenses or mask glasses of the user interface remain stationary.

The perspective of the view is, in particular, is determined in one embodiment on the basis of a determined, in particular current, pose of the user interface or robot arrangement, in particular of the mobile and / or collaborative robot, relative to the environment, and is updated in one embodiment.

Additionally or alternatively, in one embodiment the perspective of the view is determined on the basis of a determined, in particular current, viewing direction of a user, in particular wearer, of the user interface, updated in one embodiment, and relative in one embodiment on the basis of a determined viewing direction of the user for the user interface and a determined pose of the user interface relative to the environment or on the basis of a determined viewing direction of the user relative to the environment.

This perspective of the view is used in one embodiment when the view of the surroundings is superimposed with the virtual object (s), in particular the view is visualized in or with this perspective and / or the or one or more of the virtual object ( e) overlaying this perspective with the view of the surroundings, in particular in the manner described here or above and below.

For example, if a user wears a user interface in the form of a hand-held device and rotates around an axis with the hand-held device, in one embodiment the perspective of the view is (also) rotated accordingly.

For example, if a user wears a user interface in the form of an MR mask, MR lens, MR glasses or the like and rotates around an axis without eye movement, in one embodiment the perspective of the view is also rotated accordingly. If he moves his eyes without moving his head, for example horizontally from left to right, in one embodiment the perspective of the view also moves (with). With a combination of eye and user interface movement or a change in the viewing direction of the user and the pose of the user interface, the perspective of the view is also changed (determined) or updated accordingly in one embodiment. For example, if the user turns his head horizontally from left to right and his eyes from right to left at the same speed, the perspective of the view remains constant in one embodiment; for example, the user turns his head horizontally from left to right and his eyes also from left to right, In one embodiment, the perspective of the view moves with the superimposed speed of head and eye movement.

Thus, in one embodiment, the view of the environment and the virtual object (s) are augmented.

As a result, the required time and / or the effort for configuring, executing or analyzing applications of mobile and / or collaborative robots, in particular due to their complexity and / or variability, can be reduced particularly advantageously in one embodiment and / or the reliability achieved in particular can be advantageously increased.

In one embodiment, the view of the environment and the virtual object (s) are superimposed on the basis of a, in one embodiment, three-dimensional, in particular mathematical or data-technical, model of the environment. In one embodiment, a pose of the (respective) virtual object relative to or in the model corresponds to a determined pose of the surrounding object linked to this virtual object relative to the surroundings. The model can in particular have one or more points, in particular point cloud (s), groups of features localized relative to one another, 3D grids or the like.

In this way, the superimposition can be improved in one embodiment, and the precision and / or stability can be increased in one embodiment. For example, a 2D image that represents navigation of the mobile, in one embodiment collaborative, robot, in particular visualized, can be precisely (r) and / or stably (er) superimposed on the view of the surroundings.

Additionally or alternatively, in one embodiment the model is also used to navigate the robot arrangement, whereby in one embodiment the model can be used synergistically and / or the navigation can be improved, in one embodiment the precision and / or stability can be increased.

In one embodiment, the user interface has at least one, in one embodiment on the head, portable mixed reality-specific, in one embodiment AR-specific device such as Microsoft Hololens2, Magic Leap One or the like, in one embodiment MR glasses, MR lens or MR mask, in one version of an (MR) helmet or (MR) head strap.

As a result, the ergonomics and / or precision can be improved in one embodiment.

In one embodiment, the user interface has at least one portable multi-purpose device, in particular a mobile phone, tablet or the like, which in addition to the user interface functionality in one embodiment also has one or more other functionalities, in particular communication with the Internet and / or telephony, in particular video telephony such as Skype or the like has or provides.

As a result, an already existing user interface can take on several functions in one embodiment.

In one embodiment, the user interface has at least one transparent display, at least one screen, in particular a touchscreen, at least one trackpad, at least one keyboard, at least one joystick and / or at least one computer mouse.

As a result, the ergonomics and / or precision can be improved in one embodiment.

In one embodiment, the user interface detects the environment or is set up for this purpose or is used for this purpose with the aid of one or more sensors, which in one embodiment are on the user interface side or fixed.

In one embodiment, this can improve the representation of the environment and / or the self-localization of the user interface.

In one embodiment, the user interface detects the viewing direction of a wearer of the user interface with the aid of one or more sensors, which in one embodiment are on the user interface side or fixed, or is set up for this purpose or is used for this purpose.

In this way, the superimposition can be improved in one embodiment.

In one embodiment, the user interface sends, in one embodiment wirelessly, data to the robot arrangement, environment, a local network and / or a computer cloud and / or receives data wirelessly from the robot arrangement, environment, the local network and / or the computer cloud, in one embodiment , Data or is set up for this purpose or is used for this purpose.

As a result, in one embodiment, sensor and / or computing capacity external to the user interface can be used and thus in particular a sensor or computing capacity of the user interface can be spared or reduced and / or the user interface can function as a sensor.

In one embodiment, the user interface updates the overlay, in particular the view of the environment and / or the or one or more virtual object (s), in real time or is set up for this purpose or is used for this purpose.

As a result, in one embodiment, the implementation of the application can be improved, in particular the precision can be increased.

In one embodiment, the pose of the user interface and / or the pose of the robot arrangement, in particular of the mobile and / or collaborative robot, and / or the pose of the or one or more of the surrounding objects with which the or with which in each case one of the virtual object (s) is linked, determined by means of self-localization and / or with the help of an environment map, in particular by means of SLAM (Simultaneous Localization and Mapping).

In this way, the superimposition can be improved on the basis of the determined pose, in particular a precision can be increased.

In one embodiment, the pose of the user interface and / or the pose of the robot arrangement, in particular of the mobile and / or collaborative robot, and / or the pose of the or one or more of the surrounding objects with which the or with which in each case one of the virtual object (s) is linked, determined with the aid of at least one user interface-side and / or at least one robot arrangement-side, in particular robot-side and / or at least one environmental-side sensor. In one embodiment, the pose of the robot arrangement, in particular the mobile and / or collaborative robot, and / or the pose of the or one or more of the surrounding objects is, in addition or alternatively, based on a predetermined, in particular planned or controlled, positioning, in particular movement, the robot arrangement or the (respective) surrounding object is determined.

In one embodiment, the load on the user interface or robot arrangement or the mobile and / or collaborative robot can be reduced by means of sensors on the environment side; Environments are used.

As a sensor for determining the pose of the user interface, robot arrangement, in particular the mobile and / or collaborative robot, or the surrounding object, one or more cameras, in particular 2D and / or 3D cameras, one or more LIDAR devices (“Light Detection And Ranging”), one or more sonar sensors, one or more depth sensing devices, one or more radio sensors, in particular receivers of, in one embodiment, environmental-side and / or user interface or robot (arrangement) side, radio beacon, and / or one or more inertial measurement units (“Inertial Measurement Units”, IMUs), in one embodiment spatial combinations of several inertial sensors such as acceleration sensors and rotation rate sensors , used.

As a result, in one embodiment, the self-localization of the user interface can be improved, in particular a precision can be increased.

In one embodiment, the pose of the user interface and / or the pose of the robot arrangement, in particular of the mobile and / or collaborative robot, and / or the pose of the or one or more of the surrounding objects with which the or with which in each case one of the virtual object (s) is linked, determined in real time, in particular updated.

As a result, the superimposition on the basis of the determined pose can be improved, in particular a precision can be increased, and as a result in particular the execution and / or the analysis can be improved.

In one embodiment, the (current) pose of the user interface and / or the pose of the robot arrangement, in particular the mobile and / or collaborative robot, and / or the pose of the or one or more of the surrounding objects with which the or with each of which one of the virtual object (s) is linked, determined on the basis of a determined initial pose of the user interface, robot arrangement or the mobile and / or collaborative robot or (respective) surrounding object.

In one embodiment, this initial pose is, in particular, is based on an environmental feature, which in one embodiment is or is arranged and / or coded and / or detected by means of image processing, and / or with the aid of (coordinate) triangulation and / or at least one , in particular determined with the user interface and / or robot arrangement communicating, external measuring device.

For example, the initial pose can be detected using a QR code or another recognizable feature of the environment, which is detected by means of at least one user interface or robot (arrangement) -side sensor, with the known pose of the environmental feature relative to the environment and the sensed sensor Pose of the environmental feature relative to the user interface or robot arrangement, the (initial) pose of the user interface or robot arrangement relative to the surroundings can be determined.

As a result, in one embodiment, the self-localization of the user interface can be improved, in particular a precision can be increased.

In one embodiment, the user interface sends instructions to the robot arrangement, in particular the mobile and / or collaborative robot and / or a, in particular robot-external and / or central, robot arrangement controller, in one embodiment a fleet management system that controls or controls multiple mobile robots of the robot arrangement is set up or used for this purpose, in one embodiment the user interface sends movement instructions, instructions for changing a configuration of the robot arrangement, in particular of the mobile and / or collaborative robot, and / or requests for or requests for data.

As a result, in one embodiment, sensor and / or computing capacity external to the user interface can be used and thus in particular a sensor or computing capacity of the user interface can be spared or reduced and / or the user interface can function as a sensor and / or one or more of the mobile and / or control collaborative robots.

In one embodiment, the user interface receives data, in one embodiment navigation data, in particular map and / or path data, security data, configuration data and / or pose data, of the robot arrangement, in particular of the mobile and / or collaborative robot. In one embodiment, it receives the data from the robot arrangement, in particular from the mobile and / or collaborative robot and / or a robot arrangement controller, and / or stores such, in particular received, data. In one embodiment, and updated in one embodiment, the user interface determines the overlay on the basis of the stored or received data.

As a result, in one embodiment, sensor and / or computing capacity external to the user interface can be used and thus in particular a sensor or computing capacity of the user interface can be spared or reduced.

In one embodiment, at least one superimposed virtual object with map data of a navigation means of the robot arrangement, in particular the mobile, in one embodiment collaborative, Robot linked, represented in a development, in particular visualized, this virtual object the map data, for example an occupancy grid map or the like.

In one embodiment, at least one superimposed virtual object is linked to a sensor area, in particular a sensor detection direction, represented in a development, in particular visualized, this virtual object is the sensor area or the sensor detection direction, for example one or more beams from one or more scanners, in particular laser scanners, or similar.

In one embodiment, at least one superimposed virtual object is linked to a path or permitted path area of the robot arrangement, in particular the mobile and / or collaborative robot, which is planned and / or followed in an embodiment, this virtual object is represented in a further development, in particular visualized the path (area), for example in the form of a topometric or topological graph, Voronoi diagram, distance maps or the like.

In one embodiment, at least one superimposed virtual object is linked to a security area of the environment and / or the robot arrangement, in particular the mobile and / or collaborative robot, specified in one embodiment and / or two- or three-dimensional, one-part or multi-part, in a Further development represents, in particular visualized, this virtual object this security area, for example in the form of a one-part or multi-part floor area or a one-part or multi-part envelope contour.

In one embodiment, a surrounding area colliding with the robot-side safety area and / or a robot area colliding with the surrounding safety area is detected and, on the basis of the detection, a feature of the virtual object linked to this safety area is changed, for example (differently) colored or the like, and / or a own virtual object linked to this colliding or intersecting area, which represents it in a further development, in particular visualizes it.

A robot-side safety area is, in particular, is predefined or determined in one embodiment as a function of a dimension, current or planning pose and / or current, planned or extreme position of the robot arrangement, in particular of the mobile and / or collaborative robot Version updated.

In one embodiment, at least one superimposed virtual object is linked to an evaluation area of the environment and / or the robot arrangement, in particular the mobile and / or collaborative robot, specified in one embodiment and / or two- or three-dimensional, one-part or multi-part evaluation area, in one Further development represents, in particular visualized, this virtual object this evaluation area, for example in the form of a one-part or multi-part area, envelope contour or the like.

Such an evaluation area is, in particular, is specified in one version via the user interface and / or corresponds to a one-part or multi-part area of the environment that is or is or is primarily used for (self) localization, exclusively in one version . is. Correspondingly, in particular, a one-part or multi-part surrounding area specified via the user interface that is not intended to be used for (self) localization is also referred to as an evaluation area.

In one embodiment, at least one superimposed virtual object is linked in one embodiment to a predetermined and / or two- or three-dimensional one-part or multi-part reachability area of the environment and / or the robot arrangement, in particular the mobile and / or collaborative robot Further development represents, in particular visualized, this virtual object this accessibility area, for example in the form of a one-part or multi-part area, envelope contour or the like.

Such an accessibility area is, in particular, is determined in one embodiment by means of at least partially automated detection of the surroundings, in particular obstacles, is updated in one embodiment, and / or corresponds to a one-part or multi-part area of the environment that is controlled by the robot arrangement, in particular the mobile and / or collaborative robots.

In one embodiment, at least one superimposed virtual object is linked in one embodiment with a reservation area of the environment and / or the robot arrangement, in particular the mobile and / or collaborative robot, predetermined and / or two- or three-dimensional, one-part or multi-part Further development represents, in particular visualized, this virtual object this reservation area, for example in the form of a one-part or multi-part floor area or a one-part or multi-part envelope contour.

In one embodiment, such a reservation area is, in particular, depending on a dimension, current or planning pose and / or current, planned or extreme position of the robot arrangement, in particular the mobile and / or collaborative robot, specified or determined, updated in an embodiment, and / or represents or visualizes a one-part or multi-part area that the robot arrangement, in particular the mobile and / or collaborative robot, potentially, in particular when traveling through a planned path and / or performing a planned task and / or in an extreme display.

In one embodiment, a robot-side security or reservation area projects beyond an outer contour of the mobile and / or collaborative robot. As a result, in one embodiment, its determination can be simplified and / or security can be increased. In one embodiment, a safety area on the surrounding area is an area into which the robot must not or should not penetrate; In one embodiment, a (robot-side) reservation area is an area that is or should be reserved for the robot and / or in which the robot is allowed to stay and / or move.

In one embodiment, at least one superimposed virtual object is linked to a target location or a target pose, in particular the robot arrangement, in an embodiment of the mobile and / or collaborative robot, represented in a further development, in particular visualized, this virtual object is such a target location or such a target position, for example a (current and / or planned) loading and / or unloading station, conveying, loading or processing, testing and / or refueling and / or electrical (charging) charging station or the like.

In one embodiment, at least one superimposed virtual object is linked to a waiting location or a waiting pose, in particular the robot arrangement, in an embodiment of the mobile and / or collaborative robot, and in a further development is represented, in particular visualized, this virtual object is such a waiting location or a waiting position. Such a waiting position, for example a (current and / or planned) position to be approached, at which it is provided that the robot waits for a certain time and / or for a certain result, for example a release or the like.

In one embodiment, at least one superimposed virtual object is linked to a waiting time, planned or remaining in an embodiment, for a mobile and / or collaborative robot, in an embodiment at a waiting location or a waiting position of the robot, in a Further development represents, in particular visualized, this virtual object this waiting time.

In one embodiment there is at least one superimposed virtual object with an arrival time planned or forecast in one embodiment, the robot arrangement, in one embodiment the mobile and / or collaborative robot, in one embodiment at a waiting or destination location or a waiting or Target pose of the robot arrangement or the robot, linked, represented in a further development, in particular visualized, this virtual object this arrival time.

In one embodiment, at least one superimposed virtual object is linked to the mobile and / or collaborative robot, in one embodiment with an energy, navigation, error, communication and / or logistics status of the robot, represented in a development, in particular visualized , this virtual object the robot or state, for example a charge level of a battery, an accuracy of a (self) localization, current errors on the robot, a reliability of a communication link or its current position in a fleet management administration.

In one embodiment, at least one superimposed virtual object with an, in particular stationary or moving, surrounding object or point, in particular a state and / or a potential, in particular planned, interaction (of the surrounding object or (at the) surrounding point (s)) is included the robot arrangement, linked, represented in a further development, in particular visualized, this virtual object this environmental object or this environmental point, in particular its state and / or the potential, in particular planned, interaction with the robot arrangement, for example a degree of filling and / or equipment and / or readiness state of a loading and / or unloading station, conveying, loading or processing, testing and / or refueling and / or electric (charging) charging station - or parking station, a task that the mobile and / or collaborative Robots can or should perform at a station or the like.

In one embodiment, at least one superimposed virtual object is linked with a navigation aid to a predetermined point, in a further development this virtual object is represented, in particular visualized, this navigation aid. In one embodiment, the virtual object has one or more (directional) arrows, distance information, paths, in particular currently traveled, already traveled and / or possible, in particular planned paths, and / or location identifications.

In one embodiment, one or more of the virtual objects mentioned here or above and below are (in each case) with a, in particular stationary or moving, surrounding object or point, in particular in such a way that their pose, in particular position, remains constant relative to a view of this surrounding object or point in the view of the surroundings. In one embodiment, its orientation also remains constant relative to the viewing direction of the view; in particular, a virtual object in the form of a flat surface can always be oriented such that the viewing direction of the view is parallel to the surface normal of this surface.

As a result, the ergonomics, reliability, speed and / or precision of the configuration, implementation or analysis of the application can be improved in one embodiment.

In one embodiment, a position of at least one superimposed virtual object relative to an outer contour of the view of the environment remains constant when the user interface is moved (relative to the environment) and / or the perspective of the view of the environment changes, in particular on a predetermined one in an embodiment User input selectable, in particular changeable, position, for example at an image edge or the like.

Additionally or alternatively, in one embodiment an orientation of at least one superimposed virtual object relative to an outer contour of the view of the environment remains constant when the user interface is moved (relative to the environment) and / or the perspective of the view of the environment changes, in particular in a predetermined, in an execution by user input selectable, in particular changeable, orientation, for example parallel to an image plane or the like.

In one embodiment, a position and / or orientation of at least one superimposed virtual object relative to a view, in particular an image, of a surrounding object or surrounding point or path point or path section of the robot arrangement linked to this virtual object remains in the overlay, in one embodiment in the view of the Environment, in particular in the representation of the environment or the user's field of vision, constant when the user interface moves and / or the perspective of the view of the environment changes. In one embodiment, a change in the perspective of the view of the surroundings, in particular as a result of a movement of the user interface, is compensated for or the position and / or orientation of the virtual object is correspondingly adjusted relative to an outer contour of the view of the surroundings.

• - Umgebungsobjekts,
• - Umgebungspunktes oder
• - Punktes oder Abschnitts eines, insbesondere geplanten oder abgefahrenen, Pfades der Roboteranordnung, insbesondere des mobilen und/oder kollaborativen Roboters,

relativ zur Umgebung ermittelt, insbesondere aktualisiert, in einer Ausführung in Echtzeit und/oder derart ermittelt bzw. aktualisiert, dass

• - sich die Position des (jeweiligen) virtuellen Objekts in der Überlagerung bzw. Ansicht der Umgebung mit bzw. bei einer Änderung der Position des damit verknüpften Umgebungsobjekts bzw. Pfadpunktes bzw. -abschnitts relativ zu der Umgebung (entsprechend) (mit)ändert bzw.
• - die Position des (jeweiligen) virtuellen Objekts relativ zu der Ansicht des Umgebungspunkts in der Überlagerung bzw. Ansicht der Umgebung konstant bleibt bzw.
• - die Position des (jeweiligen) virtuellen Objekts relativ zu der Ansicht des Pfadpunkts bzw. -abschnitts in der Überlagerung bzw. Ansicht der Umgebung konstant bleibt.

In one embodiment is, in particular, a position of the or one or more of the virtual objects in the overlay, in one embodiment in the view of the surroundings, in particular in the representation of the surroundings or the field of view of the user, based on the position of a linked to the (respective) virtual object

• - surrounding object,
• - Surrounding point or
• - Point or section of a, in particular planned or traveled, path of the robot arrangement, in particular of the mobile and / or collaborative robot,

determined relative to the environment, in particular updated, determined or updated in an execution in real time and / or in such a way that

• - the position of the (respective) virtual object in the overlay or view of the environment changes with or when the position of the associated environment object or path point or section relative to the environment (correspondingly) changes or
• - the position of the (respective) virtual object relative to the view of the environment point in the overlay or view of the environment remains constant or
• - the position of the (respective) virtual object relative to the view of the path point or section in the superimposition or view of the surroundings remains constant.

In one embodiment, an orientation of this virtual object relative to an outer contour of the view of the environment remains constant when the user interface is moved (relative to the environment) and / or the perspective of the view of the environment changes.

As a result, the ergonomics can be improved in one embodiment.

In one embodiment, the user interface records a user input with the aid of eye recording, gesture recognition, speech recognition, recognition of virtual contact, in particular virtual contact, in one embodiment hand contact, of a user with a virtual object.

As a result, the ergonomics, reliability, speed and / or precision of the configuration, implementation or analysis of the application can be improved in one embodiment.

In one embodiment, at least one superimposed virtual object and / or at least one feature of at least one superimposed virtual object is faded in and / or faded out and / or changed on the basis of a user input, in one embodiment a pose, in particular a position and / or orientation of the virtual object or feature relative to an outer contour of the view of the environment and / or at least one parameter of the virtual object or feature.

In particular, in an embodiment, additional features are shown and / or hidden in an embodiment by means of corresponding user inputs, in (sub) menus.

In one embodiment, navigation data, in particular maps, graphs or the like, and / or the or one or more of the virtual objects, in particular their pose in the overlay, can be changed, in particular edited or changed by a user input via the user interface. changed, especially edited. In one embodiment, one or more planned tasks and / or points, in particular sections, of a planned path of the robot arrangement, in particular of the mobile and / or collaborative robot, are generated and / or changed by a user input via the user interface. in particular editable or generated and / or changed, in particular edited.

For example, a user can remove an obstacle that no longer exists from a (navigation) map by means of appropriate user input, change or add a potential, in particular possible or planned, path for the mobile and / or collaborative robot, or a virtual one Move the object in the overlay and / or link it to a (different) surrounding object and / or (re) parameterize it.

As already mentioned, in one embodiment the robot arrangement, in particular the mobile and / or collaborative robot, can be controlled and / or (re) configured by a user input via the user interface.

• Mittel zum Überlagern einer Ansicht, insbesondere Darstellung, einer Umgebung mit wenigstens einem virtuellen Objekt, wobei die Benutzerschnittstelle dazu eingerichtet ist, diese Überlagerung zum Konfigurieren, Durchführen und/oder Analysieren der Applikation, in einer Ausführung an der Benutzerschnittstelle, zur Verfügung zu stellen, in einer Ausführung die Ansicht der Umgebung und das virtuelle Objekt zu überlagern und/oder die Applikation auf Basis dieser Überlagerung zu konfigurieren, durchzuführen und/oder zu analysieren.

According to one embodiment of the present invention, a system for configuring, performing and / or analyzing an application of a robot arrangement with at least one mobile and / or collaborative robot using a user interface for inputting and / or outputting data, in particular hardware and / or software - is set up, in particular in terms of programming, to carry out a method described here and / or has:

• Means for overlaying a view, in particular a representation, of an environment with at least one virtual object, the user interface being set up to provide this overlay for configuring, executing and / or analyzing the application, in one embodiment at the user interface, in one execution to superimpose the view of the environment and the virtual object and / or to configure, carry out and / or analyze the application on the basis of this superimposition.

• Mittel zum Überlagern der Ansicht der Umgebung und des virtuellen Objekts auf Basis einer ermittelten, insbesondere aktuellen, Perspektive der Ansicht; auf Basis einer ermittelten, insbesondere aktuellen, Pose der Benutzerschnittstelle, der Roboteranordnung, insbesondere des mobilen und/oder kollaborativen Roboters, und/oder wenigstens eines, insbesondere statischen oder bewegten, Umgebungsobjekts, insbesondere eines mit dem virtuellen Objekt verknüpften Umgebungsobjekts, relativ zu der Umgebung; auf Basis einer ermittelten, insbesondere aktuellen, Blickrichtung eines Trägers der Benutzerschnittstelle und/oder auf Basis eines, insbesondere dreidimensionalen, Modells der Umgebung; und/oder
• Mittel zum Erfassen der Umgebung und/oder Blickrichtung, insbesondere mithilfe wenigstens eines, insbesondere benutzerschnittstellenseitigen, Sensors; und/oder
• Mittel zum Senden von Daten an die Roboteranordnung, Umgebung, ein lokales Netzwerk und/oder eine Computercloud (S30) und/oder Erhalten von Daten der Roboteranordnung, Umgebung, dem lokalen Netzwerk und/oder der Computercloud und/oder zum Aktualisieren der Überlagerung, insbesondere der Ansicht der Umgebung und/oder des virtuellen Objekts, in Echtzeit; und/oder
• Mittel zum Ermitteln der Pose der Benutzerschnittstelle, der Roboteranordnung, insbesondere des mobilen und/oder kollaborativen Roboters, und/oder des Umgebungsobjekts mittels Selbstlokalisation und/oder mithilfe einer Umgebungskarte, insbesondere mittels SLAM, und/oder mithilfe wenigstens eines benutzerschnittstellen-, roboteranordnungs- oder umgebungsseitigen Sensors, insbesondere wenigstens einer inertialen Messeinheit, wenigstens einer Kamera, wenigstens einer LIDAR-Vorrichtung, wenigstens eines Sonarsensors, wenigstens eines Tiefenwahrnehmungssensors und/oder wenigstens eines Funksensors, und/oder in Echtzeit und/oder auf Basis einer ermittelten Anfangspose; und/oder
• Mittel zum Ermitteln der Anfangspose auf Basis eines, insbesondere mittels Bildverarbeitung, erfassten, insbesondere codierten und/oder hierzu angeordneten, Umgebungsmerkmals und/oder mithilfe von Triangulation und/oder wenigstens einer, insbesondere mit der Benutzerschnittstelle und/oder Roboteranordnung kommunizierenden, externen Messvorrichtung; und/oder
• Mittel zum Senden von Anweisungen von der Benutzerschnittstelle an die Roboteranordnung, insbesondere den mobilen und/oder kollaborativen Roboter und/oder eine Roboteranordnungssteuerung, und/oder Speichern und/oder Empfangen von Daten, insbesondere Navigations-, Sicherheits-, Konfigurations- und/oder Posendaten, der Roboteranordnung, insbesondere von dem mobilen und/oder kollaborativen Roboter und/oder einer Roboteranordnungssteuerung, und Ermitteln, insbesondere Aktualisieren der Überlagerung auf Basis der gespeicherten bzw. empfangenen Daten; und/oder Mittel zum Verknüpfen wenigstens eines überlagerten virtuellen Objekts mit Kartendaten eines Navigationsmittels der Roboteranordnung, insbesondere des mobilen und/oder kollaborativen Roboters, mit einem Sensorbereich, insbesondere einer Sensorerfassungsrichtung, mit einem, insbesondere geplanten und/oder abgefahrenen, Pfad oder zulässigen Pfadbereich der Roboteranordnung, insbesondere des mobilen und/oder kollaborativen Roboters, mit einem, insbesondere vorgegebenen und/oder zwei- oder dreidimensionalen, Sicherheits-, Auswertungs-, Erreichbarkeits- und/oder Reservierungsbereich der Umgebung und/oder der Roboteranordnung, insbesondere des mobilen und/oder kollaborativen Roboters, mit einem Warte- oder Zielort, mit einer Warte- oder Zielpose, mit einer Warte- oder Ankunftszeit, mit dem mobilen und/oder kollaborativen Roboter, insbesondere einem Energie-, Navigations-, Fehler-, Kommunikations- und/oder Logistikzustand, mit einem Umgebungsobjekt oder -punkt, insbesondere einem Zustand und/oder einer potentiellen, insbesondere geplanten, Interaktion mit der Roboteranordnung, und/oder mit einer Navigationshilfe zu einer vorgegebenen Stelle; und/oder
• Mittel zum Erfassen einer Benutzereingabe mit der Benutzerschnittstelle mithilfe von Blickerfassung, Gestenerkennung, Spracherkennung, Erkennung von virtuellem Kontakt; und/oder
• Mittel zum Verändern von Navigationsdaten und/oder dem virtuellen Objekt, insbesondere dessen Pose in der Überlagerung, durch eine Benutzereingabe über die Benutzerschnittstelle; und/oder
• Mittel zum Ein- und/oder Ausblenden und/oder Verändern wenigstens eines überlagerten virtuellen Objekts und/oder wenigstens eines Merkmals wenigstens eines überlagerten virtuellen Objekts auf Basis einer Benutzereingabe, insbesondere seiner Pose relativ zu einer Außenkontur der Ansicht der Umwelt und/oder wenigstens eines Parameters des virtuellen Objekts bzw. Merkmals.

In one embodiment, the system or its means has:

• Means for superimposing the view of the surroundings and the virtual object on the basis of a determined, in particular current, perspective of the view; based on a determined, in particular current, pose of the user interface, the robot arrangement, in particular the mobile and / or collaborative robot, and / or at least one, in particular static or moving, environmental object, in particular an environmental object linked to the virtual object, relative to the environment ; on the basis of a determined, in particular current, viewing direction of a wearer of the user interface and / or on the basis of a, in particular three-dimensional, model of the environment; and or
• Means for detecting the surroundings and / or viewing direction, in particular with the aid of at least one sensor, in particular on the user interface side; and or
• Means for sending data to the robot arrangement, environment, a local network and / or a computer cloud (S30) and / or receiving data from the robot arrangement, environment, the local network and / or the computer cloud and / or for updating the overlay, in particular the view of the environment and / or the virtual object, in real time; and or
• Means for determining the pose of the user interface, the robot arrangement, in particular the mobile and / or collaborative robot, and / or the surrounding object by means of self-localization and / or with the aid of an environment map, in particular by means of SLAM, and / or with the aid of at least one user interface, robot arrangement or environmental sensor, in particular at least one inertial measuring unit, at least one camera, at least one LIDAR device, at least one sonar sensor, at least one depth perception sensor and / or at least one radio sensor, and / or in real time and / or on the basis of a determined initial pose; and or
• Means for determining the initial pose on the basis of an environmental feature that is detected, in particular encoded and / or arranged for this purpose, in particular by means of image processing, and / or with the aid of triangulation and / or at least an external measuring device communicating in particular with the user interface and / or robot arrangement; and or
• Means for sending instructions from the user interface to the robot arrangement, in particular the mobile and / or collaborative robot and / or a robot arrangement controller, and / or storing and / or receiving data, in particular navigation, safety, configuration and / or pose data , the robot arrangement, in particular from the mobile and / or collaborative robot and / or a robot arrangement controller, and determining, in particular updating, the overlay on the basis of the stored or received data; and / or means for linking at least one superimposed virtual object with map data of a navigation means of the robot arrangement, in particular of the mobile and / or collaborative robot, with a sensor area, in particular a sensor detection direction, with a, in particular planned and / or traveled, path or permissible path area of the Robot arrangement, in particular the mobile and / or collaborative robot, with a, in particular predetermined and / or two- or three-dimensional, security, evaluation, accessibility and / or reservation area of the environment and / or the robot arrangement, in particular the mobile and / or collaborative robot, with a waiting or destination, with a waiting or target pose, with a waiting or arrival time, with the mobile and / or collaborative robot, in particular an energy, navigation, error, communication and / or logistics status , with a surrounding object or point, in particular an add situation and / or a potential, in particular planned, interaction with the robot arrangement and / or with a navigation aid to a predetermined point; and or
• Means for capturing user input with the user interface using eye tracking, gesture recognition, speech recognition, virtual contact recognition; and or
• Means for changing navigation data and / or the virtual object, in particular its pose in the overlay, by a user input via the user interface; and or
• Means for showing and / or hiding and / or changing at least one superimposed virtual object and / or at least one feature of at least one superimposed virtual object based on a user input, in particular its pose relative to an outer contour of the view of the environment and / or at least one parameter of the virtual object or feature.

A means within the meaning of the present invention can be designed in terms of hardware and / or software, in particular a data or signal-connected, in particular digital, processing, in particular microprocessor unit (CPU), graphics card (GPU), preferably with a memory and / or bus system ) or the like, and / or one or more programs or program modules. The processing unit can be designed to process commands that are implemented as a program stored in a memory system, to acquire input signals from a data bus and / or to output output signals to a data bus. A storage system can have one or more, in particular different, storage media, in particular optical, magnetic, solid-state and / or other non-volatile media. The program can be designed in such a way that it embodies or is able to execute the methods described here, so that the processing unit can execute the steps of such methods and thus in particular configure, execute and / or analyze the application of the robot arrangement. In one embodiment, a computer program product can have, in particular, a non-volatile storage medium for storing a program or with a program stored on it To carry out a process or one or more of its steps.

In one embodiment, one or more, in particular all, steps of the method are carried out completely or partially in an automated manner, in particular by the system or its means.

In one embodiment, the system has the robot arrangement. If we are talking about a mobile and / or collaborative robot, in particular a mobile, possibly collaborative, robot or a collaborative, possibly stationary, robot or a mobile, possibly collaborative, robot and additionally a collaborative, possibly stationary one , Robot means or includes the expression “mobile and / or collaborative robot” “mobile robot”, “collaborative robot” and “(at least) one mobile and (at least) one (additional) collaborative robot”.

In one embodiment, an environment within the meaning of the present invention is a real one or physical environment, an environmental object corresponding to a real or physical, in particular a material one-part or multi-part object (this environment), in particular external to the robot arrangement, so in particular not a mobile and / or collaborative robot.

A path or a path in the sense of the present invention can in particular comprise, in particular be, a route, in particular a driving (t) route, and / or a trajectory.

In one embodiment, a user of the user interface is a human user or a natural person.

“Planned” can mean “first planned” or “only planned”. For example, “planned path” can mean in particular “planned, not yet traveled path” and “planned pose” can mean “planned, not current pose”.

In one embodiment, the user interface determines the superimposition, in particular arithmetically. Additionally or alternatively, the user interface visualizes the superimposition in one embodiment, in particular on at least one screen and / or by projection. The overlay can also be determined wholly or partially external to the user interface and (only) visualized by the user interface. Both d. H. Both the determination and the visualization are referred to here as superimposing the view of the environment and the virtual object by the, in particular at the, user interface.

• 1: ein System nach einer Ausführung der vorliegenden Erfindung;
• 2: ein Verfahren nach einer Ausführung der vorliegenden Erfindung;
• 3: ein Überlagern einer Ansicht einer Umgebung mit mehreren virtuellen Objekten nach einer Ausführung der vorliegenden Erfindung;
• 4: eine Benutzereingabe über eine Benutzerschnittstelle nach einer Ausführung der vorliegenden Erfindung;
• 5: ein Überlagern einer Ansicht einer Umgebung mit virtuellen Objekten nach einer Ausführung der vorliegenden Erfindung; und
• 6: ein Überlagern einer Ansicht einer Umgebung mit virtuellen Objekten nach einer Ausführung der vorliegenden Erfindung.

Further advantages and features emerge from the subclaims and the exemplary embodiments. For this shows, partly schematically:

• 1 : a system according to an embodiment of the present invention;
• 2 : a method according to an embodiment of the present invention;
• 3 : Overlaying a view of an environment with a plurality of virtual objects according to an embodiment of the present invention;
• 4th : a user input via a user interface according to an embodiment of the present invention;
• 5 : an overlay of a view of an environment with virtual objects according to an embodiment of the present invention; and
• 6th Figure 12 is an overlay of a view of a virtual object environment according to an embodiment of the present invention.

1 , 2 show a system or method for configuring, performing and / or analyzing an application of a robot arrangement with at least one mobile and / or collaborative robot 1 using a user interface 2 , 2 ' for inputting and outputting data according to an embodiment of the present invention.

The robot can have a robot arm indicated by dashed lines.

One user 3 the user interface, in one embodiment an MR mask or glasses 2, in another embodiment a correspondingly equipped and set up mobile radio device or tablet 2 ' . Both versions are shown together for a more compact representation and can be provided or used individually or together. The user can also use an MR lens or the like.

The user interface 2 respectively. 2 ' has one or more sensors S. for environmental detection, in particular self-localization, and is used for entering user inputs, for example by means of eye tracking, gesture recognition, speech recognition, recognition of virtual contact, touchscreen, trackpad, keyboard, joystick, computer mouse or the like, as well as for sending and receiving data to and from the mobile and / or collaborative robot 1 and / or a robot-external robot arrangement controller 1.1 equipped and furnished.

An environment object is exemplified in an environment 4.1 as well as an environmentally safe sensor 4.2 indicated.

After the in 2 The method shown is a pose of the user interface in a step S10 2 respectively. 2 ' relative to the surroundings and / or a line of sight of the user 3 determined relative to the environment or user interface. Additionally or alternatively, a pose of the mobile and / or collaborative robot can be performed in step S10 1 and / or a pose of the, possibly moving, surrounding object 4.1 be determined.

In a step S20, the user interface is superimposed 2 respectively. 2 ' a virtual object that is in an execution with the environment object 4.1 or a point of the environment is linked to a view of the environment, for example a camera image, an artificial reconstruction of the environment or a view of the user 3 on the environment.

In a step S30, the mobile and / or collaborative robot is activated, for example, in particular by or on the basis of corresponding user input (s) 1 controlled and / or the virtual Object changes, for example its position relative to an outer contour of the view and / or its content and / or one or more of its displayed features. In one embodiment, a planned path of the mobile and / or collaborative robot is established, in particular by or on the basis of corresponding user inputs 1 and / or interactions of the mobile and / or collaborative robot 1 with the environment, for example the surrounding object 4.1 , generated or changed, in particular adding, moving and / or deleting path points or sections and / or actions of the robot 1 and / or surrounding object 4.1 edited, for example a number and / or identification of goods to be unloaded or picked up or the like.

The method then returns to step S10 in order to update the pose (s).

3 FIG. 12 shows such a superimposition of a view of an environment with a plurality of virtual objects according to an embodiment of the present invention.

In 3 Fig. 3 is a view of an environment in the form of a camera image 10 superimposed with virtual objects, in the exemplary embodiment a virtual object 110, which has a navigation graph or path for the mobile and / or collaborative robot 1 represents, a virtual object 120 , which represents a node of the graph or a corresponding point in the vicinity or an arrival time, and a virtual object 130 , which represents another node of the graph or a corresponding other point in the vicinity, in particular its identity.

4th or the sequence of figures 3 ↔ 4th illustrates user input by touching the virtual objects 120 , 130 , 140 or corresponding virtual buttons such as "Open details", "Hide" etc. For example, by touching the virtual object 130 this is hidden (cf. 4th ) or additional features or details can be displayed by touching its virtual button "Open details".

The two virtual objects 120 , 130 are linked to the corresponding environment or path point, so that their position when the (perspective of the) view changes 10 , for example by rotating the user interface 2 ' or rotating the head-worn user interface 2 and / or eye movement, in the view relative to the surroundings, remain stationary, ie above the corresponding node, and thus their position changes relative to an outer contour of the view. Their orientation relative to the outer contour of the view remains constant.

The opposite is the case with the virtual object 140 in 4th anchored in a fixed position relative to the outer contour of the view in an upper right corner and maintains this position relative to the outer contour of the view even if the (perspective of the) view is different 10 by rotating the user as described above 3 changes.

5 shows an overlay of a camera image 20th with a virtual object 150 , which represents an unobstructed accessibility area or a permissible path area in the form of a monochrome virtual floor area, as well as virtual objects in the form of laser scanner beams 160 .

6th shows an overlay of a camera image 30th with a virtual object 170 , which represents a reservation area in the form of a single-colored virtual floor area, two virtual objects 180.1, 180.2, which represent different (large) security areas in the form of single-colored virtual floor areas, a virtual object 190 , which in the form described above is a navigation graph or path for the mobile and / or collaborative robot 1 represented by the in 6th a node with the identification number ("Node ID") 8 is visible, and a virtual object 200 , which is anchored in a stationary manner relative to the outer contour of the view in an upper right corner and has features in the form of virtual input buttons.

A surrounding area that collides with the inner security area is automatically recorded and by its own virtual object 210 , which is linked to this colliding or cutting area, as well as visualized by coloring the virtual object 180.1.

In 6th is also indicated how the user 3 edit the node or path via the touchscreen, especially move its pose in the area. The user interface 2 respectively. 2 ' transmits the corresponding (change) data to the path planning of the robot arrangement controller 1.1 .

Although exemplary embodiments have been explained in the preceding description, it should be pointed out that a large number of modifications are possible. It should also be pointed out that the exemplary designs are only examples that are not intended to restrict the scope of protection, the applications and the structure in any way. Rather, the preceding description provides a person skilled in the art with guidelines for the implementation of at least one exemplary embodiment, with various changes, in particular with regard to the function and arrangement of the components described, being able to be made without departing from the scope of protection as it emerges from the claims and these equivalent combinations of features results.

List of reference symbols

1

mobile and / or collaborative robot

1.1

Robot arrangement control

2

MR mask / glasses

2 '

Tablet

3

user

4.1

Environment object

4.2

sensor

10-30

Camera image (view of the surroundings)

110-210

virtual object

S.

User interface sensor

Claims (14)

Method for configuring, performing and / or analyzing an application of a robot arrangement (1, 1.1) with at least one mobile and / or collaborative robot (1), with the aid of a user interface (2; 2 ') for inputting and / or outputting data, the method comprising the step of: superimposing (S20) a view, in particular a representation (10; 20; 30), of an environment with at least one virtual object (110-210); this overlay for configuring, executing and / or analyzing the application is made available by the, in particular on the, user interface, in particular the view of the environment and the virtual object by which, in particular on the, user interface is overlaid and / or the application is configured, performed and / or analyzed on the basis of this superimposition, in particular with the aid of the user interface (S30). Procedure according to Claim 1 , characterized in that the view of the environment and the virtual object based on a determined, in particular current, perspective of the view, a determined, in particular current, pose of the user interface, the robot arrangement, in particular the mobile and / or collaborative robot, and / or at least one, in particular static or moving, environmental object, in particular an environmental object linked to the virtual object, relative to the environment, on the basis of a determined, in particular current, viewing direction of a wearer (3) of the user interface and / or on the basis of a, in particular three-dimensional, Model of the environment are superimposed. Method according to one of the preceding claims, characterized in that the user interface has at least one portable mixed-reality-specific and / or at least one portable multi-purpose device and / or at least one transparent display, at least one screen, in particular a touchscreen, at least one trackpad, at least one keyboard , at least one joystick and / or at least one computer mouse, and / or, in particular with the help of at least one, in particular user interface-side sensor, captures the environment and / or viewing direction and / or data to the robot arrangement, environment, a local network and / or a Computer cloud sends (S30) and / or receives from the robot arrangement, environment, the local network and / or the computer cloud and / or the overlay, in particular the view of the environment and / or the virtual object, is updated in real time. Method according to one of the preceding claims, characterized in that the pose of the user interface, the robot arrangement, in particular the mobile and / or collaborative robot, and / or the surrounding object by means of self-localization and / or with the help of an environment map, in particular by means of SLAM, and / or with the help of at least one user interface, robot arrangement or environment-side sensor (4.2), in particular at least one inertial measuring unit, at least one camera, at least one LIDAR device, at least one sonar sensor, at least one depth perception sensor and / or at least one radio sensor, and / or in real time and / or is determined on the basis of a determined initial pose. The method according to the preceding claim, characterized in that the initial pose is based on an, in particular by means of image processing, detected, in particular coded and / or arranged for this, environmental feature and / or with the help of triangulation and / or at least one, in particular with the user interface and / or Robot arrangement communicating, external measuring device is determined. Method according to one of the preceding claims, characterized in that the user interface sends instructions to the robot arrangement, in particular the mobile and / or collaborative robot and / or a robot arrangement controller (S30) and / or data, in particular navigation, safety, configuration and / or pose data that the robot arrangement stores and / or receives, in particular from the mobile and / or collaborative robot and / or a robot arrangement controller, and the overlay is determined, in particular updated, on the basis of the stored or received data. Method according to one of the preceding claims, characterized in that at least one superimposed virtual object with map data of a navigation means of the robot arrangement, in particular the mobile, in one embodiment collaborative, robot, with a sensor area, in particular a sensor detection direction, with a, in particular planned and / or traveled, path or permissible path area of the robot arrangement, in particular of the mobile and / or collaborative robot, with a, in particular predetermined and / or two- or three-dimensional, safety, evaluation, accessibility and / or reservation area of the environment and / or the robot arrangement , in particular the mobile and / or collaborative robot, with a waiting or destination, with a waiting or target pose, with a waiting or arrival time, with the mobile and / or collaborative robot, in particular an energy, navigation, error , Communication and / or logistics status, m it is linked to a surrounding object or point, in particular a state and / or a potential, in particular planned, interaction with the robot arrangement and / or with a navigation aid to a predetermined point. Method according to one of the preceding claims, characterized in that a position and / or orientation of at least one superimposed virtual object relative to an outer contour of the view of the environment remains constant when the user interface is moved and / or the perspective of the view of the environment changes. Method according to one of the preceding claims, characterized in that a position and / or orientation of at least one superimposed virtual object relative to a view of a surrounding object or point or path point or section of the robot arrangement linked to this virtual object in the overlay when the User interface and / or change in perspective of the view of the environment remains constant. Method according to one of the preceding claims, characterized in that the user interface detects a user input with the aid of gaze detection, gesture recognition, speech recognition and / or detection of virtual contact. Method according to one of the preceding claims, characterized in that navigation data and / or the virtual object, in particular its pose in the overlay, can be changed (S30) by a user input via the user interface. Method according to one of the preceding claims, characterized in that at least one superimposed virtual object and / or at least one feature of at least one superimposed virtual object is shown and / or hidden and / or changed on the basis of a user input, in particular its pose relative to an outer contour of the View of the environment and / or at least one parameter of the virtual object or feature. System for configuring, performing and / or analyzing an application of a robot arrangement (1, 1.1) with at least one mobile and / or collaborative robot (1), with the aid of a user interface (2; 2 ') for inputting and / or outputting data, wherein the system is set up to carry out a method according to one of the preceding claims and / or has: Means for superimposing a view, in particular a representation (10; 20; 30), of an environment with at least one virtual object (110-210); The user interface is set up to provide this overlay for configuring, executing and / or analyzing the application, in particular at the user interface, in particular to overlay the view of the environment and the virtual object and / or the application based on this overlay configure, perform and / or analyze. Computer program product with a program code, which is stored on a medium readable by a computer, for carrying out a method according to one of the preceding claims.

Images