DE202021106457U1 - system - Google Patents

Abstract

System (1) for planning an application with at least one marking (2), with a control and evaluation unit (3), with a database (4), with a display unit (5), and with at least one spatial model of a real environment, and at least one camera (7) for imaging the real environment (8),
wherein the real environment (8) in the spatial model can be displayed as a virtual environment (9) on the display unit (5),
wherein the marking (2) is arranged in the real environment (8) at a position and with an orientation,
the position and the orientation of the marking (2) being detectable at least by the camera (7),
the position and orientation of the marking (2) being linked to a virtual sensor model (10),
wherein the virtual sensor model (10) can be displayed in the spatial model of the virtual environment (9) on the display unit at the position and with the orientation of the marking (2),
wherein the virtual sensor model (10) has at least one virtual detection area (11), wherein the virtual detection area (11) can be displayed in the spatial model of the virtual environment (9) on the display unit (5).

Description

The present invention relates to a system according to the preamble of claim 1.

Applications or applications, for example in human-robot collaboration (MRK), are often difficult to plan. This applies not only to the robot path itself, but in particular to the integration of sensors. These sensors can be both automation and security sensors. These are always characterized by their effective range, for example the FOV of a camera or the range of a scanner, and by their function, for example a protective field that is formed. In an HRC application, it is often crucial where a sensor is attached and how it is embedded in the application context. The latter is often a trial and error procedure, which is time-consuming and therefore expensive. This step is often only feasible with the real sensors, i.e. during the integration of the application.

The object is achieved according to claim 1 by a system for planning an application with at least one marking with a control and evaluation unit, with a database, with a display unit, and with at least one spatial model of a real environment, and at least one camera for imaging the real environment, wherein the real environment in the spatial model can be displayed as a virtual environment on the display unit, the marking being arranged in a position and with an orientation in the real environment, the position and the orientation of the marking being detectable at least by the camera are, the position and orientation of the marking being linked to a virtual sensor model, the virtual sensor model being able to be displayed in the spatial model of the virtual environment on the display unit at the position and with the orientation of the marking, the virtual sensor model having at least one virtual detection area has, wherein the virtual detection area can be displayed in the spatial model of the virtual environment on the display unit.

For example, a set of N markings or markers or so-called visual tags is provided. The marker or markers are placed or arranged in an environment of a planned use or application. For example, fixed to an infrastructure, e.g. on a wall or at a workplace. For example, the position is a fixed position and the orientation is a fixed orientation.

For example, the markings are placed on manipulators, e.g. on a robot or on mobile platforms. For example, the position is then a movable variable position and the orientation is a movable variable orientation.

Each marking can now be assigned a virtual sensor model via software application or APP application, i.e. using graphic operating software using the control and evaluation unit, which is displayed in augmented reality, i.e. an extended reality, with the aid of the display unit or a mobile terminal device . In particular, the detection range or effective range of the virtual sensor and the sensor function of the virtual sensor are also shown in the augmented reality. For example, the marker or markers can be tracked continuously, so that the mobile terminal device and the marker or markers can be movable. The detection area can be, for example, a detection field, a protection area, a protection field, a warning area, a warning field or the like.

Here, the environment or an environmental model also includes, for example, people who are in the real environment, for example an industrial scene. The markings are now used to assign and visualize the corresponding sensor function on the spot or the position in the correct orientation to a located marking.

The effective areas of the virtual sensors do not penetrate any virtual infrastructure, i.e. e.g. walls, floors or people are not irradiated in the augmented reality visualization, but are hit photo-realistically, for example. Similar to "Post-it notes", such an application can be planned interactively and/or immersively very efficiently and comprehensibly.

The solution according to the invention improves a planning phase of an application in which, as a rule, no real sensor is available or no real sensor is yet to be used in order to save costs.

Sensors are only represented by the marker, to which certain sensor properties can be assigned. The sensor properties assigned to a marking are visualized augmented and support the planning of the application, ie an optimal choice of the type of sensors, a number of sensors, an arrangement of the sensors and/or a configuration or settings of the sensors. For example, a visualization of a synchronization or mutual interference of sensors is provided. So can at for example an alternating pulsing of sensors can be shown.

This augmented or virtual planning goes far beyond the possibilities of purely digital planning on the PC. In particular, it opens up new possibilities for the topic of safety in the sense of functional safety or machine safety in complex applications, e.g.

Safety experts can use this to play through real work situations together with workers, even before purchase, installation, commissioning and/or real work situations and plan safe and productive work processes. In this way, for example, security gaps or gaps in the combined protective field of all planned sensors and thus the entire security solution can be identified. In order to visualize this, the measurement and identification of how the markings or markers are positioned relative to one another takes place.

If a marker position is changed or another marker is added, only the area around the new position needs to be recorded until the algorithm has enough references to correctly reposition the marker in the virtual environment. This could also take place in real time if the end device used for the visualization has the appropriate sensors.

In this way, accidental and intentional changes in the position of markings could be automatically detected, displayed and, if desired, corrected in the environment model. If the environment is recorded in real time, e.g. via the end device used for visualization, moving objects can also be taken into account.

After successful planning, the results can be documented using screenshots or videos and transferred to the application documentation, e.g. as part of a risk assessment.

The virtual sensors used in the simulation can, for example, be placed in a shopping basket in a tool of the control and evaluation unit and can be purchased by the customer as real sensors.

Sensor configurations formed from the simulation can also be delivered to customers as a parameter file, for example. In addition to the simple and fast planning of applications, this also enables and promotes the simple and fast setup of the real application.

The markings applied in the real environment can also serve as assembly instructions for an integrator team. For example, installation instructions, in particular text messages, can be displayed in the virtual environment, for example with the wording: “Please attach a sensor X here in the orientation Y shown with a parameter set Z”.

Planning risks are solved in artificial or augmented reality. In this approach, the actual integration requires a minimum of time. A downtime of an existing application for a conversion, for example, is thus minimized. Deviations between the planning status (e.g. environment model from CAD data) and the real application are made visible by the virtual sensor behavior. With this approach, applications can be replanned very easily and intuitively and the best solution found is then implemented in reality with real sensors.

The solution according to the invention allows support for new digital configuration and ordering processes. A customer can, for example, use a manufacturer app including a sensor library to plan the application virtually, save the sensor configuration and place an order including parameter files directly from the app.

Alternatively, the configuration can already be loaded onto the desired real sensor during the ordering process. Additional services can be offered, such as risk assessment based on the environmental model provided by the customer, including sensor arrangement.

Based on the components used, the costs of the solution are also precisely known before concrete implementation, so that a very well-founded cost-benefit analysis for the application is possible without actually procuring and installing a single component or sensor and beyond a specific simulation would not have to be programmed again in every plan, as this is generated from the augmented solution.

In a development of the invention, the spatial model is available as a 3D CAD model based on a real environment. The spatial model can be derived from a 3D CAD model and used, for example, as a meshed surface.

For example, a global or superordinate marker is provided whose pose is known relative to the 3D CAD model. Starting from this, the markings of the virtual sensors can be measured. For example, a graph is created internally, with the markers being the nodes and the edges representing the transformations between the markers. This is also important, for example, to inspect the effective ranges of several sensors at their interfaces.

According to the development, application planning by the user in the future production environment is possible, for example human-robot collaboration (HRC). A user creates a 3D CAD model or a 3D environmental model in the virtual environment. There is a simulation and visualization of the safeguarding of the danger points by placing tags in the environment, which can be flexibly linked to various virtual sensors and their properties via a SICK library. The user can thus plan and simulate his application virtually, save sensor configurations and place an individualized order via the APP. Based on their environmental model, the customer receives preconfigured sensors including installation instructions. After the real application has been installed, validation can be carried out, whereby the simulation in the virtual environment is compared with the real application. Deviations are detected and can be readjusted.

As in the case of the described human-robot collaboration (MRK) application, the application planning of workstations can also be carried out, for example, in the case of an autonomous mobile robot application (AMR). To do this, the user creates a 3D CAD model or a 3D environmental model in the virtual environment. By placing markings or markers that are linked to virtual sensors and their properties, securing the danger zone can be simulated and visualized. The user can also plan and simulate his application in augmented reality, save sensor configurations and place an individualized order via the app. In addition, not only workstations can be simulated and visualized in an autonomous mobile robot application (AMR), but also path sections. For this purpose, a live 3D environmental model can be created and potential danger points can be simulated in the route planning.

In a further development, the spatial model is or was formed using a 3D sensor. For example, the spatial model can be formed in advance using a 3D sensor. For this purpose, for example, a real environment is spatially scanned and the spatial model is created in advance. Alternatively, the spatial model can be formed in real time using the 3D sensor.

An environment model can thus be made available in advance. It can be recorded with 3D sensors with subsequent 3D map generation. It can be recorded with laser scanners with subsequent map generation.

In a development of the invention, the position and the orientation of the marking can be detected by the 3D sensor and the camera (7).

For example, a digital end device uses the 3D sensor to optically locate the position and orientation of the markings in the vicinity of the application using the 3D sensor and the camera. Here, for example, three different directions in space and three different orientations, ie six degrees of freedom of the marking are determined. The 3D sensor generates the associated virtual environment model or a situational environment model.

Knowledge of the relative position of the marking can also be updated in the event of a change in location or a change in orientation of the marking.

The terminals already mentioned several times, which are equipped with the camera, the 3D sensor and the display unit, can also be used for this.

From the combination of image data from the camera and depth maps generated by the 3D sensor when recording the environment and generating the environment model, the markings already placed in the application can be recognized and assigned to a location or coordinates in the environment.

In a development of the invention, the 3D sensor is a stereo camera or a time-of-flight sensor.

For example, a digital end device, such as an iPhone 12 Pro or an iPad Pro, uses the time-of-flight sensor to optically locate the position and orientation of the markings in the vicinity of the application with the time-of-flight sensor and the camera. Here, for example, three different directions in space and three different orientations, ie six degrees of freedom of the marking are determined. The time-of-flight sensor, for example a LIDAR scanner generates the associated virtual environmental model or a situational environmental model.

For example, a digital end device, for example a smartphone or a tablet computer, uses an integrated stereo camera to optically locate the position and orientation of the markings in the vicinity of the application with the stereo camera and the camera. Here, for example, three different directions in space and three different orientations, ie six degrees of freedom of the marking are determined. The stereo camera generates the associated virtual environmental model or a situational environmental model.

In a development of the invention, the time-of-flight sensor is a laser scanner or a 3D time-of-flight camera.

The environment can be recorded, for example, as a real-time environment model with the time-of-flight sensor, for example a 3D time-of-flight camera or a lidar camera or a laser scanner. According to the invention, the real-time environment model can be combined with 2D image data from the camera, for example the mobile terminal device, so that real-time segmentation is carried out based on the depth data. In this way, for example, a floor, a wall, a workplace and/or a person can be identified.

Transferred to an industrial safety context and an augmented reality-based HRC application planning, this could look like this in an example with three markings, each representing a virtual laser scanner.

For example, two markings are stationarily arranged on a robot and are intended to cover and secure a specific area around the robot. It is intended that this area will also be entered or drawn into the application using a mobile device, thus identifying areas that are not detected by the safety sensors in their current installation and configuration. Such areas arise, for example, behind static objects where the protective fields are cut off due to the taught-in environmental data.

The protective fields can also be interrupted or cut off at moving obstacles, such as the worker or the AGC, if, for example, an augmented reality device with 3D sensors or another device supplying 3D data is used. To separate taught-in environmental data and dynamic objects, segmentation, e.g. of the person in the color image, and subsequent mapping to 3D data would be provided. In principle, markings are also provided on moving machine parts or vehicles.

• Die Datenmodelle von virtuellen Sensoren sind beispielsweise in einer Bibliothek der Datenbank abgelegt. Diese Bibliothek umfasst beispielsweise für jeden Sensor das 3D-CAD-Modell, einen Wirkbereich und ein Set an Funktionen. Zusätzlich werden beispielsweise für jeden Sensor Metadaten abgelegt, diese können beispielsweise einen Sensortyp (z. B. Laserscanner, 3D-Lichtlaufzeitkamera, etc), physikalische Anschlüsse (z. B. Ethernet, IO-Link, etc) oder einen Performance Level umfassen. Die Bibliothek bzw. Sensorbibliothek kann mit Suchfiltern ausgestattet sein, so dass der Anwender entscheiden kann, welche virtuellen Sensoren er für die Zuordnung verwenden möchte (z.B. eine 3D-Lichtlaufzeitkamera mit 940 nm Wellenlänge, 3 m Reichweite, 70° Sichtbereich und Ethernet Anschluss). Einem morphologischen Kasten ähnelnd können den Markierungen unterschiedliche Sensoren zugeordnet werden, die dann zur Planung der Applikation verwendet werden.

A correspondence between the markings and the virtual sensors can look like this, for example:

• The data models of virtual sensors are stored in a library in the database, for example. For example, this library includes the 3D CAD model for each sensor, an effective range and a set of functions. In addition, metadata is stored for each sensor, for example, which can include a sensor type (e.g. laser scanner, 3D time-of-flight camera, etc.), physical connections (e.g. Ethernet, IO-Link, etc.) or a performance level. The library or sensor library can be equipped with search filters so that the user can decide which virtual sensors he would like to use for the assignment (e.g. a 3D time-of-flight camera with a wavelength of 940 nm, 3 m range, 70° field of view and Ethernet connection). Similar to a morphological box, different sensors can be assigned to the markings, which are then used to plan the application.

An environment model can also be provided in advance. It can be recorded with 3D sensors with subsequent 3D map generation. It can be recorded with laser scanners with subsequent map generation. It can be derived from a 3D CAD model and used e.g. as a meshed surface.

In a development of the invention, the markings are at least two-dimensional matrix codes. The unambiguous direction and the unambiguous orientation of the marking can be recognized and determined from the two-dimensional matrix coding. The matrix coding can optionally have the type of sensor and other properties of the sensor, such as a protective field size, a protective field direction and a protective field orientation. For example, the marking has a two-dimensional coding and additional color information.

The visual markers can be AprilTags or AprilMarker, for example. These have less information than a QR code and are still orientation-sensitive. For example, AprilTags tags have a white border and contain a at for example 8x8 matrix of black and white fields as matrix code.

So-called ArUco tags are also possible. An ArUcoTag or ArUco marker is a synthetic, square marker consisting of a wide black border and an inner binary matrix that determines its identifier (id). The black border facilitates its rapid identification in the image and the binary coding allows its identification and the application of error detection and correction techniques. The size of the marker determines the size of the internal matrix. For example, a marker size of 4x4 consists of 16 bits.

Vuforia markers "Vu-Mark" can also be used. Vuforia markers have an outline, border area, free space area, code elements, and background area. It is possible and envisaged that further information will be stored on the tag. E.g.: data sheets, certificates, user manuals, etc.

In a development of the invention, the markings are at least real 3D models. This can be, for example, small spatial sensor models made of foam or other materials. The marking is formed over the 3D contour of the sensor model.

In a development of the invention, the marking is arranged in the real environment by means of a holding device.

For example, universally suitable suspensions are provided in order to be able to place the markings as freely as possible in the application for planning purposes. For example, a modular attachment concept consisting of a base plate with the marking in combination with either a magnetic device, an adhesive device, a clamping device and/or a screw device is provided.

In a development of the invention, the position and orientation of a second marking is linked to a virtual object, with the virtual object being displayed in the spatial model of the environment on the display unit at the position and with the orientation of the second marking.

The virtual object can be, for example, a virtual machine, a virtual barrier, a virtual warehouse, virtual material, virtual workpieces or the like.

The movements, walking routes, interventions, etc. of a person can also be transferred to a simulation or the virtual environment using similar technology. For this purpose markings or markers can be attached to the joints of the person and the movements can be recorded and displayed.

In a development of the invention, the display unit is a smartphone, a tablet computer, augmented reality glasses/headset or virtual reality glasses/headset.

Devices or systems that are equipped with at least one camera and a visualization option can therefore be considered as mobile end devices.

In a development of the invention, the markings have at least one transponder.

According to the development, it is provided that the visual markings are additionally provided with radio location, for example with UWB technology. Air tags from Apple can be used here, for example. An AirTag is designed to work like a key finder, helping to find keys and other items using Ultra Wide Band (UWB) technology. The U1 chip developed by Apple can also be used to display the exact distance and direction to the AirTag you are looking for.

In a further development of the invention, the virtual sensor model can be configured, with the configuration of the virtual sensor model being able to be transferred to the real sensor.

Sensor configurations of the virtual sensor, e.g. a protective field size of a virtual laser scanner, can be saved using the environmental model and the desired effective range of virtual sensors spanned by the markings and, for example, made available to a customer as part of the ordering process for real sensors. In addition to the simple and fast planning of applications, the sensor configuration also supports the setup of the real application.

In a development of the invention, after virtual planning, at least one virtual solution is converted into a simulation. This enables a virtual continuation of the planning in a simulation.

Following the virtual planning, one or more virtual solutions can be transferred to a simulation. Process-relevant parameters such as speeds, components per hour, etc., as well as safety-relevant parameters can now be included in the simulation For example, detection areas, detection fields, protective fields, warning fields, speeds, routes and/or interventions by workers can be varied and simulated. In this way, on the one hand, the security concept can be further validated and, on the other hand, productivity and added value of the security solution can be quantified and compared. For example, it is provided that the exact sensor position and orientation is optimized in the virtual planning. The parameters of the sensors can then be optimized again.

• 1 ein System zur Planung einer Anwendung mit mindestens einer Markierung;
• 2 ein System zur Planung einer Anwendung mit mindestens einer Markierung;
• 3 ein weiteres System zur Planung einer Anwendung mit mindestens einer Markierung;
• 4 eine beispielhafte Markierung;
• 5 ein weiteres System zur Planung einer Anwendung mit mindestens einer Markierung.

The invention is explained below also with regard to further advantages and features with reference to the attached drawing based on exemplary embodiments. The figures of the drawing show in:

• 1 a system for scheduling an application with at least one marker;
• 2 a system for scheduling an application with at least one marker;
• 3 another system for scheduling an application with at least one marker;
• 4 an exemplary mark;
• 5 another system for planning an application with at least one marker.

In the following figures, identical parts are provided with identical reference numbers.

1 shows a system 1 for planning an application with at least one marking 2, with a control and evaluation unit 3, with a database 4, with a display unit 5, and with at least one spatial model of a real environment, and at least one camera 7 for imaging the real environment 8, wherein the real environment 8 in the spatial model can be displayed as a virtual environment 9 on the display unit 5, the marker 2 being arranged in the real environment 8 at a position and with an orientation, the position and the orientation of the Marking 2 can be detected at least by camera 7, with the position and orientation of marking 2 being linked to a virtual sensor model 10, with virtual sensor model 10 in the spatial model of virtual environment 9 on the display unit at the position and with the orientation of the Marking 2 can be displayed, the virtual sensor model 10 having at least one virtual detection area 11 , the virtual detection area 11 being able to be displayed in the spatial model of the virtual environment 9 on the display unit 5 .

2 shows a system 1 for planning an application with at least one marking 2, with a control and evaluation unit 3, with a database 4, with a display unit 5, and with at least one time-of-flight sensor 6 for spatial scanning of a real environment, and at least one camera 7 for Representation of the real environment, the real environment being displayable in a spatial model as a virtual environment 9 on the display unit 5, the marking 2 being arranged in the real environment 8 at a position and with an orientation, the position and the orientation of the Marking 2 can be detected by the time-of-flight sensor 6, the position and orientation of the marking 2 being linked to a virtual sensor model 10, the virtual sensor model 10 in the spatial model of the virtual environment 9 on the display unit 5 at the position and with the orientation of the Marking 2 can be displayed, the virtual sensor model 10 having a virtual protective field 11 , the virtual protective field 11 being able to be displayed in the spatial model of the virtual environment 9 on the display unit 5 .

3 shows a system according to 2 . For example, a set of N markings 2 or markers or so-called visual tags is provided. The marker 2 or markers 2 are placed or arranged in an environment of a planned use or application. For example fixed to an infrastructure, for example on a wall or at a workplace. For example, the position is a fixed position and the orientation is a fixed orientation.

For example, the markings 2 are arranged on manipulators, e.g. on a robot 15 or on mobile platforms 16. For example, the position is then a movable variable position and the orientation is a movable variable orientation.

Each marking 2 can now be assigned a virtual sensor model 10 via software or APP using the control and evaluation unit 3, which is displayed in augmented reality, ie an extended reality, with the aid of the display unit 5 or a mobile terminal device. In particular, the effective range of the virtual sensor and the sensor function of the virtual sensor are also shown in the augmented reality. Provision can also be made to be able to set the relative translation and orientation between the marker and the augmented object on the digital terminal device.

For example, a digital terminal 17 uses the time-of-flight sensor 6 to optically locate the position and orientation of the markings 2 in the vicinity of the application with the time-of-flight sensor 6 and the camera 7 for example three different spatial directions and three different orientations, ie six degrees of freedom of the marking 2 are determined. The time-of-flight sensor 6, for example a LIDAR scanner, generates the associated virtual environmental model or a situational environmental model.

Here, the environment or an environmental model also includes, for example, people 18 who are in the real environment, for example an industrial scene. The markings 2 are now used to assign the corresponding sensor function to a located marking 2 on the spot or the position in the correct orientation and to visualize it.

The effective ranges of the virtual sensors 10 do not penetrate any virtual infrastructure, i.e. e.g. walls, floors or people 18 are not irradiated in the augmented reality visualization, but instead are hit photorealistically, for example. Similar to "Post-it notes", such an application can be planned interactively and/or immersively very efficiently and comprehensibly.

Sensors are only represented by the marking 2, to which certain sensor properties can be assigned. The sensor properties assigned to a marking 2 are visualized augmented.

Knowledge of the relative position of the marking 2 can also be updated in the event of a change in location or a change in the orientation of the marking 2 .

From the combination of image data from the camera 7 and depth maps generated by the time-of-flight sensor 6 when recording the environment and generating the environment model, the markings 2 that have already been placed in the application can be recognized and assigned to a location or coordinates in the environment.

If a marker position is changed or another marker 2 is added, only the area around the new position needs to be recorded until the algorithm has enough references to correctly reposition marker 2 in the virtual environment. This could also take place in real time if the end device 17 used for the visualization has the appropriate sensors.

If the environment is recorded in real time, e.g. via the end device used for visualization, moving objects can also be taken into account.

After successful planning, the results can be documented using screenshots or videos and transferred to the application documentation, e.g. as part of a risk assessment.

The virtual sensors 10 used in the simulation can, for example, be placed in a shopping basket in a tool of the control and evaluation unit 3 and can be purchased by the customer as real sensors.

Sensor configurations formed from the simulation can also be delivered to customers as a parameter file, for example. In addition to the simple and fast planning of applications, this also enables and promotes the simple and fast setup of the real application.

The markings 2 applied in the real environment 8 can also serve as assembly instructions for an integrator team. For example, installation instructions, in particular text messages, can be displayed in the virtual environment 9, for example with the wording: “Please attach a sensor X here in the orientation Y shown with a parameter set Z”.

Following the virtual planning, one or more virtual solutions can be transferred to a simulation. In the simulation, process-relevant parameters such as speeds, components per hour, etc., as well as safety-relevant parameters such as protective fields, speeds, walking distances and/or interventions by workers can now be varied and simulated. In this way, on the one hand, the security concept can be further validated and, on the other hand, productivity and added value of the security solution can be quantified and compared.

Application planning by the user in the future production environment is possible, for example human-robot collaboration (HRC). A user creates a 3D environment model in the virtual environment 9. There is a simulation and visualization of the safeguarding of the danger spots by placing markings 2 in the environment, which can be flexibly linked to various virtual sensors and their properties via a SICK library. The user can thus plan and simulate his application virtually, save sensor configurations and place an individualized order via the APP. Based on their environmental model, the customer receives preconfigured sensors including installation instructions. After installing the real application, a validation can be performed be made, with the simulation in the virtual environment 9 being compared with the real application. Deviations are detected and can be readjusted.

As in the case of the described human-robot collaboration (MRK) application, the application planning of workstations can also be carried out, for example, in the case of an autonomous mobile robot application (AMR). For this purpose, the user creates a 3D environment model in the virtual environment 9. By placing markings 2 or markers that are linked to virtual sensors 10 and their properties, securing the danger point can be simulated and visualized. The user can also plan and simulate his application in augmented reality, save sensor configurations and place an individualized order via the app. In addition, not only workstations can be simulated and visualized in an autonomous mobile robot application (AMR), but also path sections. For this purpose, a live 3D environmental model can be created and potential danger points can be simulated in the route planning.

According to 3 the time-of-flight sensor 6 is, for example, a laser scanner or a 3D time-of-flight camera.

Sensor configurations of the virtual sensor 10, e.g. a protective field size of a virtual laser scanner, can be stored using the environmental model and the desired effective range of virtual sensors 10 spanned by the markings 2 and, for example, made available to a customer as part of the ordering process for real sensors. In addition to the simple and fast planning of applications, the sensor configuration also supports the setup of the real application.

According to 3 For example, two markings 2 are stationarily arranged on a robot 15 and are intended to capture and secure a specific area around the robot 15 . It is provided, for example, that this area is also entered or drawn in the application using a mobile terminal device 17 and thus areas that are not detected by the safety sensors in their current attachment and configuration are identified. Such areas arise, for example, behind static objects where the protective fields are cut off due to the taught-in environmental data. Here, for example, as in 4 displayed warnings, for example symbolically with exclamation marks in the virtual environment.

The protective fields can also be interrupted or cut off at moving obstacles, such as the person 18 or the mobile platform 16. In principle, markings 2 are also provided on moving machine parts or vehicles.

• Die Datenmodelle von virtuellen Sensoren 10 sind beispielsweise in einer Bibliothek der Datenbank 4 abgelegt. Diese Bibliothek umfasst beispielsweise für jeden Sensor das 3D-CAD-Modell, einen Wirkbereich und ein Set an Funktionen. Zusätzlich werden beispielsweise für jeden Sensor Metadaten abgelegt, diese können beispielsweise einen Sensortyp (z. B. Laserscanner, 3D-Lichtlaufzeitkamera, etc), physikalische Anschlüsse (z. B. Ethernet, IO-Link, etc) oder einen Performance Level umfassen. Die Bibliothek bzw. Sensorbibliothek kann mit Suchfiltern ausgestattet sein, so dass der Anwender entscheiden kann, welche virtuellen Sensoren 10 er für die Zuordnung verwenden möchte (z.B. eine 3D-Lichtlaufzeitkamera mit 940 nm Wellenlänge, 3 m Reichweite, 70° Sichtbereich und Ethernet Anschluss). Einem morphologischen Kasten ähnelnd können den Markierungen unterschiedliche Sensoren zugeordnet werden, die dann zur Planung der Applikation verwendet werden.

A correspondence between the markings 2 and the virtual sensors 10 can look like this, for example:

• The data models of virtual sensors 10 are stored in a library in database 4, for example. For example, this library includes the 3D CAD model for each sensor, an effective range and a set of functions. In addition, metadata is stored for each sensor, for example, which can include a sensor type (e.g. laser scanner, 3D time-of-flight camera, etc.), physical connections (e.g. Ethernet, IO-Link, etc.) or a performance level. The library or sensor library can be equipped with search filters so that the user can decide which virtual sensors 10 he would like to use for the assignment (e.g. a 3D time-of-flight camera with a wavelength of 940 nm, a range of 3 m, a field of view of 70° and an Ethernet connection). . Similar to a morphological box, different sensors can be assigned to the markings, which are then used to plan the application.

According to 3 and 4 the markings are 2 two-dimensional matrix codes. The unambiguous direction and the unambiguous orientation of the marking 2 can be recognized and determined from the two-dimensional matrix coding. The matrix coding can optionally show the type of sensor and other properties of the sensor, such as a protective field size, a protective field direction and a protective field orientation.

According to 3 the marking 2 is arranged in the real environment by means of a holding device.

For example, universally suitable suspensions are provided in order to be able to place the markings 2 as freely as possible in the application for planning purposes. For example, a modular attachment concept consisting of a base plate with the marking in combination with either a magnetic device, an adhesive device, a clamping device and/or a screw device is provided.

For example, the position and orientation of a second marker 2 is linked to a virtual object, the virtual object in which spatial model of the environment is displayed on the display unit at the position and with the orientation of the second marker 2.

The virtual object can be, for example, a virtual machine, a virtual barrier, a virtual warehouse, virtual material, virtual workpieces or the like.

The movements, walking routes, interventions, etc. of a person 18 can also be transferred to a simulation or the virtual environment using similar technology. For this purpose, markings 2 or markers can be attached to the joints of the person 18 and the movements can be recorded and displayed.

According to 3 the display unit is, for example, a smartphone, a tablet computer, augmented reality glasses/headset or virtual reality glasses/headset.

Devices or systems that are equipped with at least one camera and a visualization option can therefore be used as mobile or digital terminals 17 .

According to 3 the markings 2 have a transponder, for example.

For example, it is provided that the visual markings are additionally provided with radio location, for example with UWB technology.

reference list

1

system

2

mark

3

Control and evaluation unit

4

Database

5

display unit

6

3D sensor

7

camera

8th

real environment

9

virtual environment

10

virtual sensor model/virtual sensor

11

virtual protective field

15

robot

16

mobile platforms

17

digital device

18

person

Claims (14)

System (1) for planning an application with at least one marking (2), with a control and evaluation unit (3), with a database (4), with a display unit (5), and with at least one spatial model of a real environment, and at least one camera (7) for imaging the real environment (8), wherein the real environment (8) in the spatial model can be displayed as a virtual environment (9) on the display unit (5), wherein the marking (2) is arranged in the real environment (8) at a position and with an orientation, the position and the orientation of the marking (2) being detectable at least by the camera (7), the position and orientation of the marking (2) being linked to a virtual sensor model (10), wherein the virtual sensor model (10) can be displayed in the spatial model of the virtual environment (9) on the display unit at the position and with the orientation of the marking (2), wherein the virtual sensor model (10) has at least one virtual detection area (11), wherein the virtual detection area (11) can be displayed in the spatial model of the virtual environment (9) on the display unit (5). System (1) after claim 1 , characterized in that the spatial model is available on the basis of a real environment as a CAD model. System (1) after claim 1 , characterized in that the spatial model is formed or was formed by means of a 3D sensor (6). System (1) after claim 3 , characterized in that the position and the orientation of the marking (2) can be detected by the 3D sensor (6) and the camera (7). System (1) after claim 3 , characterized in that the 3D sensor (6) is a stereo camera or a time-of-flight sensor. System (1) after claim 1 , characterized in that the time-of-flight sensor is a laser scanner or a 3D time-of-flight camera. System (1) according to one of the preceding claims, characterized in that the markings (2) are at least two-dimensional matrix codes. System (1) according to one of the preceding claims, characterized in that the markings (2) are at least real 3D models. System (1) according to one of the preceding claims, characterized in that the marking (2) is arranged in the real environment (8) by means of a holding device. System (1) according to one of the preceding claims, characterized in that the position and orientation of a second marking (2) is linked to a virtual object, the virtual object being displayed in the spatial model of the virtual environment (9) on the display unit (5 ) is displayed at the position and orientation of the second marker (2). System (1) according to one of the preceding claims, characterized in that the display unit (5) is a smartphone, a tablet computer, augmented reality glasses/headset or virtual reality glasses/headset. System (1) according to one of the preceding claims, characterized in that the markings (2) have at least one transponder. System (1) according to one of the preceding claims, characterized in that the virtual sensor model can be configured, the configuration of the virtual sensor model being transferrable to the real sensor. System (1) according to one of the preceding claims, characterized in that after virtual planning at least one virtual solution is converted into a simulation.

Images