WO2022100960A1

WO2022100960A1 - Method for providing navigation data in order to control a robot, method for controlling a robot, method for producing at least one predefined point-symmetric region, and device

Info

Publication number: WO2022100960A1
Application number: PCT/EP2021/078611
Authority: WO
Inventors: Stephan Simon
Original assignee: Robert Bosch Gmbh
Priority date: 2020-11-12
Filing date: 2021-10-15
Publication date: 2022-05-19
Also published as: DE102020214249A1; CN116710975A

Abstract

The invention relates to a method for providing navigation data (135) for controlling a robot. The method has a step of reading image data (105), which is provided by a camera (102), from the camera (102), said image data (105) representing a camera image of at least one predefined point-symmetric region (110); a step of determining at least one symmetry center (112) of the at least one point-symmetric region (110) using the image data (105) and a determination instruction (128); a step of carrying out a comparison of the position of the symmetry center (112) in the camera image using a predefined position of a reference symmetry center in a reference image (115) in order to determine a positional deviation (131) between the symmetry center (112) and the reference symmetry center; and optionally a step of ascertaining displacement vectors (133) for at least a sub-quantity of pixels of the camera image relative to corresponding pixels of the reference image (115) using the positional deviation (131). The navigation data (135) is provided using the positional deviation (131) and/or displacement vectors (133).

Description

description

title

Method for providing navigation data for controlling a robot, method for controlling a robot, method for producing at least one predefined point-symmetrical area and device

State of the art

The invention is based on a device or a method according to the species of the independent claims. The subject matter of the present invention is also a computer program.

In the field of visual robot control (visual servoing), an efficient and precise determination of a robot pose or a position and orientation of a robot using image data can be of particular importance.

The subsequently published DE 10 2020 202 160 A1 discloses a method for determining a symmetry property in image data and a method for controlling a function.

Disclosure of Invention

Against this background, with the approach presented here, a method, furthermore a device which uses this method, and finally a corresponding computer program according to the main claims are presented. Advantageous developments and improvements of the device specified in the independent claim are possible as a result of the measures listed in the dependent claims. According to embodiments, it can be exploited in particular when points or objects in the world are or are marked using point-symmetrical areas, so that a system with an imaging sensor and a suitable method presented here can use these point-symmetrical areas to perform a specific technical function in a robust and local manner can detect and localize with high precision, optionally without humans or living beings perceiving such markings as disturbing.

It can happen, for example, that a symmetrical area is not completely displayed in the camera image, e.g. B. because it may be partially covered by an object, or because it may partially protrude from the image, or because the pattern may have been cropped. Advantageously, a precision of a localization of the point center of symmetry can nevertheless be maintained, because the partial occlusion(s) do not falsify its position: remaining point-symmetrical correspondence pairs can nevertheless vote for the correct center of symmetry. A partial occlusion can only reduce a strength of an accumulation point in a matching matrix or the like, but the position of the center of symmetry can be retained and still be precisely and easily determinable. This is a special advantage of exploiting point symmetry.

Further advantages when finding areas or patterns based on point symmetry can result in particular from the fact that the point symmetry is invariant with respect to rotation between the point-symmetrical area and the camera or image recording and is largely invariant with respect to a perspective. For example, a point-symmetric planar surface can be invariant under an affine mapping. An imaging of an arbitrarily oriented plane by a real camera can always be approximated very well, at least locally, by an affine imaging. For example, if you look at a circular area with point symmetry from an oblique perspective, the circular shape becomes an elliptical shape, with the point symmetric property and the point symmetry center being retained. Thus, the at least one point-symmetrical area does not necessarily have to be viewed from a frontal perspective - also provide very oblique perspectives does not pose a difficulty and obtainable accuracy can be maintained. Such an invariance, in particular with respect to rotation and with respect to perspective, can make it possible to dispense with precautions for aligning the camera in a suitable manner with respect to the symmetrical area or vice versa. Rather, it may already be sufficient if the respective point-symmetrical area is at least partially recorded in the camera image so that it can be detected. A relative positional relationship or arrangement between the point-symmetrical area and the camera can be irrelevant or almost irrelevant.

A method for providing navigation data for controlling a robot is presented, the method having the following steps:

Reading in image data provided by a camera from an interface to the camera, the image data representing a camera image of at least one predefined even and/or odd point-symmetrical area in an area surrounding the camera;

determining at least one center of symmetry of the at least one even and/or odd point-symmetrical area using the image data and a determination rule;

Comparing a position of the at least one center of symmetry in the camera image with a predefined position of at least one reference center of symmetry in a reference image relative to a reference coordinate system in order to determine a position deviation between the center of symmetry and the reference center of symmetry; and or

Determining displacement information for at least a subset of pixels of the camera image relative to corresponding pixels of the reference image using the positional deviation, the navigation data being provided using the positional deviation and/or the displacement information. This method can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a control unit or a device. In the reading step, image data from a plurality of cameras can also be read in, with the image data representing a plurality of camera images of the at least one area. Optionally, a plurality of reference images can be used in the implementation step. The reference image can also be replaced by reference data that at least partially corresponds to or is equivalent to the information that can be obtained from a reference image. Working with reference data can be advantageous, particularly in terms of reducing the effort if the information that can be extracted from a reference image is already available in a form that can be used more easily in the form of the reference data. The reference data can represent the reference image in a condensed form or representation, for example as a descriptor image, signature image and/or with a listing of all coordinates and types of existing centers of symmetry. The at least one predefined point-symmetrical region can be manufactured by performing a variant of a manufacturing method described below. The determination specification can be similar to or correspond to a procedure that is disclosed in the applicant's subsequently published DE 10 2020 202 160. The reference image can represent the at least one predefined point-symmetrical area. The optional step of determining can be performed using the optical flow, in particular the dense optical flow. Controlling the robot can represent controlling or regulating a pose, location, position, orientation or the like of the robot relative to a reference point or reference object marked with the at least one predefined point-symmetrical area. At least one actuator of the robot can be controlled using the control signal to control the robot. The displacement information can represent displacement vectors or absolute coordinates.

According to one specific embodiment, the determination rule used in the determination step can be designed to cause a signature to be generated for a plurality of pixels of at least one section of the camera image is used to obtain a plurality of signatures. Here, each of the signatures can be generated using a descriptor with a plurality of different filters. Each filter can have at least one type of symmetry. Each of the signatures can have a sign for each filter of the descriptor. The determination rule can also be designed to ensure that at least one mirror signature for at least one type of symmetry of the filter is determined for the signature. The determination rule can also be designed to cause a pixel having the signature to be checked for the presence of at least one other pixel with a signature that corresponds to the at least one mirror signature in a search area in an area surrounding the pixel, in order to determine the presence of at least one other Pixels to determine pixel coordinates of at least one symmetrical pair of signatures from the pixel and another pixel. In addition, the determination rule can be designed to cause the pixel coordinates of the at least one symmetrical signature pair to be evaluated in order to identify the at least one center of symmetry. The descriptor can describe an image content in a local neighborhood around a pixel or reference pixel in a compact form. A signature can describe a value of the descriptor for a pixel, for example in a binary representation. The at least one mirror signature can thus be determined using a plurality of calculated signature images, eg one with normal filters, one with even point-mirrored filters, and one with odd point-mirrored filters. Additionally or alternatively, at least one reflector can be applied to the signs of one of the signatures in order to determine the at least one mirror signature. In this case, each reflector for a type of symmetry can have specific rules for modifying the signs that are dependent on the filters of the descriptor. The search area can be dependent on at least one of the reflectors used. Such an embodiment offers the advantage that an efficient and precise detection of a symmetry property in image data can be made possible. The detection of symmetries in images can be achieved with minimal effort.

In this case, in the step of determining for each already determined center of symmetry using the pixel coordinates of each symmetrical Signature pair that has contributed to the correct identification of the center of symmetry, a transformation rule for transforming pixel coordinates of the center of symmetry and / or the at least one predefined even and / or odd point-symmetrical area are determined. The transformation rule can be applied to the pixel coordinates of the center of symmetry and/or the at least one predefined even and/or odd point-symmetrical area in order to correct a distorted perspective of the camera image. Such an embodiment offers the advantage that a reliable and precise reconstruction of the correct grid or the correct topology of a number of point-symmetrical areas can be achieved.

A type of symmetry of the at least one center of symmetry can also be determined in the determination step. The type of symmetry can represent an even point symmetry and additionally or alternatively an odd point symmetry. Additionally or alternatively, in the performing step, the type of symmetry of the at least one center of symmetry in the camera image can be compared with a predefined type of symmetry of at least one reference center of symmetry in a reference image in order to determine a match between the at least one center of symmetry and the at least one reference center of symmetry to consider. Odd point symmetry can be generated by point reflection with inversion of gray values or color values. By using and identifying the two different point symmetries, the information content of point-symmetrical areas and patterns can be increased.

The image data read in in the reading step can represent a camera image of at least one pattern from a plurality of predefined even and/or odd point-symmetrical regions. In the step of determining, a geometric arrangement of symmetry centers of the at least one pattern can be determined, a geometric sequence of symmetry types of the symmetry centers can be determined and additionally or alternatively the pattern can be determined from a plurality of predefined patterns using the sequence. The arrangement and/or the sequence may represent an identification code of the pattern. Such an embodiment offers the advantage that the reliability of the identification of centers of symmetry can be increased and further information can be obtained by recognizing a specific pattern. Reliable identification of centers of symmetry for different distances between camera and sample can also be achieved.

In this case, in the step of determining, using the arrangement of the centers of symmetry of the at least one pattern and additionally or alternatively the sequence of types of symmetry of the centers of symmetry, implicit additional information of the at least one pattern or a reading rule for reading out explicit additional information in the camera image can be determined. The arrangement and additionally or alternatively the sequence can represent the additional information in encoded form. The additional information can be related to controlling the robot. Such an embodiment offers the advantage that additional information can be communicated through the topology of the at least one pattern.

Depending on the specific arrangement, the specific sequence and additionally or alternatively the specific pattern, the reference image can also be selected from a plurality of stored reference images or generated using a stored generation rule in the step of performing. In this way, the correct reference image can be reliably identified. Optionally, when there is a link between identified patterns and a generation specification, a storage requirement for reference images can also be minimized since only generation specifications need to be stored.

Furthermore, the determining step and additionally or alternatively the performing step can be carried out jointly for all symmetry centers independently of the symmetry type of the symmetry centers or separately for the symmetry centers of the same symmetry type depending on the symmetry type of the symmetry centers. Thus, by joint execution, a low memory and time requirement for an exact and reliable identification of the symmetry centers can be achieved. Optionally, in particular, confusion with patterns that occur randomly in images can be minimized by separate execution.

A method for controlling a robot is also presented, the method having the following steps:

evaluating navigation data provided according to an embodiment of the above method in order to generate a control signal dependent on the navigation data; and

outputting the control signal to an interface to the robot to control the robot.

This method can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a control unit or a device. In this case, the method for controlling can advantageously be carried out in connection with an embodiment of the above-mentioned method for providing.

A method for producing at least one predefined even and/or odd point-symmetrical region for use by an embodiment of a method mentioned above is also presented, the method having the following steps:

generating design data representing a graphical representation of the at least one predefined even and/or odd point symmetric region; and

Generating the at least one predefined even and/or odd point-symmetrical region using the design data on, on or in a display medium to produce the at least one predefined even and/or odd point-symmetrical region. This method can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a control unit or a device. By performing the manufacturing method, at least one predefined even and/or odd point-symmetrical region can be manufactured, which can be used within the scope of an embodiment of an above-mentioned method.

According to one embodiment, design data can be generated in the generation step, which is a graphic representation of the at least one predefined even and/or odd point-symmetrical area as a circle, an ellipse, a square, a rectangle, a pentagon, a hexagon, a polygon or a represent annulus. In this case, the at least one predefined even and/or odd point-symmetrical area can have a regular or quasi-random content pattern. Additionally or alternatively, a first half of an area of the at least one predefined even and/or odd point-symmetrical area can be specified arbitrarily and a second half of the area can be constructed by point reflection and optionally additional inversion of gray values and additional or alternative color values. Additionally or alternatively, in the step of generating, the at least one predefined even and/or odd point-symmetrical area can be generated by an additive manufacturing process, separating, coating, forming, primary forming or optical display. Additionally or alternatively, the display medium may include glass, stone, ceramic, plastic, rubber, metal, concrete, plaster, paper, cardboard, food, or a visual display device. The at least one predefined even and/or odd point-symmetrical area can thus be produced in a precisely suitable manner depending on the specific use or depending on the specific application and the boundary conditions prevailing there.

Design data can also be generated in the generation step, which represent a graphical representation of at least one pattern from a plurality of predefined even and/or odd point-symmetrical regions. Here, at least a subset of even and / or odd point-symmetrical areas on a regular or irregular grid, directly adjacent to each other and additionally or alternatively partially separated from at least one adjacent even and/or odd point-symmetrical region by a gap, be identical to or different from each other in terms of their dimensions and/or their content patterns and additionally or alternatively in a be arranged in the same plane or in different planes. Additionally or alternatively, in the generation step, design data can be generated that represent a graphical representation of at least one pattern with hierarchical symmetry. In this way, different patterns with specific information content and additionally or alternatively patterns with hierarchical symmetries can be produced for different distances to the pattern.

Humans, in particular, find it difficult to perceive the symmetries hidden in the patterns, even if a corresponding marking is known to be present. This also makes it possible, for example, to hide such markings. This can be useful or desirable for various reasons, e.g. B. especially for aesthetic reasons, because the technical markings should not be seen or are desired because z. B. a concentration should not be reduced by markings that are unimportant for humans, or because the markings should remain secret. Aesthetic reasons play an important role, especially in the field of design. For example, in a vehicle interior, on an outer skin of a vehicle, on aesthetically designed objects or in the area of interior or building architecture, conspicuous technical markers would not or hardly be accepted. However, if they were hidden, e.g. B. in a fabric pattern or in a plastic or ceramic relief or in a hologram or on a printed surface, as is possible according to embodiments, they could be beautiful and useful at the same time, e.g. B. to provide a camera with one or more reference points, e.g. B. to be able to determine a relative camera pose. The aspect of hiding can also be irrelevant or of little relevance depending on the application. A technical robustness then still speaks for the use of such designed patterns. In particular, a pattern with a random character or pseudo-random character can offer many possibilities of being able to find pairs of points that are as unambiguous as possible. This can, for example, be exploited according to embodiments, in particular with advantages for the signal-to-noise ratio of a measured response at the centers of symmetry and thus for robustness in terms of their error-free detection and precise localization. In particular, a pattern can comprise one or more point-symmetrical areas with odd or even point symmetry. These areas can, for example, have a circular, hexagonal, square, elliptical, polygonal or other shape. The point-symmetrical areas can be of the same type or different in shape and size. You can connect to each other without gaps or be spaced apart.

The approach presented here also creates a device that is designed to carry out, control or implement the steps of a variant of a method presented here in corresponding devices. The object on which the invention is based can also be achieved quickly and efficiently by this embodiment variant of the invention in the form of a device.

For this purpose, the device can have at least one computing unit for processing signals or data, at least one memory unit for storing signals or data, at least one interface to a sensor or an actuator for reading in sensor signals from the sensor or for outputting data or control signals to the Have actuator and / or at least one communication interface for reading or outputting data that are embedded in a communication protocol. The arithmetic unit can be, for example, a signal processor, a microcontroller or the like, with the memory unit being able to be a flash memory, an EEPROM or a magnetic memory unit. The communication interface can be designed to read in or output data wirelessly and/or by wire, wherein a communication interface that can read in or output wire-bound data can, for example, read this data electrically or optically from a corresponding data transmission line or can output it to a corresponding data transmission line. In the present case, a device can be understood to mean an electrical device that processes sensor signals and, depending thereon, outputs control and/or data signals. The device can have an interface that can be configured as hardware and/or software. In the case of a hardware design, the interfaces can be part of a so-called system ASIC, for example, which contains a wide variety of functions of the device. However, it is also possible for the interfaces to be separate integrated circuits or to consist at least partially of discrete components. In the case of a software design, the interfaces can be software modules which are present, for example, on a microcontroller alongside other software modules.

A computer program product or computer program with program code, which can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical memory and for carrying out, implementing and/or controlling the steps of the method according to one of the embodiments described above, is also advantageous used, especially when the program product or program is run on a computer or device. The method can be implemented as a hardware accelerator on a SoC or ASIC.

Exemplary embodiments of the approach presented here are shown in the drawings and explained in more detail in the following description. It shows:

1 shows a schematic representation of an exemplary embodiment of a device for providing, an exemplary embodiment of a device for controlling and a camera;

2 shows a schematic representation of an exemplary embodiment of a device for production;

3 shows a flowchart of an embodiment of a method for providing; 4 shows a flow chart of an embodiment of a method for controlling;

5 shows a flow chart of an exemplary embodiment of a method for manufacturing;

6 shows schematic representations of display media with patterns of predefined point-symmetrical areas according to exemplary embodiments;

7 shows schematic representations of display media with patterns of predefined point-symmetrical areas according to exemplary embodiments;

FIG. 8 shows schematic representations of the display media with the patterns from FIG. 7 with graphical highlighting of the patterns or the predefined point-symmetrical areas;

9 shows schematic representations of predefined point-symmetrical regions according to exemplary embodiments;

10 shows a schematic representation of a pattern of predefined point-symmetrical regions according to an embodiment;

11 shows a schematic representation of the use of a look-up table according to an embodiment;

12 shows a schematic diagram of a voting matrix according to an embodiment;

13 shows a schematic representation of patterns arranged in the form of a cube by way of example according to an exemplary embodiment with regard to correct identification of a raster;

Fig. 14 is an oblique perspective schematic representation of the pattern shown in the first partial representation of Fig. 6; FIG. 15 shows the pattern from the first partial representation of FIG. 14 with an emphasis on a predefined point-symmetrical area; FIG.

FIG. 16 shows a schematic illustration of the pattern from FIG. 15 after perspective rectification according to an exemplary embodiment; FIG.

Fig. 17 is a schematic representation of an embodiment of a pattern having hierarchical symmetry;

Fig. 18 is a schematic representation of an embodiment of a pattern having hierarchical symmetry;

Fig. 19 is a schematic representation of an embodiment of a pattern having hierarchical symmetry;

20 shows schematic representations of patterns according to exemplary embodiments;

FIG. 21 shows a schematic representation of an application situation for the control device from FIG. 1 and/or the method for control from FIG. 4;

22 shows a schematic representation of various display media with predefined point-symmetrical areas;

23 shows a camera image of a conveyor belt as a display medium with an exemplary embodiment of a pattern made up of predefined point-symmetrical areas and objects placed on the conveyor belt; and

Fig. 24 shows the camera image from Fig. 23 after processing using the method for providing from Fig. 3.

In the following description of advantageous exemplary embodiments of the present invention, the same or similar elements are used for the elements which are shown in the various figures and have a similar effect Reference signs used, with a repeated description of these elements is omitted.

Fig. 1 shows a schematic representation of an exemplary embodiment of a device 120 for providing, an exemplary embodiment of a device 140 for controlling and, by way of example, only one camera 102. In the representation of Fig. 1, device 120 is for providing or providing device 120 and device 140 shown separate from camera 102 for control or control device 140 . The provision device 120 and the control device 140 are connected to the camera 102 in a data-transmitting manner. According to another exemplary embodiment, the provision device 120 and/or the control device 140 can also be part of the camera 102 and/or can be combined with one another.

The camera 102 is designed to record a camera image of the surroundings of the camera 102 . In the vicinity of the camera 102, for example, only a predefined even and/or odd point-symmetrical region 110 with a center of symmetry 112 is arranged. The camera 102 is also designed to provide or generate image data 105 that represent the camera image, the camera image also showing the predefined even and/or odd point-symmetrical region 110 .

The provision device 120 is designed to provide navigation data 135 for controlling a robot. In this case, the camera 102, the provision device 120 and/or the control device 140 can be embodied as part of the robot or separately from them. For this purpose, the provision device 120 comprises a reading device 124, a determining device 126, an implementation device 130 and optionally also a determination device 132. The reading device 124 is designed to read in the image data 105 from an input interface 122 of the provision device 120 to the camera 102. Furthermore, the reading-in device 124 is also designed to forward the image data 105 representing the camera image to the determination device 126 . Determination device 126 of provision device 120 is designed to determine center of symmetry 112 of at least one point-symmetrical region 110 using image data 105 and a determination rule 128 . The specification 128 will be discussed in more detail below. At this point it should be pointed out that the determination rule 128 is similar to or corresponds to a procedure that is disclosed in the subsequently published DE 10 2020 202 160 of the applicant. Determination device 126 is also designed to pass on the at least one specific center of symmetry 112 to implementation device 130 .

The implementation device 130 is designed to carry out a comparison of a position of the at least one center of symmetry 112 in the camera image with a predefined position of at least one reference center of symmetry in a reference image 115 relative to a reference coordinate system, in order to determine a position deviation 131 between the center of symmetry 112 and the reference to determine the center of symmetry. The implementation device 130 is also designed to read in or receive the reference image 115 or reference data 115 from a storage device 150 . The storage device 150 can be embodied as part of the provision device 120 or separately from the same. Furthermore, according to one exemplary embodiment, implementation device 130 is designed to forward position deviation 131 to determination device 132 .

According to one exemplary embodiment, determination device 132 is designed to then determine displacement information 133 for at least a subset of pixels of the camera image relative to corresponding pixels of reference image 115 using position deviation 131 .

The provision device 120 is designed to provide the navigation data 135 using the position deviation 131 and/or the displacement information 133 . More precisely, the Providing device 120 designed to provide the navigation data 135 to the control device 140 via an output interface 138 of the providing device 120 .

The control device 140 is designed to control the robot. For this purpose, the control device 140 includes an evaluation device 144 and an output device 146. The control device 140 is designed to receive or read in the navigation data 135 via an input interface 142 of the control device 140 from the provision device 120. The evaluation device 144 is designed to evaluate the navigation data 135 provided by the provision device 120 in order to generate a control signal 145 dependent on the navigation data 135 . The evaluation device 144 is also designed to forward the control signal 145 to the output device 146 . The output device 146 is designed to output the control signal 145 to an output interface 148 on the robot in order to control the robot.

In particular, determination rule 128 is designed to cause a signature to be generated for a plurality of pixels of at least one section of the camera image in order to obtain a plurality of signatures. Here, each of the signatures is generated using a descriptor with a plurality of different filters. Each filter has at least one type of symmetry. Each of the signatures includes a sign for each filter of the descriptor. The determination rule 128 is also designed to cause at least one reflector to be applied to the signs of one of the signatures in order to determine at least one mirror signature for at least one type of symmetry of the filters for the signature. In this case, each reflector for a type of symmetry has specific rules for modifying the signs that are dependent on the filters of the descriptor. The determination rule is also designed to cause a pixel having the signature to be checked for the presence of at least one other pixel with a signature corresponding to the at least one mirror signature in a search area dependent on at least one of the reflectors used in a vicinity of the pixel, to at least one if present further pixel to determine pixel coordinates of at least one symmetrical signature pair from the pixel and a further pixel. In addition, the determination rule is designed to cause the pixel coordinates of the at least one symmetrical pair of signatures to be evaluated in order to identify the at least one center of symmetry.

According to one exemplary embodiment, determination device 126 is designed to use the pixel coordinates of each symmetrical signature pair that contributed to the correct identification of symmetry center 112 to generate a transformation rule for transforming pixel coordinates of symmetry center 112 and/or point-symmetrical region 110 for each already determined center of symmetry 112 to determine. The transformation rule is applied to the pixel coordinates of the center of symmetry 112 and/or the point-symmetrical area 110 in order to correct a distorted perspective of the camera image. Furthermore, it is advantageous, because it is more robust, more accurate and subject to less noise, to determine a transformation rule using a plurality of, in particular adjacent, point-symmetrical regions 110, in particular if these are located on a common plane. The application of the transformation is particularly advantageous when an arrangement of several centers of symmetry 112 is considered.

According to one exemplary embodiment, determination device 126 is also designed to determine a type of symmetry of the at least one center of symmetry 112 . The symmetry type represents an even point symmetry and additionally or alternatively an odd point symmetry. Additionally or alternatively, implementation device 130 is designed to compare the type of symmetry of the at least one center of symmetry 112 in the camera image with a predefined type of symmetry of at least one reference center of symmetry in a reference image 115 in order to determine a match between the at least one center of symmetry 112 and the at least to check a reference center of symmetry.

In particular, the image data 105 represent a camera image of at least one pattern from a plurality of predefined ones point-symmetrical regions 110. Determination device 126 is designed to determine a geometric arrangement of centers of symmetry 112 of the at least one pattern, to determine a geometric sequence of types of symmetry of centers of symmetry 112, and/or to determine the correct pattern represented by image data 105 using the sequence to be determined from several predefined patterns. The arrangement and/or sequence may represent an identification code of the pattern. According to one exemplary embodiment, determination device 126 is designed here to use the arrangement of the centers of symmetry 112 of the at least one pattern and/or the sequence of symmetry types of the centers of symmetry 112 to determine implicit additional information of the at least one pattern or a readout rule for reading out explicit additional information in the camera image to determine. The arrangement and/or the sequence represents or represent the additional information in encoded form. The additional information relates to controlling the robot. Additionally or alternatively, implementation device 130 is designed here to select reference image 115 from a plurality of stored reference images or to generate it using a stored generation rule, depending on the specific arrangement, the specific sequence and/or the specific pattern.

FIG. 2 shows a schematic representation of an exemplary embodiment of a device 200 for production. The device 200 for producing is designed to produce at least one predefined even and/or odd point-symmetrical region 110 for use by the providing device from FIG. 1 or a similar device and/or control device from FIG. 1 or a similar device. For this purpose, the device 200 for manufacturing comprises a generating device 202 and a generating device 206. The generating device 202 is designed to generate design data 204. The design data 204 represents a graphical representation of the at least one predefined even and/or odd point-symmetrical region 110. The generating device 206 is designed to use the design data 204 to generate the at least one predefined even and/or odd point-symmetrical region 110 on, on or in a To generate display medium to produce the same. According to one embodiment, the generating device 202 is designed to generate design data 204, which is a graphic representation of the at least one predefined point-symmetrical region 110 as a circle, an ellipse, a square, a rectangle, a pentagon, a hexagon, a polygon or an annulus represent, wherein the at least one predefined point-symmetrical area 110 has a regular or quasi-random content pattern, and/or wherein a first half of an area of the at least one predefined point-symmetrical area 110 is specified arbitrarily and a second half of the area by point reflection and/or inversion is constructed from gray values and/or color values. Additionally or alternatively, the generating device 206 is designed to generate the at least one predefined point-symmetrical region 110 by an additive manufacturing process, separating, coating, forming, primary forming or optical display. Additionally or alternatively, the display medium has glass, stone, ceramics, plastic, rubber, metal, concrete, plaster, paper, cardboard, food or an optical display device.

According to one exemplary embodiment, the generating device 202 is designed to generate design data 204 that represents a graphical representation of at least one pattern from a plurality of predefined point-symmetrical regions 110, with at least a subset of the point-symmetrical regions 110 being aligned on a regular or irregular grid, directly are adjacent to each other and/or partially separated from at least one adjacent point-symmetrical region 110 by a gap, are identical to or different from each other in terms of their dimensions and/or their content patterns, and/or are arranged in a common plane or in different planes. Additionally or alternatively, the generating device 202 is designed to generate design data 204 that represent a graphical representation of at least one pattern with hierarchical symmetry.

FIG. 3 shows a flow chart of an embodiment of a method 300 for providing navigation data for controlling a robot. That Method 300 for providing can be carried out using the providing device from FIG. 1 or a similar device. The method 300 for providing comprises a step 324 of reading in, a step 326 of determining, a step 330 of implementation and optionally an additional step 332 of determining.

In step 324 of reading in, image data provided by a camera are read in from an interface to the camera. The image data represent a camera image of at least one predefined even and/or odd point-symmetrical area in an area surrounding the camera. At least one center of symmetry of the at least one point-symmetrical region is then determined in step 326 of determination using the image data and a determination rule. Again subsequently, in step 330 of execution, a comparison of a position of the at least one center of symmetry in the camera image with a predefined position of at least one reference center of symmetry in a reference image relative to a reference coordinate system is carried out in order to determine a position deviation between the center of symmetry and the reference center of symmetry determine. Subsequently, according to an exemplary embodiment, in step 332 of determining, using the position deviation, displacement information is determined for at least a subset of pixels of the camera image relative to corresponding pixels of the reference image. The navigation data is provided using the position deviation and/or the determined displacement information.

According to one exemplary embodiment, the image data read in in step 324 of reading represent a camera image of at least one pattern from a plurality of predefined point-symmetrical regions. In step 326 of determining, a geometric arrangement of symmetry centers of the at least one pattern is determined, a geometric sequence of symmetry types of the symmetry centers is determined and/or the pattern is determined from a plurality of predefined patterns using the sequence. The arrangement and/or the sequence represents an identification code of the pattern. Optionally, step 326 of Determining and/or the step 330 of carrying out is carried out jointly for all centers of symmetry independently of the type of symmetry of the centers of symmetry or carried out separately for the centers of symmetry of the same type of symmetry depending on the type of symmetry of the centers of symmetry.

FIG. 4 shows a flowchart of an embodiment of a method 400 for controlling a robot. The method 400 for controlling can be carried out using the control device from FIG. 1 or a similar device. Also, the method 400 for controlling is executable in connection with the method for providing of FIG. 3 or a similar method. The method 400 for controlling comprises a step 444 of evaluating and a step 446 of outputting.

In step 444 of the evaluation, navigation data provided according to the method for providing from FIG. 3 or a similar method is evaluated in order to generate a control signal dependent on the navigation data. Subsequently, in step 446 of outputting, the drive signal is output to an interface to the robot in order to control the robot.

5 shows a flowchart of an embodiment of a method 500 for manufacturing. The manufacturing method 500 is executable to manufacture at least one predefined point-symmetrical region for use by the providing method of FIG. 3 or a similar method and/or for use by the controlling method of FIG. 4 or a similar method. The method 500 for manufacturing can also be carried out in connection with or using the device for manufacturing from FIG. 2 or a similar device. The method 500 for manufacturing comprises a step 502 of generating and a step 506 of creating.

In step 502 of generation, design data are generated which represent a graphical representation of the at least one predefined point-symmetrical area. Subsequently, in step 506 of generating using the design data, the at least one predefined point-symmetrical area generated on, on or in a display medium in order to produce the at least one predefined point-symmetrical area.

6 shows schematic representations of presentation media 600 with patterns 610 of predefined point-symmetric regions 110A and HOB according to embodiments. Each of the predefined point-symmetrical areas 110A and 110B corresponds to or is similar to the predefined point-symmetrical area from Fig. 1. In a first partial view A, a pattern 610 of 49 predefined point-symmetrical areas 110A and HOB is shown merely by way of example, and in a second partial view B there is a pattern 610 is shown from eight predefined point-symmetrical regions 110A and HOB, merely by way of example. In this case, first predefined point-symmetrical areas 110A have odd point symmetry as the type of symmetry and second predefined point-symmetrical areas HOB have even point symmetry as the type of symmetry. In this case, a noise-like image pattern with the respective pattern 610 is printed on each of the display media 600 .

A use of symmetries in the field of machine vision according to exemplary embodiments can be illustrated with reference to FIG . In this case, point symmetries are more or less hidden in the patterns 610, which are hardly recognizable to an observer. By graphically highlighting the predefined point-symmetric regions 110A and HOB in FIG. 6, they are recognizable to a human viewer in the noise-like image pattern on the presentation media 600. The first partial representation A contains 49 exemplary circular symmetrical regions 110A and HOB, of which 25 first regions 110A with odd point symmetry and 24 second regions HOB with even point symmetry are merely exemplary. In the second partial representation B, the symmetrical regions 110A and HOB, of which only five have an odd point symmetry and only three of which have an even point symmetry, are selected to be larger than in FIG the first partial representation A, and is therefore particularly suitable for larger camera distances or lower image resolutions. Circular symmetrical areas 110A and HOB are therefore located on the representation media 600 designed as plates, whereby in the case of odd or negative point symmetry, the point reflection maps light to dark and vice versa, while in the case of even or positive point symmetry this reversal does not take place. If several patterns 610 are required, they can be designed to be distinguishable. This can be done via the arrangement of the centers of symmetry of the areas 110A and 110B, as can be seen in Fig. 6, in which the first partial representation A and the second partial representation B can be easily distinguished, or by means of the sequence of negative or odd and positive or straight point symmetries of regions 110A and HOB within respective patterns 610.

FIG. 7 shows schematic representations of representation media 600 with patterns 610 of predefined point-symmetrical areas according to embodiments. The patterns 610 correspond to or are similar to one of the patterns from FIG. 6, the patterns 610 being shown in the representation of FIG. 7 without graphical emphasis. Ten presentation media 600 similar to those of FIG. 6 are shown in FIG. 7 by way of example only.

FIG. 8 shows schematic representations of the display media 600 with the patterns 610 from FIG. 7 with graphical highlighting of patterns or predefined point-symmetrical regions 110A and HOB. In this case, patterns 610 with predefined point-symmetrical regions 110A and HOB are arranged or graphically highlighted on the ten display media 600, merely by way of example.

Thus, FIG. 7 and FIG. 8 show ten patterns 610 optimized for distinguishability only by way of example. Each pattern 610 has an individual arrangement of the odd and even point-symmetrical regions 110A and HOB. The patterns 610 are thus encoded by this arrangement. The codings were selected here and matched to one another and/or optimized by training in such a way that the ten patterns 610 are still clearly recognizable and distinguishable even if they are rotated or mirrored or partially covered by be captured by the camera. In the patterns 610 of FIGS. 7 and 8, the point-symmetrical regions 110A and HOB in the four corners of each display medium 600 are intentionally made slightly more conspicuous, respectively. This is irrelevant to the function itself, but offers practical advantages when manually assembling the display media 600 with the patterns 610. The display media 600 with the patterns 610 can be arranged as desired within the scope of the manufacturing process already described, e.g. B. three-dimensional or flat in a row or as a surface. The point symmetry centers of the patterns 610 can be found correctly and precisely within the scope of the provision method already described and/or by means of the provision device already described. For example, the patterns 610 may be printed on any size solid board, which may optionally be placed in a partially orthogonal relationship relative to one another. The centers of symmetry can also be detected sufficiently well in the case of a blurred image of the pattern 610 by the camera in order to enable the functions described. Thus, the detection of the point symmetry centers is robust to a blurred image. This expands the range of applications to situations in which work is carried out with a shallow depth of field, e.g. B. Scenes with little light, or in which the focus or autofocus of the camera is set incorrectly or a perfectly sharp image cannot be achieved, e.g. B. in liquid or cloudy or moving media or in the edge area of a lens or during a relative movement between pattern 610 and camera (motion blur, directional blur). Even if point symmetries occur naturally and in particular in human-designed environments, any incorrect detections based on them are spatially distributed differently than the detections based on correct patterns 610 and the two groups can therefore be easily separated or distinguished from one another.

To convey that the above provisioning method also works for non-planar and even elastic surfaces in motion, the patterns 610 of FIGS. 7 and 8 can be printed on paper, for example, and assembled into a flexible box. Even with non-flat or elastic surfaces (such as paper), the above-mentioned provision method works without any problems. This enables the Determination of movements of these surfaces. In contrast to many substances, paper does not allow shearing, but the point symmetry is also invariant to shearing, so this does not pose a problem.

In particular, a position of the centers of symmetry in the camera images can be determined precisely. However, it may also be of interest in various applications to extend this precise measurement to the entire area of the pattern 610. In other words, to indicate for each point or pixel of the pattern 610 where it is located in the camera image. This then allows e.g. B. to determine the smallest deviations between the actually observed pattern 610 and the ideal pattern according to the construction (ground truth). This is of interest, for example, if the pattern 610 is printed on a non-smooth or non-rigid surface and thereby z. B. variable folds or dents arise in the pattern 610, the exact shape of which is to be determined. In particular, random patterns are excellent for finding corresponding points from a first image to a second image. The first and second images can be recorded in chronological order with the same camera or by two cameras from different perspectives.

In particular, the case should now be considered when the first image is a real image from a camera and the second image is an artificially generated (stored) image of the given pattern, also called a reference image, which e.g. B. was placed in the second image based on the centers of symmetry found (e.g. scaled, rotated, mapped affinely, projected) so that it comes as close as possible to the real (first) image. With the reference image, processing steps that are necessary for the first image that comes from the camera, e.g. B. Image preprocessing steps. Then following known methods z. B. the optical flow or the disparity estimation can be applied to z. B. to find the correspondence in the reference image for each pixel in the camera image - or vice versa. This results in a two-step procedure: In the first step, the centers of symmetry found and, if applicable, the coding they contain are used to register or roughly align the real image with the known pattern. This then represents the initialization to, in the second step, e.g. B. using optical flow methods to precisely determine the smallest deviations in the sense of local shifts between the registered real image and pattern, if necessary for each point or pixel of the image or pattern 610. The computational effort required for the second step is all the smaller , the smaller the search area. Here it is usually very small - due to the good initialization from the first step. Since both steps require little computing effort, a high pixel throughput is achieved on standard computer platforms, which is defined as the product of the frame rate [images/s] and image size [pixels/image]. If no match can be found in places, this can usually be explained by the fact that the view of the pattern 610 is obscured by an object. From this, conclusions can be drawn about a shape or an outline of the obscuring object.

The reference image should be provided for the two-step procedure mentioned above. This can be solved in such a way that the associated reference image is kept available in a memory for all patterns 610 in question. The associated memory overhead can be reduced by only storing the respective parameters that are necessary to recalculate or generate the reference image if required. The patterns 610 can, for example, be generated according to simple rules using a quasi-random number generator. The word "quasi" here indicates that the random number generator actually works according to deterministic rules and its results are therefore reproducible, which is an advantage here. Rules are to be understood here, for example, as to the diameter of the symmetrical areas 110A and HOB and how the mirroring is to be carried out and how the pattern 610 is composed of a plurality of patterns with different degrees of detail, e.g. B. so that it can be easily detected at short, medium and large distances. It is then sufficient to store only the initialization data (seeds) of the quasi-random number generator and, if applicable, the selection of the rules for the construction of pattern 610. By means of this formation rule, the reference pattern can be generated again and again and identically (and then deleted again) if required. In summary, the two-step procedure can be represented, for example, as follows. In the first step, the symmetry centers are found and their signs determined. The sign here means the case distinction between odd and even symmetry. By comparing the sequence of signs, it is determined which of several patterns is involved. The sequence of signs of the pattern 610 can also be referred to as a code. This can be described in a compact manner and requires a maximum of 64 bits for a pattern 610 with, for example, 8×8 centers of symmetry. For comparison purposes, all existing or candidate codes should be stored. From this set, the code is sought that matches the observation with as few contradictions as possible. This result is usually unambiguous. This search is still possible even if the camera z. B. can only detect a part of the pattern 610 due to occlusion, because the code in this example with 8 × 8 centers of symmetry offers a very large number of up to 2 ⁶⁴ possibilities, while the number of patterns 610 produced will be much smaller, see above that there is a high level of redundancy. For each stored code, the information such as parameters and selection of rules necessary to generate the reference image should also be stored. This is generated for the second step, e.g. B. as needed, i.e. only at the time when it is needed, and possibly only temporarily.

Based on the positions of the centers of symmetry in the coordinates of the camera image found in the first step and the known, corresponding positions in the reference image, a transformation rule can be calculated that maps these coordinates into one another as well as possible, for example with a projective or affine mapping, which in the sense of the least squares optimization. With this transformation and suitable filtering of the image data, the two images can be transformed (warped) into a common coordinate system, e.g. B. in the coordinate system of the camera image or in the coordinate system of the reference image or in any third coordinate system. The two images, which are already aligned with one another, are then compared more precisely, e.g. B. with optical flow methods. For example, the best corresponding pixel with surroundings of the second image is sought for each pixel of the first image, preferably taking into account its surroundings. The relative displacement of corresponding positions can be expressed as displacement information, in particular as absolute coordinates or as a displacement vector. Such a displacement vector can be determined with sub-pixel accuracy, the correspondence is then usually not on the pixel grid, but in between. This information allows highly accurate analyzes of the entire area of the pattern 610 captured in the camera image, e.g. B. to analyze deformations or distortions of the pattern 610 or its carrier/representation medium 600 with elastic patterns or deviations of the image in the optical path in the case of rigid patterns.

If a sought-after correspondence cannot be found in the expected area, a local occlusion of the pattern 610 can be inferred. The cause of the masking can e.g. B. an object lying on the pattern 610, or a second pattern that partially obscures the first pattern. Valuable information can also be obtained from this occlusion analysis, e.g. B. masks or outlines of the objects.

FIG. 9 shows schematic representations of predefined point-symmetrical regions 110A and HOB according to exemplary embodiments. Each of the predefined point-symmetrical areas 110A and 110B corresponds to or is similar to the predefined point-symmetrical area from one of the figures described above. A first partial view A shows a second point-symmetrical or even point-symmetrical area HOB including its center of symmetry 112 and a second partial view B shows a first point-symmetrical or odd point-symmetrical area 110A including its center of symmetry 112 . The predefined point-symmetrical areas 110A and HOB here represent areas formed with gray levels.

The use of point symmetry has the advantages over other forms of symmetry that it is retained when the pattern and/or the at least one predefined point-symmetrical area is rotated about the viewing axis, that when the pattern and/or the at least one predefined point-symmetrical area is tilted, i.e under oblique perspectives, is also preserved. Rotation and tilting of the pattern / or at least one predefined point-symmetrical range do not pose a problem for the detection of odd and even point symmetries, because these are retained. As a result, the method for providing or providing method already mentioned above is also suitable for oblique perspectives of the pattern or the at least one predefined point-symmetrical area. In the case of straight point symmetry, for example, a gray or color value is retained in point reflection.

Point-symmetrical to the center of symmetry 112 in the first partial representation A of FIG. 9, one finds the same partner gray value gpG=g for each gray value g. Odd point symmetry is shown in the second partial representation B of FIG. 9, with the gray value being inverted in each case: for example, white becomes black and vice versa, light gray becomes dark gray and vice versa. In this example with gray values g in the interval 0<g<1, the point-mirrored gray values gpu were formed in the simplest possible way according to gpu=1-g from the original gray values g from one half of the region 110A shown in the representation of FIG. 9 at the top. Non-linearities can also be integrated into the inversion, for example gamma corrections, e.g. B. to compensate other non-linearities in the image display and image acquisition. The formation of suitable odd or even point-symmetrical patterns is correspondingly easy. For example, the half of the respective area 110A or HOB shown at the top in the illustration in FIG. 9 is defined arbitrarily or generated at random. The half shown below in the representation of FIG. 9 then results from this, namely by point reflection with inversion of the gray values for odd point symmetry or without inversion of the gray values for even point symmetry.

This consideration or generation can also be extended to include colored patterns and/or predefined point-symmetrical areas. With odd point symmetry, the point-mirrored RGB values can be formed by inverting the individual original RGB values, which is the simplest option, i.e. rpu = 1 - r (red), gpu = 1 - g (g stands for green ), bpu = 1 - b (blue). For example, dark violet maps to light green and blue maps to orange. Colored patterns can represent more information than monochrome patterns, which can be advantageous. precondition The use of this advantage is that the color information is also used in the conversion of the original image, ie the color image of a camera or another imaging sensor, into descriptors.

A specific embodiment of the pattern 610 and/or the at least one predefined point-symmetrical region 110 or 110A and/or HOB will also be discussed below with reference to the figures described above.

With regard to an arrangement of the pattern 610 and/or the at least one predefined point-symmetrical region 110 or 110A and/or 110B, such as e.g. 6, the point-symmetrical regions 110 or 110A and/or HOB can be circular, for example, and these can in turn usually be arranged in a regular grid in the pattern 610. For example, areas can remain unused between the circular areas 110 or 110A and/or HOB. There are alternatives to this: For example, the areas 110 or 110A and/or HOB can be square and connect to one another without gaps, so that the entire area is used, or the symmetrical areas 110 or 110A and/or HOB can be regular hexagonal areas that also connect to each other without gaps, so that the entire area is used.

In this context, FIG. 10 shows a schematic representation of a pattern 610 of predefined point-symmetrical regions 110A and HOB according to an exemplary embodiment. The predefined point-symmetrical areas 110A and HOB correspond to or are similar to the predefined point-symmetrical areas from FIG. 1, FIG. 6 and/or FIG. 8. The areas 110A and HOB in FIG. 10 are each circular and are arranged on a hexagonal grid. In this case, a distance between grid points or centers of symmetry can correspond to the circle diameter. Thus, an unused area 1010 between the areas 110A and HOB in the pattern 610 can be minimized.

Other arrangements and shapes, e.g. B. rectangles, polygons, etc., are possible, which are also combined with each other in shape and / or size be able. For example, an alternation of pentagons and hexagons like on an ordinary soccer ball. The shapes can also be arranged differently, e.g. B. rotated, possibly with asymmetrical areas in between. It is also possible that the center of symmetry lies outside the point-symmetric region itself. This is the case, for example, with a circular ring as the shape. It is also not necessary for all point-symmetrical areas to lie in a common plane. Instead, they can lie on different surfaces arranged in space, which may also be uneven.

The pattern 610 and/or the at least one predefined point-symmetrical region 110 or 110A and/or HOB can be formed in a wide variety of ways. Only a few examples are given below. Random pattern or quasi-random pattern such as noise pattern. By including low spatial frequency components, these were formed in such a way that the camera still perceives them as sufficiently high-contrast noise patterns even at medium and large distances. So-called white noise, i. H. uncorrelated gray values would be unsuitable for this. Aesthetic, possibly regular patterns, such as floral patterns, tendril patterns (leaves, twigs, flowers), ornamental patterns, mosaics, mathematical patterns, traditional patterns, onion patterns, patterns of iconic symbols (hearts, etc.). Imitation of random patterns from nature, e.g. B. farmland, forest floor, lawn, pebble beach, sand, bulk material (gravel, salt, rice, seeds), marble, rubble stone, concrete, brick, slate, asphalt surface, starry sky, water surface, felt, hammered paint, rusty sheet iron, sheepskin, scattered particles , etc. Photos of scenes with any content. In order to generate a point-symmetrical area and/or a pattern from such a pattern that is suitable for the purposes mentioned here, one half of the respective area is specified arbitrarily and the second half is constructed by point reflection and, if necessary, inversion of the gray values or color values. See also Fig. 9 as a simple example.

With regard to the material, surface and production of the pattern 610 and/or the at least one predefined point-symmetrical region 110 or 110A and/or 110B, there are numerous possibilities. The following list is not exhaustive: black and white, grayscale or Multi-color printing on a wide range of materials, printing on or behind glass or transparent film, printing on or behind frosted glass or semi-transparent film, relief in stone or glass or plastic or rubber, relief in fired materials such as earthenware, terracotta or ceramics, relief casting in metal or concrete or plaster, embossing in plastic or paper/cardboard, etching in glass or metal or ceramic surfaces, milling in wood, cardboard, metal, stone etc., burnt surface in wood or paper, photographic exposure of paper or other materials, transient or Rotting or water-soluble patterns for short-term applications in plant material, ash, sand, wood, paper, on fruit, food shells, etc., representation as a hologram, representation on a monitor or display, possibly also changing over time, representation on an LCD film or other display film, possibly also changing over time, etc.

With regard to the possibility of manufacturing in relief, as in the case of milling, embossing, punching, etc., it should be noted that the area should be perceived by the camera as odd and/or evenly symmetrical. It may therefore be necessary to include e.g. B. a later illumination, such as oblique incidence of light on a relief, and non-linearities in the optical image and other disturbances to be considered. It is not essential that the 3D shape or relief itself has the even and/or odd type of point symmetry, but that the image captured by the camera shows this symmetry. The incidence of light or the direction of illumination and the reflection of the light on the surface are also relevant and should be taken into account in the design. With regard to image recording and illumination, it should be noted that the recording technique should be designed to be suitable for recording the pattern 610 and/or the at least one predefined point-symmetrical area 110 or 110A and/or HOB. In particular, in the case of fast relative movements between pattern 610 and/or area(s) 110 or 110A and/or 110B and camera, it is advisable to use suitable lighting (e.g. flashlight or stroboscopic light or bright LED light) so that the exposure time and thus the motion blur in the image can be kept small. For various applications, it makes sense to apply patterns 610 and/or area(s) 110 or 110A and/or HOB to a transparent or semi-transparent surface to bring up. This allows pattern 610 and/or area(s) 110 or 110A and/or HOB to be illuminated from one side and observed from the other side. With this solution, disruptive reflections from the light source on the display medium can be effectively avoided. For the arrangement of pattern 610 and/or region(s) 110 or 110A and/or 110B, light source and camera, there is always the freedom to choose the front or the back of the carrier or display medium. When choosing, the risk of contamination of sample 610 and/or area(s) 110 or 110A and/or HOB or camera or wear and tear of sample 610 and/or area(s) 110 or 110A and/or HOB can also be a Play role: So it can z. B. make sense to attach pattern 610 and / or area (s) 110 or 110A and / or HOB and camera on the back because they are better there z. B. be protected from dust or water or because pattern 610 and/or area(s) 110 or 110A and/or HOB is or are protected there from mechanical wear and tear.

The subsequently published DE 10 2020 202 160 discloses a method that is also used in exemplary embodiments in order to find symmetrical regions or patterns in an image reliably and with very little computing effort. Here, the original image, i. H. a color image or grayscale image from a camera or another imaging sensor, is converted into an image of descriptors, with a descriptor being formed based on a local environment of the original image. The descriptor is another form of representation for the local image content that prepares it in a form that is easier to process. Under simpler is to be understood here, among other things: contains information about the environment of the point and not only about the point itself, extensive invariance with respect to brightness or illumination and their change and lower sensitivity to noise of the sensor. The descriptor image can have the same resolution as the original image, such that there is approximately one descriptor per pixel of the original image. Other resolutions are also possible, alternatively or additionally.

A signature is formed from the respective descriptor, which is represented in a computer unit as a binary word, or from a number of adjacent descriptors, which signature represents the local environment of the pixel of the original image describes as characteristically as possible. The signature can also be identical to the descriptor or a part of it. The signature is used as an address to access a lookup table. If the signature consists of N bits, a look-up table of size 2N (ie: 2 to the power of ^N ) can be accessed. It is advantageous not to select a word length N of the signature that is too large, since the memory requirement for the table grows exponentially with N: for example 8 < N < 32. The signature or the descriptor is constructed in such a way that signature symmetries with simple operations, e.g. B. bit-wise XOR (exclusive or) on a part of the bits can be determined. Example: SP = s ^A RP, where s is a signature of length N bits and RP is a matched reflector (R) for point symmetry (P). The character ^A stands for the bitwise XOR operation. The signature SP stands for the point-symmetrical counterpart of the signature s. This relationship also applies in the opposite direction.

If the construction of the descriptor or signature is fixed, the reflector is automatically fixed (and constant). By applying it to any signature, it can be transformed into its symmetric counterpart. There is an algorithm that finds one or more symmetrical signature pixels at the current pixel for the given signature within an optionally restricted search window. The center of symmetry then lies in the middle of the connecting line between the positions of these two pixels. A voting weight is cast there or as close as possible and collected in a voting matrix (voting map). In the voting matrix, the weighted voting weights accumulate at the locations of the sought-after centers of symmetry. These can thus be found, e.g. B. by searching the voting matrix for accumulation points. This works for point symmetry, horizontal axis symmetry, vertical axis symmetry, as well as other symmetries if required, e.g. B. mirror symmetry on other axes, as well as rotational symmetry. An even more precise localization with sub-pixel accuracy is possible if the local surroundings are also included in the analysis when evaluating the coordination matrix for determining the accumulation points and precise localization of the centers of symmetry. 15 of DE 10 2020 202 160 illustrates an algorithm which can find point-symmetrical correspondences to the signature currently being considered. However, only straight point symmetries were taken into account.

According to embodiments, this algorithm is extended to odd point symmetries. It is particularly advantageous that the odd and even symmetries can be determined simultaneously in just one common pass. This saves time because only one pass through the signature image is required instead of two, and latency. When only one pass (rather than two) is required, stream mode processing can provide the result of the symmetry search with much lower latency. Processing starts as soon as the first pixel data arrives from the camera and the processing steps are carried out in quick succession. This means that the signature is already calculated as soon as the necessary image data from the local environment of the current pixel are available. The search for symmetries is immediately carried out for the signature just formed. As soon as parts of the voting matrix are finished, which is the case when they are no longer part of the search area and will no longer be, they can be evaluated immediately and any symmetries found (strong centers of symmetry) can be output immediately. This procedure leads to very low latencies, which typically correspond to only a few image lines, depending on the height of the search area. Low latencies are important if you want to react quickly, e.g. B. within a control loop, in which an actuator influences the relative pose between the symmetry object and the camera. Memory can also be saved. The voting matrix (voting map) can be used jointly for both forms of symmetry, ie even point symmetry and odd point symmetry, with the two forms of symmetry or types of symmetry with different signs participating in the vote, e.g. B. Subtraction of the voting weight in case of odd point symmetry and addition of the voting weight in case of even point symmetry. This is explained in more detail below. Furthermore, energy can also be saved by saving memory. The low-latency implementation option described above also means that only a small amount of intermediate data needs to be stored, in contrast to entire images. This Working with little memory is particularly important for cost-critical embedded systems and also leads to savings in energy consumption.

11 shows a schematic representation for using a lookup table 1150 according to an embodiment. The look-up table 1150 is usable by the determining means of the provisioning apparatus of FIG. 1 or a similar apparatus. In other words, an exemplary embodiment of an algorithmic procedure when searching for point-symmetrical correspondences in FIG. 11 is a snapshot in connection with the device for providing from FIG. 1 or a similar device and/or the method for providing from FIG. 3 or a similar procedure shown.

In particular, the representation in FIG. 11 is also similar to FIG. 15 of the published DE 10 2020 202 160, with FIG. 11 additionally including an extension to include even and odd point symmetry.

The lookup table 1150 may also be referred to herein as an entry table. A pixel grid 1100 is shown, in which a signature s with the exemplary value 2412 was generated for a currently viewed or processed pixel. In other words, FIG. 11 shows a snapshot during the formation of a concatenation of pixels or pixel coordinates with an identical signature s. For better clarity, two of up to N possible chains are shown, specifically for the signature SPG=364 and for the signature SPU=3731. A reference to the position of the last predecessor signature that had the same signature value can be stored per pixel in the pixel grid 1100 . This creates a concatenation of positions with an identical signature. The signature values themselves therefore do not need to be stored. A corresponding entry position in the pixel grid 1100 is stored for each signature value in the lookup table 1150 or entry table with N table fields. In this case, N corresponds to a number of possible signature values. The stored value can also be "invalid". The contents of the lookup table 1150 or entry table and the image of references (linking image) change dynamically. a z. B. line-by-line processing pixel by pixel in the pixel grid 1100 began z. 11, as illustrated by an arrow, and has currently progressed to a pixel that has the signature s=2412. Concatenations between pixel positions each with an identical signature s are only stored for a first image area 1101 . For a second image area 1102 in the lower part of the image, the concatenations and signatures are not yet known at the time shown, and for a third image area 1103 in the upper part of the image, the concatenations, e.g. B. due to a limitation of a search area, no longer needed, wherein a concatenation memory for pixels in the third image area 1103 can be released again.

For the signature s just formed, the just point-mirrored signature SPG=364 is formed by using a reflector RPG. The index PG stands for point symmetry, straight line. In the following, the index PU is also used, which stands for point symmetry, odd. This value is used as an address into look-up table 1150 to enter the concatenation of pixel locations associated with the same signature value SPG=364. At the time shown, the look-up table 1150 comprises two elements: the entry pixel positions for the respective signature s together with the references thereto illustrated by curved arrows. Other possible contents of the lookup table 1150 are not shown for the sake of clarity. The concatenation for the signature value SPG=364 comprises three pixel positions shown here merely by way of example. Two of them are in a search area 1104, which can also have a different form than that shown here, e.g. B. rectangular or circular. Here, when unidirectionally traversing along the concatenation, starting from the bottom, two point-symmetric correspondence candidates that lie within the search region 1104 are found. The third correspondence, which is the first element of the even point symmetry concatenation, is not of interest here because it is outside the search region 1104 and thus too far from the current pixel position. If the number of symmetry center candidates 1112 is not too large, a voting weight for the position of the respective symmetry center can be cast for each symmetry center candidate 1112 . The symmetry center candidate 1112 is in the middle on the Connection axis between the position of the signature s and the respective straight point mirrored signature SPG. If there is more than one symmetry center candidate 1112, the voting weight can be reduced in each case, for example the reciprocal of the number of symmetry center candidates can be used as the respective voting weight. As a result, ambiguous symmetry center candidates are weighted less than clear symmetry center candidates.

Consideration and use of an odd point-mirrored signature will now be made. In the snapshot shown in FIG. 11, the odd point-mirrored signature SPU=3731 is formed for the signature s just formed by using a further reflector RPU. Analogously to the sequence described above for the even point-mirrored signature, the same steps are also carried out for the odd point-mirrored signature. The entry into the corresponding concatenation is found via the same lookup table 1150 . Here, the lookup table 1150 refers to the concatenation shown for odd point symmetry for the signature 3731. The first two pixel positions along the concatenation again lead to the formation of symmetry center candidates 1112 because they are located in the search region 1104 and because the number of symmetry center candidates 1112 is not too large . The last pixel position along the concatenation lies in the third image area 1103. This area is no longer required at all, since it can no longer come into the search area 1104, which is sliding line by line here.

If the next link within a link points to the third image area 1103, the snaking along the link can be terminated. Of course, the shimmying is also terminated when the end of the chain is reached. In both cases it makes sense to limit the number of symmetry center candidates 1112, ie to discard all of them if there are too many competing symmetry center candidates 1112. Furthermore, it makes sense to terminate the shimmy along a link prematurely if, after a specified maximum number of steps along the link, neither its end nor the third image area 1103 could be reached. In this case, too, all symmetry center candidates 1112 found up to that point should be discarded. Memory for the concatenation in the third image area 1103 can already be released again, so that concatenation memory only needs to be reserved for the size of the first image area 1101 . The concatenation memory requirement is therefore low overall and essentially depends here only on one dimension of the search area 1104, here the height of the search area, and on one dimension of the signature image, here the width of the signature image.

It may be that a symmetry center candidate 1112 or candidate for a symmetry center does not always fall exactly on a pixel position, but there are three other possibilities. There are four options in total:

1. Point or symmetry center candidate 1112 falls on a pixel position.

2. Point or symmetry center candidate 1112 falls in the middle between two horizontally directly adjacent pixel positions.

3. Point or symmetry center candidate 1112 falls in the middle between two vertically directly adjacent pixel positions.

4. Point or symmetry center candidate 1112 falls in the middle between four directly adjacent pixel positions.

In the ambiguous cases 2. to 4., it is advantageous to distribute the voting weight to be cast evenly over the pixel positions involved. The voting weights given are entered in a voting matrix and added up or accumulated therein.

Not only positive, but also negative voting weights are used at the same time. In particular, the even symmetries are given a different sign, here positive, than the odd symmetries, here negative. This leads to clear results: In image areas without symmetries, which in practice can usually represent the majority, positive and negative voting weights balance each other out and thus cancel each other out in the voting matrix. On average, there is about a zero in the voting matrix. In contrast, in odd symmetrical or even symmetrical areas, there are strong extremes in the voting matrix, specifically in in this exemplary embodiment, negative-valued minima in the case of odd point symmetries and positive-valued maxima in the case of even point symmetries.

According to the exemplary embodiment shown here, the same resources are used for the odd and even point symmetries, i.e. lookup table 1150 or entry table, chaining diagram, matching matrix, which in particular saves memory requirements, and both forms of symmetry or types of symmetry are considered in a joint run, which saves time and cache saves.

12 shows a schematic diagram 1200 for a voting matrix according to an embodiment. The diagram 1200 relates to a matching matrix as a 3D plot for a camera image processed by the device for providing of FIG. 1 or a similar device, in which the pattern from the second partial representation of FIG. 6 was recorded by a camera. Three maxima 1210B and five minima 1210A, which stand for the three even point-symmetrical and five odd point-symmetrical areas of the pattern from the second partial representation of FIG. Outside of these extremes, the values in the tuning matrix are close to zero. The extrema can thus be determined very easily and the positions of the centers of symmetry in the camera image can be determined clearly and precisely.

FIG. 12 shows that these extremes are very clearly pronounced and can therefore be detected easily and unequivocally for the device for providing from FIG. 1 or a similar device and/or the method for providing from FIG. 3 or a similar method. The information about the type of symmetry, ie odd or even, is contained in the sign. A high-precision determination of the position of the centers of symmetry with sub-pixel accuracy is possible if the local surroundings of the respective extremum are also taken into account when evaluating the coordination matrix. Appropriate methods for this are known to those skilled in the art. If the patterns are properly constructed, odd and even point symmetries do not compete with each other. Then an image area has, if at all, either the odd or the even point symmetry form. Even if odd and even If point-symmetrical areas are close together in the camera image, it can be ensured that their centers of symmetry remain spatially separated or distinguishable from one another. Then, by treating negative and positive symmetries together, there are advantages in terms of resources and speed.

According to an embodiment, a separate treatment of odd and even point symmetry can be provided. It makes sense to split them up before making entries in the voting matrix: Instead of a common, signed voting matrix, two unsigned voting matrices are then provided, with the voting weights for negative symmetries being entered in the first and the voting weights for positive symmetries in the second voting matrix. This gives rise to a potentially interesting advantage: Patterns can also be constructed and taken into account by the detection algorithm which have odd and even point symmetry at the same time and whose center of symmetry coincides locally. While such a mixed symmetry form is very unusual, this unusualness ensures that confusion with random patterns occurring in the images is extremely unlikely. The two matching matrices are then to be searched for maxima that are present at the same location in both matrices. Another possible advantage of treating odd and even point symmetry separately is that it is easier to parallelize and, as a result, it can be executed more quickly. Because by using two voting matrices, access conflicts when entering the voting weights can be avoided, which saves waiting times.

Fig. 13 shows a schematic representation of patterns 610 arranged in the form of a cube, for example, according to an embodiment with regard to correct identification of a grid 1311. The patterns 610 shown in Fig. 13 are, for example, patterns from Fig. 7 or Fig. 8, three of which are here arranged in a cube shape. Detected or identified centers of symmetry 112A and 112B of the respective predefined point-symmetrical areas of the patterns 610 are shown for the patterns 610, with signs and values of the associated extrema in the voting matrix may be known. First centers of symmetry 112A are assigned to predefined point-symmetrical areas with odd point symmetry and second centers of symmetry 112b are assigned to predefined point-symmetrical areas with even point symmetry. A correct grid 1311 is drawn in for one of the patterns 610, on which the predefined point-symmetrical areas and thus the centers of symmetry 112A and 112B are aligned. The correct grids are to be searched for the other two patterns 610 , incorrect solutions of the grid search being illustrated by first markings 1313 and correct solutions of the grid search being illustrated by second marking 1314 in FIG. 13 .

Finding the associated correct grids is an ambiguous task. After the odd/even coded symmetry centers 112A and 112B have been detected, the next step is often to group them and determine which pattern 610 this group belongs to, since it is not always known in advance which and how many patterns 610 are included in the picture. Part of this task can be finding the grid 1311 on which the symmetry centers 112 A and 112 B are arranged. Instead of a square grid 1311, other topologies for the arrangement of the centers of symmetry 112 A and 112 B are also possible, e.g. B. annular concentric arrangements, see e.g. B. the second partial representation in Fig. 6. As a representative, square grids 1311 are considered below.

The task of determining the position of the correct grid for all patterns 610 solely from the positions of the centers of symmetry 112A and 112B in FIG. 13 presents an ambiguous problem under certain circumstances Grid 1311 is already drawn in, it is not difficult (for the viewer) to specify the correct grid 1311. However, with the other two patterns 610 captured by the camera from a much more oblique perspective, it is clear that the output can be ambiguous. There are several possible solutions as to how a grid could be placed through the symmetry centers 112A and 112B. The solution that is most obvious when viewed locally, namely the one with almost vertical axes, is not the right one, as is the case with the first one Markings 1313 can be seen. In contrast, the second markings 1314 lie correctly on the grid. This shows that a naive approach, e.g. B. searching for the nearest neighbors of the respective center of symmetry, may lead to an incorrect solution in the case of an oblique perspective. In practice, solutions with a very oblique perspective are ruled out because the centers of symmetry 112A and 112B can then no longer be found.

FIG. 14 shows a schematic representation of the pattern 610 shown in the first partial representation of FIG. 6 in an oblique perspective. In a first partial representation A, the representation medium 600 with the pattern 610 made up of the predefined point-symmetrical regions 110A and HOB is shown in FIG. 14 . A second partial illustration B in Fig. 14 shows the centers of symmetry 112A and 112B of the pattern 610 identified or detected by means of the device for providing from Fig. 1 or a similar device and/or the method for providing from Fig. 3 or a similar method shown. The symmetry centers 112 A and 112 B have been detected and at least their positions are available.

FIG. 15 shows the pattern 610 from the first partial representation of FIG. 14 with an emphasis on a predefined point-symmetrical region 110B. A predefined straight point-symmetrical area HOB is graphically emphasized here merely by way of example in order to illustrate a distortion of the pattern 610 or of the areas 110A and HOB as a result of the oblique perspective. The point-symmetrical regions 110A and HOB, which are predefined as circular here by way of example, are distorted into ellipses by the oblique perspective.

A reconstruction of the correct grid or topology of the pattern 610 is discussed below with particular reference to FIGS. 14 and 15 and general reference to the figures described above.

From an oblique perspective, each circular area 110A and HOB, from which the voices for the respective center of symmetry 112A and 112B originate, becomes an ellipse. By tracing back the voices belonging to the respective center of symmetry 112A, 112B, for example the one highlighted in FIG Center of symmetry 112 B with straight point symmetry, have contributed, the shape and orientation of the respective ellipse can be deduced. The direction and relationship of the major axes of the ellipse reveal how it can be stretched or straightened to convert it back into a circle. Consider the exemplary highlighted predefined straight point symmetric region HOB of the pattern 610 that contributes to the highlighted center of point symmetry 112B. By design, this area 110B is circular or approximately circular, e.g. B. hexagonal. From an oblique perspective, this circle becomes an ellipse. When matching to identify the center of symmetry 112B, symmetrical pairs of points contribute to forming the extremum in the matching matrix that lies within this ellipse.

According to one exemplary embodiment, it is traced back to where in the camera image the pairs of points that have led to the formation of a sufficiently strong extremum come from. A further processing step is carried out for this purpose. It is initially assumed that the tuning has already taken place and that the sufficiently strong symmetry centers have already been found. The starting point is therefore a situation as shown in the second partial representation B of FIG. The voting process is then repeated in a modified form. However, the already existing reconciliation matrix is not created again. Instead, for each pair of symmetrical points that would contribute to the matching matrix, it is checked whether the contribution would contribute to one of the found centers of symmetry 112A, 112B and thus has already contributed in the first pass. If this is the case, the two positions of the pair of points are saved or processed immediately. Advantageously, the index of the center of symmetry 112A, 112B, to which the symmetrical pair of points contributes, is also stored or used. In this way, all contributions to the successful symmetry centers can be subsequently determined and (temporarily) stored or reused.

The beginning of the further processing step does not have to be the end of the first processing step, ie the formation of the voting matrix and Determination of the centers of symmetry, but it can be started beforehand and the already finished intermediate results, ie found centers of symmetry 112A, 112B, of the first processing step can be used. In the information formed in this way, all image positions that contributed to each center of symmetry 112A, 112B found can then be read. These positions lie essentially or apart from a few outliers within the ellipse, as shown in FIG. 15 by way of example for a center of symmetry 112B.

The person skilled in the art is familiar with methods for determining the parameters of this ellipse. For example, a major axis transformation can be formed over the set of all points contributing to a center of symmetry 112A, 112B in order to determine the orientation of the major axes and the two diameters of the ellipse. This is even possible without the contributing image positions needing to be temporarily stored: Instead, they can be charged immediately after they become known. Alternatively, the elliptical envelope around the set of points can also be determined, with which the largest possible part of the set of points, excluding any outliers, is enclosed as closely as possible.

Alternatively, instead of storing a set of points in the sense of a list, an index image, equivalent to an index matrix, can be created. It serves the same purpose, which is to form the parameters of all ellipses, but it stores the information in a different way. The index image ideally has the same size as the signature image and is set up to store indices, specifically the indices assigned to the centers of symmetry 112A, 112B found. A special index value, e.g. B. 0, is provided to indicate that there is no entry yet. If, when running through the further processing step, a symmetrical pair of points or pair of signatures is found that contributes to an i-th index, then the index i is entered at the two associated locations of the respective signatures. An index image is thus obtained at the end of the run, in which all the indices assigned to the centers of symmetry 112A, 112B occur multiple times, these forming elliptical areas: apart from a few outliers, each elliptical area then only contains entries a uniform index, as well as the index 0 at the unused positions. The index image can then be easily evaluated to determine the parameters of each ellipse. Otherwise, it is not necessary to save the entire index picture. As soon as the data no longer changes in a section of the index picture, this section can already be evaluated and the memory can then be released again. This also leads to lower temporal latency, so that intermediate results can be provided earlier.

With the known ellipse parameters, the two-dimensional arrangement of the detected centers of symmetry, see FIG. 14, can then be corrected in such a way that they subsequently lie on the grid of the pattern 610, which is at least approximately square here only as an example.

FIG. 16 shows a schematic representation of the pattern 610 from FIG. 15 after a perspective rectification according to an embodiment. In other words, for the sake of illustration, FIG. 16 shows the pattern 610 from FIG. 15 after it has been stretched normal or perpendicular to the direction of the found ellipse or the highlighted elliptically distorted area HOB by the ratio of the two main axis lengths. Thus the correct grid 1311 can be found in a simple manner. The ellipse is thus rectified in comparison with FIG. 15 in such a way that the original circular shape of the region 110B is restored. It is then easy to determine the grid 1311 on which the centers of symmetry 112A and 112B lie, or to determine the neighboring relationships between the centers of symmetry 112A and 112B without errors. Fig. 16 is for illustrative purposes only. In practice, it is not necessary to warp the image. Since the information about the positions of the symmetry centers 112A and 112B is already available in a condensed form, it makes sense to continue working only with this data and to transform its coordinates, with the transformation rule being formed from the determined ellipse parameters in such a way that the ellipses become circles.

In the case of camera images that are recorded with a telephoto focal length, one global transformation per subsection may be sufficient to create the raster 1311 to determine. In the case of camera images recorded with a wide-angle lens (e.g. fish-eye lens), local transformations can be used at least in some areas. Thus, the transformation rule mentioned above can be applied globally and/or locally. In the global variant, all projection centers are transformed using the same, common transformation rule. This is useful and sufficient in many cases. The common transformation rule can be formed from the joint consideration of all ellipses. If the symmetry centers 112A and 112B lie spatially on several surfaces, the ellipses can be classified into groups according to their parameters. The ellipses belonging to a surface have very similar parameters - especially if the surface is flat. A global transformation rule can then be determined and applied for each group. This procedure is appropriate for telephoto lenses. The local transformation makes sense when the majority of the circular areas are imaged in differently shaped or differently oriented ellipses by the imaging of the camera. This is especially the case with wide-angle cameras or lenses with high distortion.

After applying the transformation, the symmetry center positions belonging to the same face are at least approximately on a common grid 1311. The next task is to associate the symmetry centers 112A and 112B with the grid positions. This can e.g. B. be carried out iteratively in small steps. For example, for a center of symmetry 112A, 112B, up to four nearest neighbors are searched that have approximately the same distance, see also the markings from Fig. 13. From the neighbors you work your way to the other neighbors until all centers of symmetry 112 A and 112 A and 112 B, which belong to a pattern 610, are assigned to a common grid 1311 or can be excluded from it. If one encounters centers of symmetry during this search that do not match the grid 1311 just considered in terms of distances, these are not recorded since they are probably outliers or centers of symmetry that belong to other areas. This iterative search can be repeated for the other faces, so that in the end, apart from the outliers, each Center of symmetry 112A, 112B is associated with a surface. The patterns 610 can then be identified for the surfaces, preferably using the binary coding associated with the centers of symmetry 112A and 112B, which is contained in the sign of the extremum in each case.

17 shows a schematic representation of an embodiment of a pattern 1710 with hierarchical symmetry. The pattern 1710 corresponds or resembles a pattern from the figures described above. More specifically, pattern 1710 has a two-level hierarchy of four predefined point-symmetrical regions 110A and HOB, by way of example only. According to the exemplary embodiment illustrated here, the pattern 1710 has two predefined odd point-symmetrical regions 110A and two predefined even point-symmetrical regions 110B, merely by way of example. The pattern 1710 as a whole has an odd point-symmetrical structure. On a first hierarchical level are the even point-symmetrical areas HOB and the odd point-symmetrical areas 110A. The overall arrangement of the odd point-symmetrical pattern 610B is on a second hierarchical level. The center of symmetry 112 of the second hierarchical level is marked with the quartered circle.

18 shows a schematic representation of an embodiment of a pattern 1810 with hierarchical symmetry. The pattern 1810 in FIG. 18 is similar to the pattern from FIG. 17. More precisely, FIG. 18 shows a further example for a two-level hierarchy of predefined point-symmetric regions HOB. In the first hierarchical level, the predefined point-symmetrical areas HOB are each point-symmetrical. In the second hierarchy level, there is odd point symmetry at the level of pattern 1810, with center of symmetry 112 at the center of a six-point hexagon shown for illustrative purposes. The odd symmetry manifests itself here in an inversion of the predefined point-symmetrical areas HOB, e.g. B. A dark symbol on a light background is reflected in a light symbol on a dark background. 19 shows a schematic representation of an embodiment of a pattern 610 with hierarchical symmetry. In this case, the pattern 610 is constructed from the patterns 1710 and 1810 from FIGS. 17 and 18 or their inverted and/or point-mirrored form. The pattern 610 has a three-level hierarchy of two patterns 1710 from FIG. 17 and two patterns 1810 from FIG. 18, by way of example only. Patterns 1710 and 1810 are odd and thus inverted mirrored at the center of symmetry 112 of pattern 610 at the center of a sixteenth hexagon shown for illustrative purposes. For example, pattern 1710 shown in the lower right of Figure 19 is an inverted form of pattern 1710 in the upper left. This hierarchical principle can be continued at will, i.e. a fourth and fifth level can be constructed, and so on.

Patterns with hierarchical symmetry are discussed further below with reference to FIGS. 17, 18 and 19. FIG. The symmetrical patterns 610, 1710, 1810 can be constructed in multiple stages, so that there are smaller intrinsically symmetrical areas in a first hierarchical stage, for example, which when viewed together result in symmetry at the next higher hierarchical stage. FIG. 17 and FIG. 18 each show an example of how a two-stage hierarchical pattern 1710 or 1810 can be constructed. Based on this, a three-level hierarchical pattern 610 is constructed in FIG. The example from FIG. 19 therefore contains three hierarchical levels. The third hierarchical level extends over the entire surface of the pattern 610 (area framed by dashed lines) and includes the center of symmetry 112. In the second hierarchical level, there are the four patterns 1710 and 1810 (each framed by a solid line), each with a center of symmetry in the middle ( not explicitly mentioned here). In the first hierarchical level, there are consequently 16 predefined point-symmetrical regions, each with a center of symmetry, according to the exemplary embodiment illustrated here. The symmetry of the third hierarchical level is already visible from a greater distance. During an approach, the four symmetries of the second hierarchical level also become visible. At a shorter distance, or if the recorded resolution of the pattern 610 is sufficient, the symmetries of the first hierarchical level also become visible. Thus, for example, a visual control (visual servoing) z. B. a robot z. B. towards the pattern 610 or in any other direction over a large range of distances. It is generally not necessary to record coarser or higher hierarchical levels if finer or lower hierarchical levels can already be recorded. It is also not necessary to be able to capture all symmetries of the respective hierarchical level at the same time, for example if it is no longer possible to capture the entire pattern 610 in the camera image at a very short distance. It is obvious that even and odd symmetries can be freely chosen and combined. Additional information can also be contained in this definition, in particular one bit each for the choice between odd and even symmetry, with such additional information being able to be transmitted in this way to a detecting system. "Partially free" means here that the remaining part of the symmetry forms on the respective hierarchical level inevitably result from the next higher hierarchical level. In other words, z. B. in Fig. 18 for the top row, the patterns "X" and "O" can be freely selected. The second row is then inevitable, here with inversion, because a negative point symmetry is selected on the next hierarchical level.

20 shows schematic representations of patterns 610 according to exemplary embodiments. In a first partial depiction A, FIG. 20 shows a pattern 610 which is, by way of example, one of the patterns from FIG. 8 . The first partial representation A of FIG. 20 is an example of implicit additional information, here just an example of 8 • 8 = 64 bits, which results from the type of symmetry or the associated sign of the point symmetry of the predefined point-symmetrical regions 110A and HOB of the pattern 610 . In a second partial representation B, a pattern 610 is shown in FIG. 20, which is made up of four predefined point-symmetrical areas 110A and 110B, for example, here, for example, a predefined odd point-symmetrical area 110A and three predefined even point-symmetrical areas HOB on a square grid. Furthermore, a code matrix 2010 for explicit additional information is arranged in the pattern 610 here. The implicit additional information from the first partial representation A is explicitly contained in the code matrix 2010 purely by way of example. The predefined region 110A indicates odd point symmetry or marks the beginning line of the 8 • 8 matrix so that a readout order is clearly defined.

A transmission of implicit or explicit additional information is discussed in more detail below with reference to FIG. 20 .

It may be useful or necessary to use the pattern 610 to transmit additional information to a recipient, for example to a computer, autonomous robot, etc. The additional information can be more or less extensive. Some illustrative examples of additional information include stopping point, picking up charge, position at location 52°07'01.9"N 9°53'57.4"E facing southwest, turn left, speed limit 20 km/h, charging station for lawn mower, etc. For the There are various options for transmission by means of the imaging sensor or the camera. In particular, a distinction can be made between implicitly and explicitly contained additional information, see the two examples in FIG. 20, in which 64 bits of additional information are provided once implicitly and once explicitly. Implicit overhead means that it is somehow included in the symmetric patterns 610 themselves, while explicit overhead is generally designed and captured separately from those patterns 610.

One possibility for transmitting implicit additional information is illustrated using the first partial illustration A of FIG. 20: implicit additional information as binary code. Since the choice between odd and even point symmetry is made when constructing the pattern 610 for each symmetrical region 110A and HOB, additional binary information (corresponding to 1 bit) can thus be transmitted in each case. If you also allow patterns that are odd and even with point symmetry at the same time, the binary additional information becomes ternary additional information, i.e. three cases instead of two.

A further possibility for the transmission of additional information results from the use of non-uniform distances between the centers of symmetry of the areas 110A and 110B, ie implicit additional information based on the arrangement. Unlike the arrangement shown in FIG. 20, where the centers of symmetry lie on a square grid, they would then be arranged irregularly, with the additional information or part of it being encoded in this arrangement. Example: If the respective center of symmetry is allowed to be shifted by a fixed distance to the left/right and up/down, there are 9 possible positions, with which Iog2(9) = 3.17 bits of additional information per center of symmetry can be encoded. Diagonal perspectives between the imaging sensor and the pattern 610 do not pose a problem with any of the options mentioned. For example, some of the centers of symmetry (for example the outermost four in the corners) can be used to define a coordinate system or the regular basic grid. The deviations or binary/ternary codes used for coding then relate to this basic grid.

The symmetrical areas 110A and HOB for implicit additional information should not be too small so that sufficiently prominent extremes form in the matching matrix. If a large amount of additional information (in particular static, location-based) is to be transmitted to the recipient (e.g. a mobile robot), it can be advantageous to encode it explicitly.

The second partial illustration B of Fig. 20 shows how static, location-based additional information in particular can be explicitly transmitted to the recipient (e.g. a mobile robot): It can be agreed, for example, that in a coordinate system defined by the centers of symmetry at certain coordinates there is more information that z. B. are encoded in binary (black/white) or in further gradations (gray levels) or in colors. The procedure then consists of two steps: In a first step, a field is found using odd and even symmetries, for example the code matrix 2010, in which further information is encoded. In a second step, the field and thus the information contained therein is read out. Diagonal perspectives between the imaging sensor and the pattern 610 do not pose a problem, because the readout of the explicit additional information does not require that the base vectors of the coordinate system found are perpendicular to one another, nor that they are of the same length. Optionally, the image can also be corrected in such a way that then a Cartesian coordinate system is available. Optionally, a display can also be installed in the field with the pattern 610, which in addition to information that is static over time can also transmit information that changes over time and/or that transmits information about the change over time.

High-resolution additional information can also be contained in the pattern 610 itself, with implicit error detection. There is thus a further possibility of transmitting (in particular static, location-based) additional information via the pattern 610 itself: This means that the additional information is contained in the sequence of the black-and-white or colored or greyscale pattern 610 itself. With the aforementioned classification, this additional information would be implicit and explicit at the same time. Because the pattern 610 or at least parts of it has symmetries, the additional information is automatically contained redundantly, typically twice in each case. This applies to both odd and even point symmetry. This fact can be used for error correction or error detection. If, for example, a pattern 610 is soiled, for example by bird droppings, then it is possible with a high degree of certainty to detect the error that has arisen as a result in the additional information, because the same error is most likely not present at the associated symmetrical position.

21 shows a schematic representation of an application situation for the control device from FIG. 1 and/or the method for controlling from FIG 2165 on a platform 2170 of an offshore wind turbine 2175. At least one camera 102 is arranged on the gangway 2160 and/or the supply ship 2165. At least one predefined point-symmetrical area 110 and/or at least one pattern 610 from at least one predefined point-symmetrical area 110 is arranged on the offshore wind turbine 2175 in the area of the platform 2170 . The at least one predefined point-symmetrical region 110 corresponds to or resembles one of the predefined point-symmetrical regions described above from one of the figures described above. The pattern 610 corresponds to or resembles one of the above-described patterns from one of the previously described figures.

In other words, Fig. 21 shows a situation in which the supply ship 2165 with a movable gangway 2160, which can be moved by lifting, pivoting, pitching and/or telescoping, as illustrated by arrows, on the platform 2170 of the offshore wind power plant or the Offs h o re wind power plant 2175 should dock. For this purpose, the described areas 110 and/or patterns 610 are attached to the platform 2170, while cameras 102 are installed on the gangway 2160 and on the supply ship 2165, which monitor these areas 110 or patterns 610. Using the control device from Fig. 1 and/or the method for controlling from Fig. 4 or similar, the relative position and orientation between platform 2170 and gangway 2160 is continuously determined and this is adjusted in such a way that docking and undocking can be carried out reliably and gently and in meanwhile the position can be held.

This results in an opportunity to improve visual robot control or visual servoing with point-symmetrical (hidden) areas 110 or patterns 610. A mechanical actuator system, here the gangway 2160, is controlled on the basis of the detection of the symmetrical areas 110 or patterns 610 in images from cameras 102 or other imaging sensors, the actuation of the actuators leading to a relative movement between pattern 610 and camera 102 or sensor. In relation to the application of the pressure maneuver at sea shown and explained in FIG. 21, there is a challenge in that the supply ship 2165 is moved by waves and wind, while the platform 2170, which can also be floating, is also moved by waves and wind, but with a different dynamic. Nevertheless, a safe transfer of service personnel between the supply ship 2165 and the platform 2170 should be achieved, in all weather conditions if possible.

The described areas 110/or patterns 610 are attached on the side of the wind power plant or off-shore wind power plant 2175, in the direct vicinity of the docking point or platform 2170 and above it. The Sample 610 are passive, meaning they don't require an electrical connection or anything control. The necessary light for the night operation comes from the support ship 2165. The at least one camera 102 and associated image processing is located on the ship's side. For example, the camera 102 is mounted on the gangway 2160 and therefore moves with it. However, it can also be mounted further away from the docking point, e.g. B. on the rear part of a telescopic mechanism, because z. B. the extension path of the telescope is known and can be taken into account when calculating relative movements. For practical reasons, several cameras 102 are used, for example, which are mounted at different locations on the ship, so that there is always at least one camera 102 that has favorable viewing conditions (unobscured view, favorable distance, favorable viewing angle, no glare) of the pattern or patterns 610 . For the same reasons, several patterns 610 are used, for example, which can be designed in such a way that they are distinguishable from one another, as explained, for example, with reference to FIGS. 7 and 8 . A triangulation can be made possible with a plurality of patterns 610, which makes the determination of the relative orientation easier or more precise.

Since the distance between camera 102 and pattern 610 can vary over large areas during such a docking or undocking process, patterns 610 with hierarchical symmetry are used, for example, as explained, for example, with reference to FIGS. These include both a few large-area point symmetries that can still be detected from a great distance and in fog, rain or snow, as well as many small-area point symmetries that are used for precise operation at short distances.

In the general case, the relative motion between a camera and a sample has six degrees of freedom, namely three degrees of rotational freedom and three degrees of translational freedom. The movement with these degrees of freedom leads to a change in the image content, namely the relative rotation - imagine that the camera rotates around itself, around its projection center - leads to a distance-independent displacement and rotation of the image content and leads to the relative Translation to a distance-dependent change in image content, in particular change in size of the objects shown and change in perspective.

It is assumed that the arrangement of the symmetries in the observed pattern 610 is known, for example from a reference pattern. For example, a pattern 610 as shown in FIG. 7 or FIG. 8 can be used. Then it would already be known, for example, that there are 8×8 point symmetries lying on a square grid with a known distance. Optionally, the sequence of odd and even point symmetries of the reference pattern would also be known. Alternatively, this sequence can also be determined from the observation and, if necessary, it can then be determined by a database comparison which reference pattern is present.

The six degrees of freedom sought can be determined from each observation of pattern 610, ie with each new camera image. For this purpose, the centers of symmetry of the regions 110 of the pattern 110 are first determined. Then these are registered on the reference pattern, so that for each observed center of symmetry the assignment to the corresponding reference center of symmetry is known. It is not a problem if a significant proportion, e.g. B. significantly more than half, the symmetry centers can not be observed, z. B. because of occlusion, because the arrangement of the symmetries in the respective pattern 610 has a sufficient degree of redundancy, so that the registration and even the unique identification remain possible. If the assignment is known for a sufficient number of symmetry centers, the six degrees of freedom can be determined from this. If the geometry of the respective rigid part of the arrangement is known, the six degrees of freedom determined between camera 102 and pattern 610 can be transformed into other references, for example for the reference between the docking point of the gangway 2160 and the docking point of the platform 2170, or between an optional second sensor and the pattern 610.

Optionally, a stereo or multi-camera system can also be used, so that multiple images of the same pattern 610 can be recorded from different perspectives and evaluated together. A stereo or multi-camera system offers the advantage that distances and Surface orientations of the pattern 610 can be determined via triangulation, which can make the desired determination of the six degrees of freedom more accurate, easier and/or faster.

According to exemplary embodiments, it is possible to quickly and precisely determine the relative arrangement and, if necessary, the relative orientation between camera 102 or imaging sensor and pattern 610 . This can be used to control a mechanical actuator that changes this relative arrangement or orientation. For example, the hydraulics for controlling the gangway 2160 with all the necessary degrees of freedom. In particular, this can be embedded in a control loop.

21 represents only one of many conceivable applications. In general, the camera 102 can be located on the part moved by the actuator and the pattern 610 on the non-moving part—or vice versa. Likewise, both sides can be moved, as is the case with the floating wind power plant and the supply ship. This allows various actions to be carried out in a controlled manner, for example deliberately approaching a certain point, aligning to the intended orientation, and gentle docking, the reverse of this, i.e. undocking and directed and safe removal, parallel movement of a robot to a surface, movement relative to a or multiple patterns 610, along an intended trajectory with intended location-dependent speed and orientation, capturing an object provided with pattern 610 by a robot while the object is moving, etc.

Another application example is a visual docking of a contact point, e.g. B. an electrical contact to a mating contact or a pressure connection or tank connection for gases, liquids, etc. to a counterpart. For example, for charging or refueling of ground, air, water vehicles, robots or drones. Since contactless charging for electric and electric hybrid vehicles has not yet been able to establish itself and may not become established due to the power loss, charging via electrical contacts remains interesting and important. That plugging in plugs manually also remains unpopular because it is cumbersome, time-consuming, dirty or cold hands, tripping hazards, risk of the cable being stolen, risk of electric shock if the cable is damaged. An alternative possibility is via a contact on the ground in connection with a system similar to that from FIG. 21, whereby it is arbitrary on which side the actuators for the contact are located, ie either on the ground or on the vehicle, and on which side camera or imaging sensor and pattern 610 or at least one symmetrical area 110 are located. In an application such as this, a single point-symmetrical region 110 can already be sufficient. For example, the contact point to be sought is then to be placed in the center of symmetry of area 110, because this then represents the only designated location. For further applications, something could be hidden behind the center of symmetry, for example, which only the robot should find, but not the human being. Or the center of symmetry could mark the place where a flying drone should land, without this marked point being recognizable to humans.

Another application example is navigation in buildings and systems with the help of hidden symmetries using the at least one area 110 and/or pattern 610. Visual orientation and navigation can be implemented here by exploiting or using hidden symmetries in the form of patterns 610. especially in buildings and systems, underground, but also in the open air. This enables machines, especially mobile robots, as well as people equipped with tools such as a smartphone with a camera, to locate themselves and navigate independently.

If there is visual contact with the satellite, a GPS-based system is usually sufficient for many localization and navigation applications. In many places, however, this option is not available, e.g. B. inside buildings, in parking garages, in tunnels, in mining or under water. Especially in large, uniformly designed buildings with many floors, orientation and navigation can be very difficult for people and machines. All currently known SLAM systems (SLAM = Simultaneous Localization And Mapping), which when creating and using their maps can only be based on the landmarks that are already present in the area, may work unreliably and/or imprecisely there. Their use has therefore so far been limited to a few simple use cases and demonstrators. In practice, technical markers are used in practically all applications where reliability, high availability and accuracy are required. There are many possibilities for this, for example beacons, different types of tags, e.g. B. radio frequency, electromagnetic or acoustomagnetic, pulsed lights, as well as many types of passive markers, e.g. B. April Tag Markers, ArUco Markers, AR Tag Markers, QR Codes, etc. All of these markers are eye-catching. Some of them are also difficult to install, such as B. beacons to be embedded in the ground, or to maintain.

Patterns 610 according to embodiments provide non-perceptibility, i. H. the patterns 610 are not perceived as markers. They are designed to be particularly difficult for humans to perceive, but particularly easy for a machine to perceive using the method of providing shown in FIG. In addition, they are passive, meaning they require no electrical energy, no networking and no maintenance. With the options already described above for the execution of the areas 110 and pattern 610 in terms of material and symmetrical pattern content, there are a variety of options for making the useful pattern 610 attractively beautiful or at least inconspicuous and to ensure that they blend into an environment insert as best as possible. These possibilities can be used, for example, by interior designers, decorators, planners of hospitals, offices, hotels, airports, train stations, etc.

Retrofitting, for example, a multi-storey building with samples 610 in preparation for the use of mobile robots can also be carried out without such an intervention changing the appearance of the corridors and rooms or even impairing it. For this example, existing elements of the wall or ceiling paneling can be modified or replaced, for example by elements of the same size and color, but with z. B. relief embedded, milled, embossed, drilled, punched or printed, lasered or etched patterns 610. For example, by replacing acoustic ceiling tiles, where the hole pattern used is simply replaced by a hole pattern with symmetrical properties. From this, new services and business models and expertise in equipping rooms, buildings and systems with such patterns 610 can also be made possible. Even in applications where the benefit of hiding may be irrelevant, such as B. in warehouses, in tunnels, in mining, under water, on offshore platforms or the like, the already described superiority of patterns 610 from areas 110 with a random character in terms of robust detectability and accuracy speaks for their use. In this context, hierarchical patterns 610 corresponding or similar to the hierarchical patterns described with reference to FIGS. 17 to 19 can also be advantageously used.

If a large number of patterns 610 are required, e.g. B. in a large building such as a hospital, it can be advantageous to make the patterns 610 distinguishable or to provide them with additional information, as is also explained in particular with reference to FIG. If large areas or long rooms are to be covered with only a few patterns 610, these should each be detectable over a large distance. A solution with hierarchical patterns 610 is proposed for this with reference to FIGS.

FIG. 22 shows a schematic representation of various display media 600 with predefined point-symmetrical regions 110 according to exemplary embodiments. The centers of symmetry 112 of the predefined point-symmetrical regions 110 are also drawn in. At least one predefined point-symmetrical area 110 is generated on, on or in each display medium 600 . The display media 600 are landmark elements. In Fig. 22 useful or decorative objects for the garden are shown as examples of landmark elements. Point symmetries are hidden in the surfaces of the landmark elements Humans do not perceive, but as landmarks, for example, support a robotic lawnmower in localization and navigation. The display media 600 are therefore objects with hidden symmetries or predefined point-symmetrical areas 110 for the garden and house to support mobile robots. Concrete examples of the display media 600 are planters, flower pots, vessels, supports, borders, stones, floor panels, wall panels, garden lights, water butts or decorative elements.

Thus, symmetrically designed objects in the garden and house can be advantageously used as display media 600 to support self-localization and navigation of mobile robots, these objects being unobtrusive in the sense that their function is not apparent. Optionally, any existing, usually non-mobile surveillance cameras can also be included, with support being able to take place in both directions between the robot and the surveillance camera. In particular, the self-localization and navigation of autonomous lawn mowers as robots can be improved or supported according to exemplary embodiments.

Conventionally, for example, a border wire and possibly a guide wire are used, which are manually laid in the turf and which can be susceptible to damage, e.g. B. when working on the ground. In addition, a robotic lawnmower may orientate itself so strongly or frequently on the ground wire that this can leave unsightly marks in the lawn. Alternative conventional solutions may also be unsatisfactory for a variety of reasons. A GPS localization in combination with lawn area detection can be inaccurate, in particular the distinction between lawns and beds as well as adhering to the border to the neighbor can be unreliable. For example, conventional solutions with battery-operated radio beacons, which are to be placed in several places in the beds, require maintenance and must not be moved or knocked over. It can be assumed that this solution is not readily suitable for several years of trouble-free operation. According to one exemplary embodiment, it is provided that elements such as the landmark elements or display media 600 from FIG. B. as a planter, flower pot, vessel, support, border, stone, base plate, wall plate, garden light, rain barrel or as a decorative element. There are many options for designing them in terms of material, surface, texture, structure and colour. A wall design on the house or on a garden shed is also an option, as well as the design of railings, fences or privacy protection elements with point-symmetrical areas 110. The landmark elements or display media 600 should be as stable or heavy as possible or anchored in the ground or on the building or on the fence so that accidental displacement is unlikely. The robot, which is preferably equipped with a camera or other imaging sensors, should have at least one landmark element in view if possible. It is better if he has two or more elements in view, then triangulation is also possible, so that the distance and the position of the robot relative to the landmark elements or display media 600 can be determined more precisely. As an alternative or in addition to the use of many landmark elements or display media 600, the robot can also have several sensors, e.g. B. two cameras or a camera and a laser scanner to always ensure a precise distance measurement by stereoscopy or time-of-flight measurement to the respective landmark element and possibly other points of the scene.

The areas 110 or patterns of the landmark elements or representation media 600 can be encoded, see also FIG. 8, or provided with additional information, see also FIG. 20, in order to make them easily distinguishable for the robot. If large areas or long stretches are to be covered with only a few landmark elements or display media 600, these should each be detectable over a large distance. For this purpose, the hierarchical patterns presented with reference to FIGS. 17 to 19 can be used, the large-area point symmetries for detection from a great distance and small-area ones Have point symmetries for the precise determination of the pose between the robot and landmark element or display medium 600 at small distances.

A practical application based on exemplary embodiments could be summarized as follows: The user buys three compatible landmark elements or

Display media 600, for example a planter, a decorative stone and a self-adhesive decorative film for the rain barrel, and places them in different places in his garden. You can get by with three landmark elements or display media 600 in addition to a charging station and find out an optimal placement of the same, for example on a website, where you can optionally upload photos of your garden, sketch your garden or the area to be mowed and/or your garden on a can mark map. The larger and more angled the garden is, the more landmark elements or display media 600 are required.

The user then carries out a manually controlled teach-in run with the robot. He controls this either with a smartphone, tablet, remote control, game console controller, gesture control or acoustically by shouting commands. The teach-in drive is used to show the robot the boundaries of the areas to be mowed and how to get from one section of the garden to the next. The robot automatically recognizes obstacles, so they do not have to be taken into account during the teach-in drive. The obligatory charging station is also provided with point-symmetrical areas 110 and always serves as a landmark element, for example. During the teach-in run, the robot detects the landmark elements or display media 600 that are visible from the respective location and by which it will later orient itself.

The robot can then drive autonomously through the garden and mow lanes. In doing so, it is based on the known landmark elements or display media 600 with the areas 110 or patterns. Temporarily missing or covered view of the landmark elements or The robot optionally bridges display media 600 using odometry, inertial navigation and/or GPS. If it turns out during the teach-in drive or during mowing operation that there are too few landmark elements or display media 600, the robot (e.g. via smartphone app) makes a recommendation as to how many elements should be added and where should be set up. If a landmark or a display medium 600 is covered over time by leaf growth, the robot gives a corresponding indication and requests that the covered view be cleared. If a landmark element or a display medium 600 was inadvertently moved or rotated significantly, the robot recognizes that the relative relationships between the landmark elements or display media 600 have changed and prompts the user for an update teach-in for the relevant part of the garden .

If surveillance cameras are also present on the property, these can optionally be included in the system: The method for detecting the areas 110 of the landmark elements or display media 600 also runs on the surveillance cameras. The normally fixed positions of the landmark elements or display media 600 are recorded by the surveillance cameras. The positions are saved. Now and then, e.g. B. once a minute, the positions are re-recorded and compared with the stored positions. If a landmark position was changed, e.g. B. accidentally, the surveillance camera system networked with the robot reports the change in position to the robot. An update teach-in can then possibly be omitted. The surveillance camera system can also provide the information that nothing has changed and mowing operation can take place normally. Conversely, the surveillance cameras can also benefit from the presence of the arrangement of landmark elements or display media 600: the robot creates a 2- or 3-dimensional digital map of its surroundings while it is driving. The landmark elements or display media 600 have fixed positions in this map. The respective surveillance camera finds part of the landmark elements or display media 600 and can locate itself relatively therein using the map constructed by the robot, ie localize itself. This requires networking of robot and camera. If there is no network, the surveillance camera can still locate itself with respect to the landmark elements or display media 600 .

Networking can go even further: the map created by the robot is uploaded back to a server where the planning was previously carried out. Planning and actual status can be compared with each other. The user receives a digital image of his garden, in which he can influence the control of the robot even more precisely. Optionally, camera images from the robot camera can also be used with this digital image, so that the user can also check the condition of his garden afterwards, e.g. B. about the seasons, in detail in a virtual reality view. In return, manufacturers can voluntarily receive data about all the gardens of the voluntarily participating users and use this information for a further development of their robotic lawnmowers and landmark elements or display media 600 . The application and mode of operation illustrated here for the garden can also be transferred to the interior of a house, in particular for vacuum cleaning robots, floor cleaning robots, window cleaning robots, flying surveillance drones and non-mobile surveillance cameras.

Fig. 23 shows a camera image of a conveyor belt as a display medium 600 with an embodiment of a pattern 610 of predefined point-symmetrical areas and objects 2300 placed on the conveyor belt. Thus, a conveyor belt is provided with a pattern 610 similar to or corresponding to one of the patterns described above. The pattern 610 is made up of predefined point-symmetric regions with odd and even symmetry. The centers of symmetry 112A and 112B of the predefined point-symmetrical regions are shown in FIG. In other words, FIG. 23 shows a camera image with a snapshot of the conveyor belt in a view from above. Objects 2300 are stored on the belt, for example in the form of various lamps with a GU10 base. Even if the objects 2300 occlude more than half of the band, a large enough portion of the pattern 610 remains visible for the function. When evaluating the image, the drawn-in point symmetry centers 112A and 112B were found. Based on an encoding of the pattern 610 is the Absolute positions on the conveyor belt can be clearly determined in a belt coordinate system.

24 shows the camera image of FIG. 23 after processing using the method of providing of FIG. 3. Thus, in FIG. 24, due to the partial occlusion of the pattern 610, detected outlines of the objects 2300 located on the conveyor belt are out of the camera image 23 shown. Here, after determining the coordinate range of the conveyor belt contained in the camera image, the corresponding known reference pattern was compared pixel by pixel with the camera image. Wherever the camera image and reference pattern deviate from each other, there is occlusion. These areas are marked in FIG. 24 by means of brightening. Thus, the exact outlines of the objects 2300 and their position can be detected.

With reference to Fig. 23 and Fig. 24 it is noted that a use of the patterns 610 with included symmetries can be realized for a moving belt, in particular a conveyor belt, with the pattern 610 applied to the outside or inside of the belt and with at least one imaging sensor, e.g. a camera, particularly while the belt is in use and in motion. For example, if a surface of a conveyor belt for industrial production is provided with a suitable pattern 610, then there are possible advantages for the observation and control of processes that are carried out at least in part by robots. The centers of symmetry 112A and 112B create distinct locations on the conveyor belt, which remain unambiguous even under mechanical stretching of the belt or if the belt runs imprecisely (e.g. laterally offset) or if the belt vibrates. These distinguished locations can define a flexible coordinate system that is firmly attached to the band.

By knowing and using the relative arrangement of the centers of symmetry 112A and 112B to one another, in particular taking into account the odd and even symmetries, the absolute coordinates of each point of the belt visible in the camera image in the belt Coordinate system determined will. This can be used, for example, to deposit an object 2300 at a well-defined position on the conveyor belt, which can be removed from there again at a different location without it being necessary to detect or recognize the object 2300 . For this purpose, the depositing location, which is equipped with a corresponding camera including evaluation, digitally provides the coordinate(s) where the object 2300 was deposited. Further additional information can be supplied, e.g. B. what type of object it is, how the object 2300 is oriented on the belt, which side is down, where it can be grabbed, etc. At the next pick-up point, which is also equipped with a camera including evaluation, are the digital transmitted data ready in time.

By observing the tape with the camera and (continuous) evaluation, the tape coordinates and tape speed at the point of tapping are constantly known. Normally, at this point in time, the object 2300 is still exactly at the belt coordinates where it was previously placed. In this way, the pick-up mechanism can be ready in good time or a robot can be reliably controlled in order to pick up the object 2300 from the conveyor accurately and without jerks. It is not necessary to identify or measure the object 2300 itself in the camera image. Just knowing the digitally transmitted data about the type, alignment and position on the strip coordinate system is sufficient to implement the reliable tapping function. FIG. 23 illustrates the evaluated image of a camera that observes the conveyor belt on which objects 2300 are located from above. As can be seen, despite partial occlusion, a more than sufficient number of symmetry centers 112A and 112B are found to determine their respective location on the tape coordinate system.

In a simple embodiment, the evaluation unit or determination device from FIG. 1 requires only very little information about the pattern 610 applied to the tape, namely at which coordinates the centers of symmetry 112A and 112B are located and their sign, i.e. whether it is an odd number in each case (negative sign) or even (positive sign) point symmetry. In a further embodiment, the evaluation unit or determination device from Fig. 1 is additionally complete reference samples provided, e.g. B. in the form of an image file, also known as a reference image, in order to carry out comparisons. Alternatively, the reference pattern can also be made available as a calculation rule, e.g. B. in the form of the parameters of a quasi-random number generator, which always delivers the same sequence of numbers with the same initialization, and which was called up several times here to create a reference pattern that has shares in several spatial frequency ranges, so that it can be used simultaneously for short, medium and is suitable for long distances. In this way, the reference image (or a part of it) can be calculated if required and takes up little storage space when not in use.

With knowledge of the reference image, it is then possible to use the camera image, e.g. B. pixel by pixel to compare with a corresponding section of the reference image. The necessary scaling adjustment and rotation, so that the two images match in terms of resolution and alignment, is a prerequisite here. This additional step of the image comparison is only made possible or considerably simplified because the position in the tape coordinate system has already been determined beforehand using the pattern 610 with coded point symmetries. Any remaining uncertainty regarding the position is very small. In other words, the two images to be compared are already registered very well with one another, apart from a small residual local uncertainty.

A high-precision evaluation can then be carried out based on the exact comparison of camera image and reference image, e.g. B. to determine the local distortion of the belt due to current loads or aging or poor belt tracking. With an empty and clean conveyor belt, it should then be possible to determine an almost perfect match between the camera image or camera image section and the corresponding reference image section over the entire surface. However, if there are objects 2300 on the conveyor belt that locally cover the pattern 610, the correspondence between the camera image and the reference image is no longer given at these locations or pixels. Masks of the objects 2300 on the conveyor belt can be generated from this information or the precise outlines of the objects 2300 can be derived. See Fig. 24. Further advantages are listed below in a non-exhaustive manner. Additional sensors and associated processing steps can be saved, e.g. B. light barriers, laser scanners, time-of-flight sensors, ultrasonic sensors, rangefinders, mechanical switches and dedicated camera-based algorithms that are conventionally used for presence detection, localization, surveying, classification or position detection. In addition to the possibilities already mentioned of determining an absolute position in the belt coordinate system based on the found centers of symmetry 112 A and 112 B and determining the speed from the time derivative of the position, there is also the possibility of determining an expansion or compression of the belt, in particular the stretching in the longitudinal direction. This can e.g. B. be caused by aging or overloading of the belt, by stiff rollers or other mechanical parts, by loads on the belt, by poorly synchronized drive units. Determining the strain can help to detect unfavorable influences at an early stage. If the strip deviates laterally from its target position, this can also be detected in good time without the camera having to monitor the edge that may be covered. Such a lateral deviation could otherwise possibly have a negative effect, e.g. B. on the lifespan. Longitudinal or transverse vibrations of the strip can be detected. Longitudinal vibrations would make themselves felt as small periodic deviations in the determined longitudinal speed. They can be caused by defective or sluggish bearings and can lead to objects slipping on the conveyor belt.

Retrofitting through integration into existing systems is relatively easy. Only the tape needs to be replaced, with a pattern 610 tape. Existing cameras looking at the tape can continue to be used if necessary. The software or embedded hardware for evaluating your images can be replaced or supplemented with the algorithm for detecting the centers of symmetry and their further processing or an exemplary embodiment of one of the methods mentioned above. The band can be made of different materials or components, for example a rubber band or rubberized fabric band or plastic Link strap or metal link strap. The pattern 610 can be applied to the top of the tape or to the bottom. The upper side means the side that comes into contact with the transported objects 2300 or goods, while the lower side is more likely to come into contact with the transport equipment, transport rollers, and so on. Attaching pattern 610 to the bottom can

Offer advantages because it cannot be covered by the transported objects 2300 and because patterns 610 and camera z. B. can be better protected against contamination in a dusty environment. The pattern on the top can offer the advantage of enabling higher accuracy, also because the objects 2300 are placed on the top and are visible together with the pattern 610 in the same camera image.

If an embodiment includes an "and/or" link between a first feature and a second feature, this should be read in such a way that the embodiment according to one embodiment includes both the first feature and the second feature and according to a further embodiment either only that having the first feature or only the second feature.

Claims

- 72 -

Expectations

1. Method (300) for providing navigation data (135) for controlling a robot (2160), the method (300) having the following steps:

Reading in (324) image data (105) provided by a camera (102) from an interface (122) to the camera (102), the image data (105) being a camera image of at least one predefined even and/or odd point-symmetrical region (110 ; 110A, HOB) in an environment of the camera (102);

determining (326) at least one center of symmetry (112; 112A, 112B) of the at least one even and/or odd point-symmetrical region (110; 110A, 110B) using the image data (105) and a determination rule (128);

Carrying out (330) a comparison of a position of the at least one center of symmetry (112; 112A, 112B) in the camera image with a predefined position of at least one reference center of symmetry in a reference image (115) relative to a reference coordinate system in order to calculate a position deviation (131) between the to determine the center of symmetry (112; 112A, 112B) and the reference center of symmetry; and or

Determining (332) displacement information (133) for at least a subset of pixels of the camera image relative to corresponding pixels of the reference image (115) using the positional deviation (131), wherein the navigation data (135) using the positional deviation (131) and/or the displacement information (133) are provided. - 73 - Method (300) according to Claim 1, in which the determination rule (128) used in the step (326) of determining is designed to cause a signature (s) to be generated for a plurality of pixels of at least one section of the camera image is to obtain a plurality of signatures (s), each of the signatures (s) being generated using a descriptor with a plurality of different filters, each filter having at least one symmetry type, each of the signatures (s) for each filter of the descriptor has a sign, that for the signature (s) at least one mirror signature (SPG, SPU) is determined for at least one type of symmetry of the filter, that a pixel having the signature (s) indicates the presence of at least one other pixel with one of the at least one signature (s) corresponding to a mirror signature (SPG, SPU) in a search area (1104) in an environment around the pixel is checked in order, if at least ei of a further pixel to determine the pixel coordinates of at least one symmetrical pair of signatures from the pixel and a further pixel, and that the pixel coordinates of the at least one symmetrical pair of signatures are evaluated in order to determine the at least one center of symmetry (112; 112A, 112B) and/or wherein at least one reflector (RPG, RPU) is applied to the signs of one of the signatures (s) in order to determine the at least one mirror signature (SPG, SPU), each reflector (RPG, RPU) for a type of symmetry specific and dependent on the filters of the descriptor for modifying the signs, the search range (1104) being dependent on at least one of the reflectors used (RPG, RPU). The method (300) of claim 2, wherein in the step (326) of determining, for each already determined center of symmetry (112; 112A, 112B) using the pixel coordinates of each symmetric signature pair, it is necessary to correctly identify the center of symmetry - 74 -

(112; 112A, 112B), a transformation rule for transforming pixel coordinates of the center of symmetry (112; 112A, 112B) and/or the at least one even and/or odd point-symmetrical region (110; 110A, HOB) is determined, wherein the transformation rule is applied to the pixel coordinates of the center of symmetry (112; 112A, 112B) and/or the at least one even and/or odd point-symmetrical area (110; 110A, 110B) in order to correct a distorted perspective of the camera image. Method (300) according to one of the preceding claims, in which in the step (326) of determining a type of symmetry of the at least one center of symmetry (112; 112A, 112B) is determined, the type of symmetry representing an even point symmetry and/or an odd point symmetry, and /or wherein in the step (330) of performing a comparison of the type of symmetry of the at least one center of symmetry (112; 112A, 112B) in the camera image is performed with a predefined type of symmetry of at least one reference center of symmetry in a reference image (115) in order to determine a match between the at least one center of symmetry (112; 112A, 112B) and the at least one reference center of symmetry. Method (300) according to Claim 4, in which the image data (105) read in in step (324) of reading in is a camera image of at least one pattern (610; 1710, 1810) from a plurality of predefined even and/or odd point-symmetrical regions (HO ; 110A, HOB), wherein in the step (326) of determining a geometric arrangement of symmetry centers (112; 112A, 112 B) of the at least one pattern (610; 1710, 1810) is determined, a geometric sequence of symmetry types of the symmetry centers ( 112; 112A, 112 B) is determined and/or the pattern (610; 1710, 1810) is determined from a plurality of predefined patterns using the sequence, the arrangement and/or the sequence having an identification code for the pattern (610; 1710, 1810 ) represented. - 75 - Method (300) according to claim 5, wherein in the step (326) of determining using the arrangement of the centers of symmetry (112; 112A, 112B) of the at least one pattern (610; 1710, 1810) and/or the sequence of Types of symmetry of the centers of symmetry (112; 112A, 112B), implicit additional information of the at least one pattern (610; 1710, 1810) or a reading rule for reading out explicit additional information in the camera image, the arrangement and/or the sequence of the additional information being encoded Form represented, wherein the additional information on controlling the robot (2160) is related. Method (300) according to one of Claims 5 to 6, in which in the step (330) of carrying out the reference image (115) from a plurality of stored reference images is selected or is generated using a stored generation rule. Method (300) according to one of Claims 5 to 7, in which the step (326) of determining and/or the step (330) of performing independently of the type of symmetry of the symmetry centers (112; 112A, 112B) for all symmetry centers (112; 112A, 112B) is carried out jointly or is carried out separately for the centers of symmetry (112; 112A, 112B) of the same type of symmetry depending on the type of symmetry of the centers of symmetry (112; 112A, 112B). Method (400) for controlling a robot (2160), the method (400) having the following steps:

Evaluation (444) of navigation data (135) provided by the method (300) according to one of the preceding claims in order to generate a control signal (145) dependent on the navigation data (135); and - 76 -

Outputting the control signal (145) to an interface (148) to the robot (2160) to control the robot (2160). Method (500) for producing at least one predefined even and/or odd point-symmetrical region (110; 110A, HOB) for use by a method (300; 400) according to one of the preceding claims, the method (500) having the following steps:

generating (502) design data (204) representing a graphical representation of the at least one predefined even and/or odd point symmetric region (110; 110A, 110B); and

Generating (506) the at least one predefined even and/or odd point-symmetrical region (HO; 110A, HOB) using the design data (204) on, on or in a presentation medium (600) around the at least one predefined even and/or odd produce a point-symmetric region (HO; 110A, HOB). Method (500) according to Claim 10, in which, in the step (502) of generating, design data (204) are generated which represent a graphical representation of the at least one predefined even and/or odd point-symmetrical region (HO; 110A, HOB) as a circle, represent an ellipse, a square, a rectangle, a pentagon, a hexagon, a polygon or an annulus, wherein the at least one predefined even and/or odd point-symmetrical area (110; 110A, HOB) has a regular or quasi-random content pattern, and/or wherein a first half of an area of the at least one predefined even and/or odd point-symmetrical area (HO; 110A, HOB) is specified arbitrarily and a second half of the area is constructed by point reflection and/or inversion of gray values and/or color values , and/or wherein in the step (506) of generating the at least one predefined straight line - 77 - and/or odd point-symmetrical areas (110; 110A, HOB) are generated by an additive manufacturing process, cutting, coating, forming, archetypes or optical displays, and/or wherein the display medium (600) is glass, stone, ceramics, plastic , rubber, metal, concrete, plaster, paper, cardboard, food or a visual display device. Method (500) according to one of Claims 10 to 11, in which, in the step (502) of generating, design data (204) are generated which contain a graphical representation of at least one pattern (610; 1710, 1810) from a plurality of predefined straight and/or or represent odd point-symmetrical areas (110; 110A, 110B), with at least a subset of point-symmetrical areas (HO; 110A, HOB) being aligned on a regular or irregular grid (1311), directly adjoining one another and/or partly of at least one neighboring straight and/or odd point symmetric regions (HO; 110A, HOB) are separated by a space, are identical to or different from each other in terms of their dimensions and/or their content patterns and/or are arranged in a common plane or in different planes, and/or wherein in the step (502) of generating design data (204) are generated, the at least one graphical representation t of a pattern (610; 1710, 1810) with hierarchical symmetry. Device (120; 140; 200) arranged to carry out the steps of a method (300; 400; 500) according to one of the preceding claims in corresponding units (124, 126, 130, 132; 144, 146; 202, 206) execute and/or control. Computer program set up to execute and/or control the steps of a method (300; 400; 500) according to one of Claims 1 to 12.

15. Machine-readable storage medium on which the computer program according to claim 14 is stored.