DE102014106718B4 - System that presents a field of view representation in a physical position in a changeable solid angle range - Google Patents

Abstract

System (1), which presents a visual field representation (119, 119') in a physical position in a changeable solid angle range, and carries out the following steps: determining a first and second physical position (2, 222, 322, 288, 388), in particular after taking a Pose of a marker carrier (3, 3', 3'', 103, 236, 336), with an initial position (208a, 288, 388, 308a) as a first physical position, by detecting marker image areas (5, 5', 5 '', 5''', 5IV, 105, 206, 210, 306, 310, 350, 351, 352, 355, 356, 357, 358, 359, 260, 262, 362) of a first image data pixel array (21, 221, 221', 221'', 321, 321', 321'') in a computing unit (13), and at least one further image data pixel arrangement (21, 221, 221', 221 '', 321, 321', 321'') is fed to the computing unit, which after a movement of the marker carrier (3, 3', 3'', 103, 236, 336) or after a movement of the digital camera (7, 207, 307) g changed marker image areas (5, 5', 5'', 5''', 5IV, 105, 206, 210, 306, 310, 350, 351, 352, 355, 356, 357, 358, 359, 260, 262, 362 ) having, wherein a position (x, y, z) and/or an orientation (Θ, B) of the second physical layer (2, 222, 322, 288, 388), which can be predetermined using a marker position configuration (43), from , the marker position configuration (43) defining marker image areas (5, 5', 5'', 5''', 5IV, 105, 206, 210, 306, 310, 350, 351, 352, 355, 356, 357, 358, 359 , 260, 262, 362) of the further image data pixel arrangement (21, 221, 221', 221'', 321, 321', 321'') and from a marker model configuration (242, 342, 264, 364) of the initial position (208a, 288 , 388, 308a) by means of a comparison algorithm (214, 314, 216, 316, 220, 320, 270, 370, 286, 386), while determining a nearest neighbor (214, 314) to a marker image area (5, 5', 5 '', 5''', 5IV, 105, 208a, 308a) of the initial position (208a, 288, 388, 308a), with extrapolation of hidden, further marker image areas (5a) is calculated using a numerical optimization method (220, 320), the marker position configuration (43) having a plurality of marker elements (5, 5', 5'', 5''', 5IV, 105) which are located on the at least two surface elements (27, 27', 27'', 27''', 27IV, 27v, 127, 127', 127'', 127''', 127IV) which have a marker carrier (3, 3', 3'' , 103, 236, 336) for the marker position configuration (43), in a surface element-wise arrangement (41, 41', 41'', 41''', 41IV, 41V, 234, 334), with an arrangement of marker elements ( 5, 5', 5'', 5''', 5IV, 5a, 105), which is one of the surface elements (27, 27', 27", 27''", 27IV, 27v, 127, 127', 127'' , 127''', 127IV) of an arrangement of marker elements (5, 5', 5'', 5''', 5IV, 5a, 105) which form another, adjacent, of the surface elements (27, 27' , 27'', 27''', 27IV, 27v, 127, 127', 127'', 127''', 127IV) using a noise reduction algorithm in the form of a quaternion based motion tracking filter (224, 324), which filters the position (x, y, z) and/or the orientation (Θ, B) by means of a movement course prognosis (228, 328), is provided, with a portable marker carrier (3, 3', 3 '', 103, 236, 336), which comprises a plurality of marker elements (5, 5', 5'', 5''', 5IV, 105, 5a), with exactly one digital camera (7, 207, 307), with which a spatial region (11) can be recorded and imaged in a two-dimensional pixel data field (21), with a computing unit (13) for executing image data processing algorithms (200, 201, 300, 301, 204, 304, 218, 318, 220, 320, 224, 324, 268, 268', 368, 368', 270, 370, 286, 386), with at least one electronic image display (19, 119, 119', 23, 123) for the presentation of the visual field representation (119, 119') , with a data transmission unit (15) with which a data transmission connection (17, 17'), in particular wireless, is established between the digital camera (7, 207, 307) and the computing unit (13) and the image display (19, 119, 119', 23 , 123) can be created andwith at least one light source for emitting light that can be registered with the digital camera (7, 207, 307) into the spatial area (11), the visual field display (119, 119') being processed by the processing unit (13) for the image display (19, 119, 119' , 23, 123) can be provided and wherein the marker carrier (3, 3', 3'', 103, 236, 336) can be moved in the spatial region (11) without obstruction and the marker carrier (3, 3', 3'', 103, 236, 336) comprises at least five surface elements (27, 27', 27'', 27''', 27IV, 27v, 127, 127', 127'', 127''', 127IV) which are arranged like a ring, wherein a surface element (27, 27', 27'', 27''', 27IV, 27v, 127, 127', 127'', 127''', 127IV) at least three marker elements (5, 5', 5'', 5''', 5IV, 105, 5a), characterized in that the marker elements (5, 5', 5'', 5''', 5IV, 105, 5a) have a configuration (41, 41') that differs by surface element , 41'', 41''', 411IV, 41V, 234, 334, 236, 336) and in each angular position (O, B) of the marker carrier (3, 3', 3'', 103, 236, 336) within the spatial region (11) with respect to a camera image plane (9) at least two surface elements (27, 27', 27'', 27''', 27IV, 27v, 127, 127', 127'') having marker elements, 127''', 127IV), a first surface element (27, 27'', 127) and a second surface element (27', 27''', 27IV, 27v, 127', 127'', 127''', 127IV ), can be imaged at least in regions by the digital camera (7, 207, 307), with at least four marker elements (5, 5', 5'', 5''', which can be detected algorithmically by the computing unit (13) in the pixel data field (21), 5IV, 105, 5a) are present, and the four marker elements (5, 5', 5 '', 5''', 5IV, 105, 5a) to a marker model configuration (242, 342, 286, 386, 220, 220', 320, 320', 230) calculated from marker configuration data (234, 334, 236, 336), 330), which defines a spatial position of the field of vision (232, 332), can be assigned and which electronically e the image display (19, 119, 119', 23, 123) is video glasses (23, 123) which comprise at least one (125), preferably two, measuring devices, such as an eye position measuring device, a position sensor system or an acceleration sensor (125).

Description

The present invention relates to a system that presents a field of view representation in a physical location according to the preamble of claim 1. In the system for presenting a field of view representation, a determinable, physical location is integrated into the system as a spatially variable field of view in a computer-generated view of a spatial image, e.g. B. a virtual space.

State of the art

From the field of visualization technology, especially in the field of so-called virtual reality, various procedures are already known, with which z. B. a body position can be detected in order to use the resulting data z. B. for a computer simulation. In the document GB 2 451 461 A (Applicant: Naven Shavla; filing date: July 28, 2007) describes a system in which a person wearing a marker on his head and holding a stick in one hand is simultaneously recorded by two or more cameras. This is intended to enable three-dimensional human-computer interaction. A switching unit is attached to each rod in order to supply switching signals to the system. Due to the two cameras required, the amount of data to be processed in the image evaluation is usually very high and the evaluation of the data is correspondingly time-consuming.

A different method with apparatus, in which only one camera is used, is described in document U.S. 4,884,219 B2 (Owner: W. Industries Ltd.; priority date: 01/21/1987). Here, too, the appearance of computer-generated images should be influenced by a user. The movement of the user's eyes is detected and transmitted to a computer. Furthermore, changes in the orientation of the head should also be evaluated, with the head position being determined by means of electric fields from an arrangement of coils on a helmet.

For the detection of a body shape z. B. a human body, is in the document US 2012/0 095 589 A1 (Inventor: Arcadiy Vapnik; filing date: 10/17/2011) proposed the use of a full body costume with markers spread over it. Here white markers are used, which are printed on the costume. The costume wearer's movements are filmed by a camera and the 3D coordinates of the body are determined from the camera images and scaled with a measuring tape.

A method for determining a position of a body according to Cartesian coordinates (x, y, z) using at least one marker object of known dimensions is from document WO 98/30977 A1 (Applicant: Qualisys AB; priority date: 01/13/1997). The movements of the marker are to be derived from the video signal of a resolved image from a CCD camera, e.g. B. infrared camera, are calculated. The temporal structure of the video signal is evaluated. A segment that corresponds to a marker in an image line is determined using a comparator. In the method, coordinates of an image of an object are first determined. After that, a dimension parameter of the object is determined and then proportions are calculated from it. Due to the very special image evaluation, technical information on the camera type must be taken into account.

The rotation of a body is described in the patent application US 2002/0001397 A1 (Inventors: Takatoshi Ishikawa et al.; priority date: 10/30/1997). A control unit should increase an image sequence of a presented space during rapid rotations so that there is no impression of jerking. When evaluating the rotation, a calculation of an angular velocity of a head movement should be used. It remains open how the angular velocity is to be specifically inferred from image data.

The tracking of the movement of objects provided with actively or passively radiating marker elements is described in document WO 2011/141531 A1 (Applicant: Movolution GmbH; priority date: May 11, 2010). Digital color images are continuously recorded and stored with a camera. The color images are then converted to gray images to facilitate localization of the marker elements as bright spots. For this purpose, each gray value is compared with a defined threshold value and a pixel above a threshold value is registered as a potential marker element. After this gray image evaluation, the digitally stored color images are used again to identify the marker elements in order to carry out the motion analysis. Passively retro-reflective-radiating markers should have a high directivity. Alternatively, color LEDs can also be considered as markers. Due to the use of threshold values when identifying passive markers, however, there is a risk that individual markers will be lost during image analysis.

To improve the identifiability of individual markers, the patent application US 2012/0 121 124 A1 (Applicant: The Board of Trusties of the Leeland Stanford Junior University; filing date: 06/15/2011) proposed, Marker see above to design that they each have a printed chessboard-like pattern from which the position of the marker emerges. This is intended to avoid errors when determining positions in tomographic measuring devices in medical technology. A disadvantage of using such markers, however, could be seen in the fact that the markers can move out of the camera focus during free movement in space, so that the position-determining markings can no longer be resolved.

The ones in the document WO 02 063 456 A1 (Applicant: Anderson Technology PTY Ltd.; Priority date: 02/08/2001) for tracking a movement, has strips, which are colour-coded, within a circle on a helmet. In a color analysis of image data, the areas corresponding to the markers are first filtered out, the assigned pixels are counted and average values for x and y coordinates are determined. If the pixel area is found reasonable, an evaluation of various radii on the image is made with a determination of straight line parameters for the relationship between various locations. One problem is that rapid movements in the image sequence can result in sequences in which it is not possible to determine the position of the helmet. When transferring a movement from the real space to the virtual space, non-uniformities can therefore occur, which can impair the experience of perceiving the virtual space.

A method and apparatus for video-tracking an object using a head-mounted image display device is US 2010/0000011 DE 603 01 987 T2 (Owner: Samsung Electronics Co., Ltd.; Union Priority Day: 03/07/2002). The movement of the object, ie position, orientation and direction of movement, should be recorded by a camera. The environment of a virtual reality should be designed in interaction with the movements of the user. Main markers and additional markers are used, which are marked in different colors. At least one main marker must be recorded. Lens distortions are compensated for in the determined video data. Image areas corresponding to the markers are determined using a noise reduction unit. A movement direction of the points is estimated from the video data. Intrinsic parameters of the camera must be taken into account for an orientation and position estimation. A morphological image processing method is to be used in the noise reduction unit, which is applied to marked video data. A movement is estimated from center coordinates of main marker areas by a movement estimation unit, with the addition of images following in time, using a color histogram and a color object tracking method, with a Kalman filter being used. The distinction between main markers and secondary markers and different colors makes an image evaluation time-consuming and may require the color spectrum of the light sources used to be taken into account.

Other aspects of determining a pose with a camera are e.g. B. in the documents US 2010/0 232 727 A1 (Applicant: Metaio GmbH; filing date: May 22, 2008) and document US 2012/0 169 887 A1 (Applicant: AiLive Inc.; filing date: 01/05/2011) and DE 69 132 952 T2 (Owner: Sun Microsystems, Inc.; priority date: 11/30/1990), U.S. 7,529,387 B2 (Owner: Canon Kabushiki Kaisha; filing date: 05/11/2005), U.S. 8,081,815 B2 (Owner: Canon Kabushiki Kaisha; filing date: 08/08/2007), U.S. 8,217,995 B2 (Owner: Lockhead Martin Corporation; filing date: 01/17/2009) and WO 2008/055 262 A2 (Applicant: Sensics, Inc.; priority dates: 11/02/2006 and 06/19/2007).

In the article "virtual and yet realistic", SBZ issue 06-2014, March 5, 2014 realistically), a 3D bathroom planning with room glasses and a camera arranged on the ceiling of a room is described as the closest prior art. The camera detects a black pentagonal ring on the ceiling of the room, which is fitted with white balls and is worn on the head.

From Varracella, Alberto; De Momi, Elena; Valentini, Martha; Ferrigno, Giancarlo et al. "Application of unscented Kalman filter for robust pose estimation in image-guided surgery", Medical Imaging 2012, image guided procedures, robotic interventions and modelling; The International Society for Optics and Photonics (SPIE) has disclosed a Kalman filter that can be used in a tracking system.

Individual aspects that can be helpful in the mathematical design of algorithms, as well as explanations of terminology, are contained in the following publications: For structure-from-motion, see Linda G. Shapiro, George C. Stockman in “Computer Vision “, Prentice Hall, (2001); on the control algorithm to exclude error regions (RANSAC algorithm) see MA Fischler, and RC Bolles in “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography”, Commun. ACM, 24(6): p.381 to p.395 (1981); for ensemble adjustment algorithm (bundle adjustment algorithm) see B. Triggs, P. Mclauchlan,
R. Hartley and A. Fitzgibbon, Chapter "Bundle adjustment - a modern synthesis" in "Vision Algorithms: Theory and Practice", LNCS, page 298 to page 375, Springer Verlag, (2000); for Hough transformation see U.S. 3,069,654 (Inventor: Paul VC Hough; filing date: 03/25/1960); for 4-Parametric representation see JB Kuipers "Quaternions and Rotation Sequences: A Primer with Applications to Orbits, Aerospace, and Virtual Reality", Mathematical Sciences Series, Princeton University Press (1999), ISBN 9780691058726; on Kalman filters, see Y Bar-Shalom and Xiao Rong Li, "Estimation with Applications to Tracking and Navigation", John Wiley & Sons, Inc., New York, NY, USA (2001), ISBN 047141655X; on the linearized Kalman filter, see Y. Bar-Shalom and Xiao Rong Li, "Estimation with Applications to Tracking and Navigation", John Wiley & Sons, Inc., New York, NY, USA (2001); on particle filters (condensation algorithm, sequential Monte Carlo) see B Ristic, S Arulampalam, N Gordon, “Beyond the Kalman Filter: Particle Filters for Tracking Applications”, Artech House (2004), see also AJ Haug, “A Tutorial on Bayesian Estimation and Tracking Techniques Applicable to Nonlinear and Non-Gaussian Processes”, The MITER Corporation, USA, Tech. Rep., Feb. Retrieved 2008-05-06 (2005); for quaternion-based tracking filter see FL Markley, "Attitude Error Representations for Kalman Filtering" in Journal of Guidance, Control, and Dynamics, 26 (2), page 311 to page 317, (March 2003), see also IY Bar-Itzhack, and Y Oshman "Attitude Determination from Vector Observations: Quaternion Estimation" in IEEE Transactions on Aerospace and Electronic Systems, 21 (1), page 128 to page 136, (Jan. 1985), and I. Y Bar-Itzhack, J. Deutschmann and FL Markley, "Quaternion Normalization in Additive EKF For Spacecraft Attitude Determination," in NASA Technical Documents, pp. 403-421, (Feb. 1993).

A method for monocular omnidirectional detection of head movements in the visible light spectrum is described in the article by Giroen Lichtenauer and Maya Pantic, 2011 IEEE International Conference on Computervision Workshop, page 430 to page 436, (2011). The marker structures used are attached to a metal frame that can be worn on the head. The marker array can be used with angles up to 90° in frontal view. The markers are attached to sticks protruding from the head to avoid possible masking. In a first step it is necessary to determine an initial model of the marker structure. Here, point-to-point measurements of markers must be made. The metal frame adapts flexibly to the head, which is why the marker positions can be changed. To determine the position, markers are selected that form an almost equilateral triangular arrangement. In this way, markers that are far apart tend to be preselected for evaluation. The use of a flexible marker structure increases the risk that the structure will change during wear and that the tracking of the markers will exhibit major errors or be interrupted as a result.

The company Immersight GmbH has already presented an embodiment of a marker carrier with a pentagonal cross section, on which several markers are arranged on surface elements. The marker carrier can be worn around a person's head at approximately eye level using video goggles. The position of the head and the movement of the person in space are determined in a system using a camera. The head position can be incorporated into a presentation of a virtual environment, e.g. B. from a computer-aided development program (CAD), convert to display on video glasses. Impressive visual representations can already be generated with this.

task

The object of the present invention is to avoid the disadvantages of the prior art and to further improve the determination of a position and to provide position data that is as exact as possible in real time, or at least in a time sequence that is below the human perception threshold. According to a further aspect, an object consists in using position data obtained to further improve a visual impression of a virtual environment within a system. This should z. B. Movements in a virtual world appear even more impressive because the representation of the movement corresponds even better to a natural or an actual movement sequence.

invention description

The object according to the invention is solved by the characterizing part of a method according to claim 1 and the characterizing part of a method according to claim 8, and by a system having the features of the characterizing part of claim 18.

The pose of an object, which can also be referred to as the position of an object or the position of an item, is determined from camera images. Position data can be of good quality even with fast movements or turns, ie in particular uninterrupted. Even a single camera image can be sufficient to obtain a precise estimate of the position of an object.

Methods according to the invention that make it possible to determine the position of an object in any pose that can theoretically be assumed are particularly reliable. From a temporal perspective, an adoptable pose can also be referred to as a changed physical location. A changed objective situation can e.g. by a first image data pixel array and a second image data pixel array. The changed objective position can be taken from an initial position. The starting position can therefore also be referred to as a first physical position. Preferably, the images are captured with a single camera, such as a black and white camera. Commercially available digital cameras can be used. The recordings are made with a frame rate. The image refresh rate has a size with which a movement sequence can be imaged, in particular step by step, in other words without omitting individual movement phases. The position of an object can be determined particularly quickly if prior information about the geometric structure of the object is available. An initial determination of a physical situation is also referred to as initialization.

An object structure can be defined using individual marker points. A marker point can be a location of the object that differs from a surrounding area of the object. It is also possible to provide marker elements on an object. A marker point can e.g. B. be a center point of a marker element. In other words, a point on a structure of an object can be measurably defined by a marker element. The definition is clear in one respect. A position of an object point corresponds to a position of a marker point. Also, the elevated position of a marker point defines the position of an object in three degrees of freedom. For objects that B. are variable in shape or color, it is advantageous for the determination of a physical situation when the variable object is assigned a marker carrier. Marker points are arranged on the marker carrier in the form of marker elements. The arrangement of the marker elements relative to each other is fixed. The marker carrier is preferably shape invariant. The marker carrier can be carried by the object or by a person. According to one aspect, the marker carrier itself can also be referred to as an object that has a physical position. A physical situation is clearly described by three spatial dimensions and an orientation. A position of an object point is indicated by a marker element, in particular a center point of a marker element.

The marker elements form a marker configuration in a respective position on the marker carrier. A plurality of marker elements are present in a configuration. The marker configuration can also be referred to as a configuration of marker points. The marker configuration is e.g. B. already known from the production of the marker carrier. According to one aspect, the known marker configuration is available as a data record, which is also referred to as marker configuration data. It is also possible to determine the marker configuration on a changeable marker carrier using an algorithm, in particular from a sequence of two-dimensional images. The marker carrier or also the marker configuration can be assigned to at least one coordinate tuple of an object, the coordinate tuple describing at least three spatial dimensions and one orientation. A position taken by a marker configuration in a room, in particular in relation to a fixed point in the room, is also referred to as a marker position configuration. A marker model configuration includes a configuration of marker point coordinates that represents a computationally provided marker position configuration hypothesis.

The pose or position is determined with a digital camera. The digital camera can film or photograph a marker carrier. In the digital camera, the optical camera images are converted into electronic data, which are also referred to as image data. The image data are supplied to a computing unit, which in particular has a memory unit. There is an image data acquisition connection between the processing unit and the digital camera. B. via a line connection or as a radio connection such as an infrared connection can be formed. The physical position is determined in the processing unit. The computing unit evaluates image data. The image data are used to determine the position of an object at the time the image was captured. Experiments have shown that methods for determining a position of objects work particularly accurately if a marker carrier is used which has at least six marker elements in a previously known marker configuration. The marker elements are arranged at a distance from one another. The marker configuration occupies a position on the marker carrier. A direction relative to the marker carrier can be defined by the marker configuration on the marker carrier. For example, a direction can B. be specified with an orientation. The direction assigned to the marker carrier can be a direction of an object, such as a direction of rotation, a direction of movement, a direction of travel or a direction of flight, which can also be referred to as a direction of translation or direction of rotation. The marker carrier can be attached to an object. However, the direction can also be a head position or a line of sight of a person who z. B. carries on the head. Position, direction or orientation can e.g. B. by means of coordinates or vectors abstract, in particular summarized as data tuples.

A marker element occupies a marker position in space. The marker position can be captured by the digital camera. The digital camera records light emanating from a marker element. Light, which can be actively or passively emitted by the marker element, is imaged by a camera chip onto an electronic image data pixel arrangement. The spatial area in three dimensions, in which a marker element is located, is mapped onto a two-dimensional pixel arrangement, from which the image data to be captured emerge. The image data or the image data pixel arrangement are evaluated in the processing unit. A marker element in the image data pixel arrangement preferably occupies at least one pixel, with which a resolution to be provided by the digital camera can be defined. According to another aspect, a pixel size in relation to a marker size determines a, preferably maximum, distance between the digital camera and the marker element for a given, fixed imaging ratio by a digital camera. Due to the brightness integration that took place within a pixel and the possible distribution of the brightness of a marker element, e.g. B. over four pixels arranged in a square, the resolution should not be undershot, at least in the case of passive, light-scattering marker elements. A marker element can thus be found in an image data pixel arrangement as a marker image region.

A method, such as the method for determining a physical position, can be programmed in the computing unit by means of a software code. The computing unit executes algorithms. According to a system aspect, an algorithm can also be referred to as a unit that is present in the computing unit. In other words, algorithms are sub-units of an arithmetic unit that are kept available for execution alongside the internal operating units of the arithmetic unit.

The computing unit is equipped with a marker detection algorithm. The marker detection algorithm is used to find a marker image area in the two-dimensional image data pixel arrangement. Each of the mapped marker elements is preferably found as a marker image area by the marker detection algorithm. A suitable marker detection algorithm is e.g. B. the so-called Hough transform. The Hough transform is a robust global method for recognizing straight lines, circles or any other parameterizable geometric figures in a binary gradient image, i.e. a black-and-white image, based on finding edges. The Hough transform is useful for detecting spherical markers or marker elements in an image data pixel array.

The two-dimensional image data pixel array is used to calculate a three-dimensional marker position configuration. The three-dimensional marker position configuration can be referred to as the spatial arrangement in which the marker elements are present in the field of view of the digital camera. In one aspect, the marker position configuration also includes the distance between the marker carrier and the digital camera. Changes in distance between a marker configuration and a digital camera can be evaluated quickly. Markers can also be referred to as marker elements. In one aspect, markers can include marker elements.

The marker carrier has at least two carrier surface elements which are firmly connected to one another and which can be referred to as a first and a second carrier surface element. A carrier surface element is a surface element which preferably has a ratio of a long side to a narrow side, e.g. B. does not exceed a ratio of 10:1. The carrier surface element can also have a thickness on the long side and narrow side, which is in particular less than 10 mm (millimeters), preferably less than 2 mm. A carrier surface element is preferably planar. Marker elements arranged on the carrier surface element do not disappear behind the horizon of a curvature when the carrier surface element is in an angular position relative to the digital camera. The first and the second carrier surface element have an angular position in relation to one another. An angle between two carrier surface elements is preferably at least 90°. At least three of the at least six marker elements are assigned to the first carrier surface element. The marker elements have a two- or three-dimensional shape that distinguishes them from the carrier surface element. Marker elements are e.g. B. spherical objects, which are arranged on the support sheet. Further marker elements are assigned to the second surface element or to other surface elements that are present. The first carrier surface element with the associated marker elements differs from the second carrier surface element by the associated marker elements and in particular from all other carrier surface elements with associated marker elements. The distinguishability results from the different configuration of the marker elements on a respective carrier surface element. In the case of marker elements of the same type, the distances between the marker elements on a carrier surface element preferably differ. If the marker elements are present on a carrier surface element in a triangular configuration, at least one of the triangular angles is preferably present only once in the previously known marker configuration. It is also advantageous if all triangular angles that are present in the marker configuration, in particular on a carrier surface element, deviate from one another. The marker elements of the marker configuration are configured so that they can be differentiated by area.

The marker position configuration is calculated from three marker elements that are jointly visible to the digital camera and that are present in the marker configuration. A comparison algorithm is used for the calculation. The comparison algorithm is applied to three marker elements that belong to the marker configuration and a marker model configuration that are captured jointly by the digital camera in an image data pixel arrangement. In other words, the comparison algorithm is applied to the image areas of the marker elements, in particular to a set of coordinates of the respective image area. The marker model configuration emerges from the known marker configuration. The marker model configuration is a perspective representation of the marker configuration, e.g. B. is present as a data set in the computing unit. The marker model configuration is projected onto the plane of the marker image areas, in particular area by area, preferably using a projection algorithm. Mathematically, the projection is carried out with a so-called projection equation, the solution of which takes place in the computing unit. The projection is directed to three marker image areas. The three marker image areas are a subset of all found marker image areas of an image data pixel arrangement. The subgroup can e.g. B. also include five marker image areas from which different combinations of three marker image areas are formed.

Different marker model configurations are sequentially compared to three marker image areas. The comparison can also be carried out on parallel processors of a computing unit. The comparison algorithm determines at least one accuracy value of the comparison. The accuracy value can e.g. B. a coordinate difference value, an increment value or a step number value. The at least one accuracy value is associated with a marker model configuration. A comparison of accuracy values of different marker model configurations indicates which marker model configuration corresponds to the marker position configuration. It is determined by calculation which spatial position and which orientation the marker position configuration has. Position and orientation are selected based on the assignable marker model configuration, namely the marker model configuration that best fits the marker image areas. The present situation is thus recognized. In particular, the three spatial dimensions and an orientation are present in a coordinate representation. The clearly determined coordinates, which describe the position, can be further used by the computing unit in additional algorithms. The situation in question, the z. B. is occupied by an object or a person to which the marker carrier is assigned, at least according to one aspect, is thus determined. Accordingly, the situation is recognized or, to put it another way, agreed.

It is also possible with a method according to the invention to quickly determine a changed physical position. A changed objective situation is present in particular when a marker carrier assumes a second position, starting from a first position. For example, a person or an object associated with the marker carrier can assume a pose. For example, a pose is B. an inclination of a marker carrier, which corresponds to a head inclination. A pose is also a rotational position of a marker carrier, which corresponds to a head turn. According to one aspect, the pose includes a position of a marker carrier in space and an orientation of the marker carrier. One can also speak of a location and an inclination of the marker carrier. According to a further aspect, the orientation is a direction assigned to the marker carrier. In any case, there is a changed objective situation in the course of a movement. The changed position can be recorded with a digital camera. A changed physical position is a physical position assumed from an initial position. A change in the physical situation can be described as a chronological sequence of change steps, which can also be referred to as movement steps. The physical situation after a change step can be recorded in an image data pixel arrangement. According to one aspect, a starting position can be a known marker configuration that is present in the computing unit as a set of coordinates that individually includes all marker elements. The starting position can B. be determined by a detection of marker image areas with a digital camera in a first image data pixel arrangement in a computing unit. However, the starting point can also be iterative from a point in time that is in the past Evaluation of marker image areas have been determined. For this purpose z. a sequence of two-dimensional image data pixel arrays recorded in a sequence of motion with the marker carrier, preferably showing all sides of the digital camera marker carrier.

When the movement is recorded by the digital camera, a time sequence of image data pixel arrangements is recorded. Starting from a first image data pixel arrangement, from which an initial position is defined, at least one further image data pixel arrangement is recorded and supplied to the computing unit as a data set. This further, two-dimensional image data pixel arrangement has changed marker image areas as a result of a movement. Changed marker image areas indicate a changed physical location. On the one hand, it is possible that the marker carrier has moved. On the other hand, changed marker image areas can also be present as a result of a movement of the digital camera, with the marker carrier preferably being in a stationary position. It is also possible that there are changed image areas due to a relative movement between the marker carrier and the digital camera. In particular, a changed relative physical position can be determined by means of the method according to the invention.

A changed physical situation can be described by a position and/or an orientation. The changed position and/or the changed orientation can be assigned to an object. The position and the orientation can be z. B. by coordinates of a coordinate system. Data tuples can be specified for a point in the coordinate system or for a direction in the coordinate system. The origin of the coordinate system can theoretically be set in the digital camera. However, the origin of the coordinate system can also be set in a point, such as the center point of a marker carrier. It is also possible that the origin of the coordinate system is fixed in space as a fixed point. The point in time at which the changed physical position is present also determines the presence of an associated marker position configuration. On the other hand, knowledge of the configuration of the marker position can also be used to specify a physical position, in particular a changed one. A marker position configuration is definable from the marker image areas contained in a further image data pixel arrangement. When determining the marker position configuration, a marker model configuration is also used, which corresponds to the initial position. A comparison algorithm is applied to an array of selected marker image areas and a marker model configuration. During the comparison, a marker image area in the further image data pixel arrangement can be found which is at a smallest distance from a marker image area in the initial position, in particular a coordinate point assigned to the marker image area in the initial position, in comparison to other marker image areas of the further image data pixel arrangement, in particular to their respective center of coordinates . Such an image area is also referred to as the nearest neighbor. Nearest neighbors can also be used for at least three marker image areas, e.g. B. for respectively assigned coordinate points, in particular the further image data pixel arrangement.

If a marker image region does not appear in the image data pixel array because a corresponding marker element e.g. B. is covered by a second marker element, it is possible to deduce such a covered, further marker image area based on the known marker configuration. The method is particularly robust because not all marker elements of the marker carrier have to be recorded by the digital camera. Based on prior knowledge of a geometry of the marker carrier and prior knowledge of a marker configuration, it is known in which model view marker carriers and associated marker elements are visible for the digital camera. Such "visible" marker elements must be given special consideration when evaluating a marker model configuration. A result of the comparison algorithm, which is applied to marker model configurations and to marker image areas, is thus available more quickly. Furthermore, the performance of a numerical optimization method applied to the marker model configuration preselected in the comparison algorithm is improved even more.

The method gains particular accuracy in that a marker position configuration has a marker configuration of a plurality of markers at one position in space. The marker configuration is available on a marker carrier. The marker carrier has at least two surface elements. The markers on the surface elements represent a marker position configuration in the form of the marker carrier at a location in space and in an orientation in space. The markers are present in a surface element arrangement. The arrangement on a first surface element and the arrangement on a second surface element differ. However, the markers can also be spatially integrated into a surface element, e.g. B. as white or colored areas. In relation to the respective area element, the markers on the first area element have a different position than the markers on the second surface element. The markers on a first and a second surface element cannot be superimposed congruently. Preferably, the marker positions on the first surface element and the marker positions on the second surface element cannot be converted into one another by rotation, displacement, uniform stretching of the configuration, or a combination of these operations. In other words, the arrangement of markers that a first surface element has differs from an arrangement of markers that has a second surface element. It is true that three non-linearly arranged markers can always be brought into a spatial arrangement by means of a spatial rotation. Their projection onto a flat surface, such as the sensor of a digital camera, could resemble a second, flat triangular arrangement of marker elements. However, this possibility, which is known in principle, is ruled out by the marker configuration, which is related to surface elements. Marker elements that are configured on surface elements cannot be freely changed spatially in groups. Consequently, the marker elements can also be distinguished by area elements in the projection onto an image data pixel arrangement. In particular, the arrangement of marker elements on surface elements that are adjacent to one another is configured in such a way that, in the projection onto a surface in a digital camera, marker elements of a first surface element are not z. B. triangular to marker elements of a second surface element. A second surface element is adjacent to a first surface element if there is no further surface element between the first and the second surface element. A comparison algorithm, applied to the markers of the first surface element and the markers of other surface elements, will not determine any agreement between the respective marker configuration. Two surface elements that can be selected as desired from all the planned carrier surface elements differ in the configuration of the respective markers of a surface element.

A numerical optimization method is used to determine the parameters of a system that minimize or maximize an objective function. The determination is made in such a way that either a predetermined result ε of the target function is not reached or exceeded or no other parameters are found that supply a smaller or larger result of a target function. The result ε can also be referred to as the accuracy value. These parameters are said to be optimal with respect to the objective function. In a measurement system for determining a physical position in space from a two-dimensional image data pixel arrangement, the "optimal" coordinates are ideally the actual coordinates of the marker configuration.

The best fit calculation is an example of an optimization method that can be used, whereby the unknown parameters of their geometric-physical model (projection of object points in an image plane) are determined or estimated for a series of measurement data (e.g. markings in a two-dimensional image).

A method for solving such so-called adjustment problems is the least squares method, in which the squared distance of the measuring points to an estimator function is minimized as a target function in such a way that either the parameters found fall below a given distance ε, or none there are other parameters that provide a smaller squared distance from measurement points to an estimator.

• • Eine Zielfunktion unterschreitet mit den iterativ gefunden Parametern einen vorgegebenen Grenzwert ε.
• • Das Ergebnis der Zielfunktion ändert sich um weniger als beispielsweise 1% des Vorergebnisses.
• • Es wurden die maximale Anzahl an Iterationsschritten, zum Beispiel 20, durchgeführt.
• • Es können keine Parameter mehr gefunden werden die das Ergebnis der Zielfunktion weiter verkleinern.

Iterative methods for solving non-linear adjustment problems gradually approximate the parameters to be estimated, in particular coordinates, to the optimal parameters, such as coordinates, starting from initial values. Iterative processes are continued until a termination criterion is met, such as:

• • A target function falls below a given limit value ε with the iteratively found parameters.
• • The result of the objective function changes by less than, for example, 1% of the previous result.
• • The maximum number of iteration steps, for example 20, has been carried out.
• • No more parameters can be found that further reduce the result of the target function.

Examples of numerical optimization methods are the Levenberg-Marquardt algorithm or the Gauss-Newton method.

A physical position or a changed physical position can be recorded in a particularly advantageous manner using a system for presenting a field of view representation. A position is determined and evaluated in the system. A position provided by the system as a data set can be used further within the system. In this case, a position is preferably used to calculate a field of view representation. The field of view representation can B. include a view of a virtual space. A virtual space is, at least according to one aspect, a computationally generated spatial image that can be represented in visualization data che geometry. A view is defined in particular by a solid angle range, which according to one aspect can also be referred to as a field of view. The solid angle range can be changed by a movement. The movement is performed in real space. The real movement can be transferred to a representation of the virtual space. A changing field of view due to movement can be displayed to a viewer.

The system includes a portable marker carrier z. B. covers an area of a person's visual field, such as a temple area. The marker carrier preferably encloses the head on all sides on one level. The eyes can lie in the plane. The plane can also intersect the ears or the ear canals. The plane preferably extends from the eyes to the back of the head. The plane can also be parallel to a plane associated with the eyes, such as a plane of a hat brim. The marker carrier can be z. B. extend along a headband. The marker carrier can be worn with the help of a strap that encloses the back of the head. However, the portable marker carrier can also be worn on a device in the manner of a spectacle frame. The marker carrier has a very low weight of less than 1 kilogram. The marker carrier does not restrict a person's freedom of movement, or at least only marginally.

A plurality of marker elements are provided on a surface area which is directed outwards relative to an inner area of the marker carrier provided for receiving a head or an object. A marker element can be an ellipsoid. A marker element can also be represented from cylinder, cuboid, sphere, cone elements or combinations thereof. It is also possible to use marker elements that have the shape of a polyhedron. The shape of a marker element can be a body shape that contrasts with the marker carrier. There is no spacer between a marker element and the marker carrier. Rather, a marker element can also be integrated in a flat area of the marker carrier, although the marker element must differ from the surface area of the portable marker carrier in at least one optical property, such as absorption, reflectivity or a spectral property. Suitable materials for marker elements are e.g. B. paper or lightweight ceramics such as zeolite, in particular of almost white color. Suitable materials for marker carriers are e.g. B. carbon fiber panels, black anodized light metal panels or plastic panels, especially black color. Other materials can also be used, which in particular have a light weight that does not impair the wearing comfort.

The system has at least one digital camera. The digital camera can B attached to a tripod or ceiling. It is also possible to use several digital cameras in order to expand a spatial area that can be recorded by one digital camera. In particular, the digital camera has a lens with which a spatial region can be imaged sharply. It can e.g. B. a monocular lens can be used. Lenses that image a spatial area with only insignificant distortion are advantageous. The mapping takes place on a two-dimensional field of a sensor. The sensor converts the incoming light pixel by pixel into electronic data that form a two-dimensional pixel array. A photograph or an image of a film sequence from the digital camera can be transmitted to a computing unit as a first pixel data field. The computing unit is z. B. a mobile computing unit, such as a so-called smartphone, a laptop or a tablet. Algorithms, such as image data processing algorithms, can be executed on the computing unit. The system also includes an electronic image display. On the electronic image display z. B. a pixel data field can be made optically visible. However, a calculated field of view representation can also be visualized on the electronic image display.

A data transmission unit is provided in the system. The data transmission unit takes care of the data transmission connection between the digital camera and the computing unit. The communications link may also provide for the provision of electronic data for the creation of a head-up representation on the electronic image display. In other words, the electronic image display can be in connection with the computing unit. However, the electronic image display can also be an independent device independent of the processing unit and can be supplied with electronic data, in particular also with electrical energy for operation, via the data transmission connection. A wireless data transmission connection is preferably used. It is beneficial for applications of the system when the image display is freely movable. A data transmission connection between the image display and the computing unit is not interrupted by movements of a marker carrier.

The system includes at least one light source. The light source is used to emit light that is preferably visible. The light provided by the light source must be registerable with the digital camera. It is also possible that a conversion of the light, e.g. B. by fluorescence or phosphorescence on a marker element. An ultraviolet light source in particular can be useful here. Marker elements can have a fluorescent or phosphorescent coating. However, visible light is preferably used. Marker elements can serve as a light source in the manner of a glowing moon. It can also be said that the daylight provided by the sun can be used as a light source, directly or indirectly through reflection or scattering. The light can be registered by the digital camera. The light from the light source passes through the spatial area surveyed by the digital camera, at least in certain areas. It is also possible to use the fluorescence properties of marker elements to convert a non-visible spectral range of a light source into a visible spectral range on marker elements and make it recordable for the digital camera. A contrast enhancement between the marker element and the immediate vicinity of the marker element can thus be achieved, in particular in an image area of the pixel data field, which can also be referred to as a pixel data arrangement. A commonly used light source is B. a fluorescent tube or a light emitting diode (LED).

The processing unit calculates a field of view representation. The head-up representation is provided for the image display. An observer position that is occupied almost at the same time is included in the visual field representation. The marker carrier can be moved freely in the spatial area imaged by the digital camera. There are no optical barriers between the marker carrier and the digital camera. However, it is possible that parts of the marker carrier are temporarily damaged by e.g. B. hair or a scarf are covered. The space between the marker carrier and the digital camera is free of obstructions. It is Z. B. provided by a space limitation that a minimum resolution of a marker element in the pixel data field is not undercut by a change in distance between marker carrier and digital camera. The minimum resolution preferably includes one pixel for one marker element. In other words, a marker element should occupy at least one pixel. However, the minimum resolution can also be limited to a predeterminable number of, e.g. B. four, pixels for a marker element are set to ensure reliability of error-free marker detection, z. B. by excluding electronic single pixel errors to improve.

The marker carrier comprises at least five surface elements. A surface element can also be referred to as a carrier surface element. A surface element is planar at least in regions. A surface element can have an edge, such as a folding edge. According to one aspect, surface elements with a fold edge can also be referred to as two surface elements. A marker carrier that has at least five surface elements that are assigned to one another in an edge region has proven to be particularly favorable in tests. However, ten surface elements can also be advantageously put together. If surface elements are designed in a trapezoidal manner, they can be placed next to one another in a particularly favorable manner. Edge areas of further surface elements preferably in turn extend along three edge areas of the trapezoidal shape. A fourth edge area of the surface element can be free-standing. The surface elements are arranged like a ring. At least five surface elements can be connected to form a ring. In one aspect, the ring may have a pentagonal cross-sectional area. The token carrier can be placed around a head like a wreath. The marker elements are present in a configuration on a surface element of the marker carrier. The configuration of the marker elements on a surface element is unique. In other words, the arrangement of the marker elements on one surface element differs from the arrangement of the marker elements on a second surface element. The configuration of the marker elements differs from one another for each surface element.

The portable marker carrier can assume an angular position in the spatial area measured by the digital camera. The angular position is relative to a camera image plane. According to one aspect, the angular position is relative to a sensor plane of the digital camera. The angular position can be changed by moving the marker carrier. It can e.g. B. lead to a tilting movement or a rotational movement of the marker carrier to an angular position. The marker carrier can be seen by the digital camera in any angular position in the form of at least two surface elements having marker elements, a first surface element and a second surface element, which can also be called carrier surface elements. The two surface elements can be adjacent to one another, with the first surface element in particular adjoining the second surface element with an edge region. At least two surface elements that have marker elements can be filmed or photographed together by the digital camera, or in other words can be imaged. The two surface elements are preferably designed in each case in the manner of a trapezium and are planar. The two surface elements enclose an angle that is less than 180°. In other words, the second surface element is not arranged as a rectilinear, in particular not as a parallel-like continuation of the first surface element. The two surface elements can be present in one piece, but still count as different surface elements, in particular with regard to mapping onto an image data pixel arrangement.

The digital camera receives light from marker elements. There are at least four marker elements that are algorithmically recognizable by the processing unit in the pixel data field. Using the detectable marker elements, a spatial position of the field of view is determined in the computing unit by at least one comparison unit, such as a comparison algorithm. The comparison algorithm checks whether or to what extent there is a deviation between the four marker elements in a two-dimensional image and a marker model configuration. The comparison algorithm identifies the marker model configuration that has the best possible match for four marker elements. The selected marker model configuration, which can also be referred to as a marker position configuration, includes coordinates that define a spatial position of the field of view. The field of view is aligned with the marker position configuration. With the system, the spatial position of the field of view can be calculated in a time-efficient manner and fed to the image display, so that the impression of a flowing change in the field of view in the course of a movement is created.

Advantageous refinements and developments are presented below, which, viewed individually, both individually and in combination, can also reveal inventive aspects.

The marker model configuration can be determined based on a pre-selection of marker elements. A pre-selection preferably comprises three marker elements. The marker elements can be arranged on a single surface element. The marker configuration from which the marker elements are selected is previously known. For the comparison with marker image areas, configuration hypotheses of the three marker elements are preferably created, which can also be referred to as marker model configurations of the three marker elements. In particular, up to five configuration hypotheses are used. Based on the five configuration hypotheses, the comparison algorithm comes to a decision as to whether the three marker elements are suitable as a marker model configuration for further calculations. Such a procedure allows a quick checking of different marker model configurations.

The quality of a pre-selection of a marker model configuration can be checked. At least one further marker image area, in particular from the same two-dimensional image data pixel arrangement, is selected. The ability to assign the at least one further image area to the marker model configuration can strengthen the selection decision regarding the marker model configuration. It has turned out to be advantageous if a perspective visibility of marker elements is taken into account during the check. It is possible that at least one marker element is covered by the marker carrier, in particular by a surface element of the marker carrier in a position of the marker carrier or an orientation of the marker carrier in front of the digital camera. A hidden marker element of the marker model configuration can be excluded from checking another marker image area. Perspective visibility is predetermined in particular on the basis of the assignment of marker elements, in particular configured marker elements, to a carrier surface element. According to one aspect, a marker model configuration is a hypothetical image of a predetermined arrangement of marker elements on a marker carrier, which is present as a data record in a computing unit. The marker model configuration can include all marker elements arranged on a marker carrier.

A bright spot filter has proven to be a particularly reliable marker detection algorithm. In particular, bright image areas in an image data pixel arrangement against a background that is less bright are recognized as a marker image area with the bright spot filter. It can e.g. B. white marker elements can be used, which cause bright, at least partially whitish, image areas. The bright spot filter can filter out a spot geometry that can be assigned to a marker element. Bright spots in an image data pixel arrangement that do not originate from a marker element can be at least partially ruled out.

The reliability of the method can be further improved if it is taken into account that a marker element is temporarily, e.g. B. can be covered by hair or shadows. It is possible to calculate where a hidden marker element could exist. This allows z. B. refine a detection of marker image regions using a previously known marker model configuration.

The method can also be designed favorably in that an advantageous way of representing a position and a direction is used to determine the position. A position and a direction can e.g. B. can be represented mathematically with a translation vector and a rotation representation, which lead to a marker model configuration. Examples of a rotation representation are a rotation matrix, Euler angles, a quaternion, a Rodriguez representation or an angle-vector representation.

Orientation representations are used in order to be able to mathematically describe an angular position of objects in relation to one another. Furthermore they are used to mathematically describe a change in angular position, i.e. a rotational movement, of objects.

Euler angles and Rodriguez representation are 3-parametric representations. With a 3-parameter representation of the orientation, all parameters are independent of each other. This means that each parameter can be changed on its own and the parameters are still a guide. This is a great advantage, especially for numerical optimization methods. A problem can arise when an orientation representation in an angular position can be expressed by several variants of the same representation form. For example, there are 12 ways to represent a single orientation using Euler angles. Another possible problem with a form of representation is the occurrence of a gimbal lock, in which two parameters describe the same change in angular position, i.e. a rotation about the same axis. One consequence is that the descriptive parameters jump, although the angular position does not change. If one would normally like to infer an angular velocity from two measurements of the angular position by dividing the difference in the measured position parameters by the elapsed time, an incorrect result would be obtained in this case. 3-Parametric forms of representation are therefore unfavorable for applications in which the parameter is updated over time (e.g. noise filter). The Rodriguez representation offers advantages for numerical optimization methods because it is particularly easy to calculate from a quaternion representation.

Quaternion or angle-vector representation are examples of 4(four)-parametric representations. A 4-parameter display form has no undefined state like the gimbal lock and is therefore suitable for so-called "tracking" applications when tracking movement and for noise filters. However, due to overdetermination, the four parameters are not independent of each other. A change in a single parameter always results in a change in at least one other parameter. If constraint conditions of the parameters to one another are not observed, it is possible that the parameters describe rotation and deformation. This representation is therefore less suitable for methods that change individual parameters, such as numerical optimization methods, and often requires an additional correction of the deformation, e.g. B. when optimizing an orientation, the object is not only rotated but also deformed.

However, it is advantageous if the method according to the invention is equipped with a numerical optimization method. The accuracy in determining the position can be improved by means of a numerical optimization process. In a numerical optimization process, e.g. B. a Levenberg-Marquart algorithm can be used. The Levenberg-Marquardt algorithm is a numerical, iterative method for solving non-linear adjustment problems using the least squares method, which is very robust against poor initial values of the parameters to be determined. The Gauss-Newton method is another example of a numerical optimization method that can be used.

The numerical optimization method is applied to a pre-selection of marker image areas. The pre-selection includes the three marker image areas that were recognized as marker image areas with the smallest deviation from a marker model configuration. At least one further marker image area is added. In other words, the numerical optimization method is applied to at least four marker image areas. The determination of the position can thus be made more precise. In particular, an orientation can be determined more precisely. The numerical optimization method is preferably applied to a three-parametric representation of the orientation of the marker model configuration. All other points of the marker model configuration that do not match the marker image areas are also carried along by the previously known marker configuration.

In determining a change in physical location, the digital camera uses image data pixel arrays, e.g. B. with a sequence of sixty image data pixel arrays, which can also be referred to as images, provided in one second. A change in marker position configuration may have occurred from a first image to a second image. The change in the marker position configuration can also be based on a displacement of markers, which can also be referred to as marker elements, on the marker carrier. In doing so, it may be necessary to recalculate the marker model configuration, which defines the initial position. The method can be carried out particularly quickly if the markers arranged on the marker carrier are of the same design. Preferably, the marker elements assigned to the marker carrier are uniform, have the same color, in particular white or black, and are the same size. The marker carrier is designed to contrast with this, preferably black or white. A marker configuration can be calculated from the detected marker image areas. In this case, marker image areas from a plurality of image data pixel arrangements are taken into account. All marker elements are preferably shown in the plurality of image data pixel arrangements. In other words, each carrier surface element of the marker carrier should be recorded in at least one image data pixel arrangement at least with the markers arranged on the carrier surface element. The changed marker configuration is determined algorithmically. In particular, good results are obtained using a structure-from-motion algorithm. This algorithm is based on a photogrammetry method in which a three-dimensional structure that is viewed from two perspectives in two-dimensional images is measured three-dimensionally. In addition, an ensemble adjustment algorithm (bundle adjustment) can also be used to further improve the accuracy of the determined marker combination. Here, a simultaneous numerical refinement of the camera parameters (position, orientation, focus length, pixel dimensions, optical distortion) and a three-dimensional structure, which was recorded from several perspectives, takes place. A precise estimate of the point coordinates can be determined from an approximately determined point model, which was acquired from different viewing angles with uncertain camera parameters. The error between the projected three-dimensional marker model configuration and the corresponding pixels is minimized. A marker configuration is calculated recursively as a marker model configuration. The calculated marker configuration defines the initial position marker model configuration.

The marker position configuration of the changed location can be derived from e.g. B. a first and a second image data pixel array. A further image data pixel arrangement, such as the second image data pixel arrangement, can also be included in the calculation. According to one aspect, a marker image area, which is present in a first image data pixel arrangement, is assigned to a nearest neighbor image area, which is a marker image area of a further image data pixel arrangement. It is possible that a second image data pixel arrangement is skipped during assignment, because e.g. B. too few marker image areas could be found in the second image data pixel arrangement, so that a further image data pixel arrangement is used. A nearest neighbor image area is an image area which, starting from a position of a detected marker image area of a first image data pixel arrangement in a further image data pixel arrangement, has the smallest linear distance compared to all other marker image areas of the further image data pixel arrangement. A nearest-neighbor image area of the further image data pixel arrangement can preferably be found for at least three marker image areas of a first image data pixel arrangement. The first image data pixel arrangement is the starting position. By changing the three marker image areas of the initial position to the three nearest neighbor image areas, the marker position configuration that defines the changed position, e.g. B. by means of a projection or a shift of an initial position, in particular can be determined using a marker model configuration. This allows the evaluation speed to be increased even more.

The accuracy of determining the position and/or the orientation can be further optimized. An optimization is possible by means of an additive change. The optimization procedure is carried out numerically. In the computing unit z. B. a Levenberg-Marquardt algorithm can be used to optimize a marker model configuration. Here, a Jacoby matrix is approximated by numerical differentiation. The orientation is preferably optimized in a three-parameter representation of the orientation in the marker model configuration. The optimization takes place e.g. B. by means of a program code that is executable in the processing unit.

The method for determining an object's position, in particular after a change in position, can deliver even better results if a control algorithm (RANSAC, Random Sample Consensus) is provided in the numerical optimization method. At least one result of the numerical optimization process is checked. According to a first aspect, error image areas z. B. are imaged by highly reflective objects in the spatial area of the digital camera in the image data pixel arrangement, which, however, are not marker elements or markers, are excluded from the control algorithm. According to a second aspect, marker image areas that were not visible or could not be found in a first image data pixel arrangement, but which should have been able to be found according to the marker configuration, are recognized as errors and, in particular, after finding them in a second image data pixel arrangement, e.g. B. recursively, are included. The result of the numerical optimization process is checked using a random selection of at least one marker image area configuration. In other words, the RANSAC algorithm is used to clean a series of measured values, such as marker image areas or coordinates assigned to marker image areas, from so-called outliers, ie measured values with gross errors. A randomly selected subset _mi is taken from an existing set of measurements m, which contains exactly the minimum number of measurements required to generate an estimation hypothesis. This minimizes the probability that there are outliers in the subset m _i . Values dependent on a limit k, also called accuracy value that strengthen the hypothesis generated from m _i are considered as hits, so-called “inliers”. The subset whose hypothesis produces the most hits is considered the best guess. The subset is z. B. a selection of three marker image areas with which an unambiguous determination is possible. The result obtained in this way is compared with the result of the numerical optimization method, which in particular included more than three marker image areas. By repetition, it is also possible to check different marker image area configurations to see whether a similar result is consistently obtained. A hypothesis can also be referred to as a pose hypothesis or position hypothesis. The control algorithm is used in particular to check the assignment of nearest neighbors of marker image areas between a first and a second image data pixel arrangement. In other words, a match between a projection of a marker model configuration and a configuration of marker image regions from a subsequently recorded image data pixel arrangement is checked. In this way, a fundamental possibility that the method fails to determine a changed physical position with an erroneous assignment of a nearest neighbor in the time sequence can be almost completely ruled out.

A further improvement in the accuracy when determining a physical position is possible through the use of a noise suppression algorithm. It is favorable if the noise suppression algorithm filters the position and/or the orientation by means of a movement profile prediction. Position and/or orientation are obtained from image data pixel arrangements, for example in the case of a changed physical position, and are refined using a numerical optimization method. A course of motion is continuous. It is therefore possible to make a prognosis about the course of movement. A state space representation of marker model configurations is preferably used. The state space representation, which can also be referred to as a state space model, is a form of description of a dynamic transmission system. The state space model enables an analysis and synthesis of dynamic, variable systems in the time domain and can also be used efficiently, e.g. B. in the control engineering treatment of multivariable systems, as well as in non-linear and time-variable transmission systems. All relationships between the state variables, the input variables and the output variables are represented in the form of matrices and vectors. The state space model is described by two equations - the first-order state differential equation and the output equation.

• - Prädiktion: Unter Zuhilfenahme eines Prozessmodells (z. B. Differentialgleichung eines bewegenden Objekts) wird der letzte geschätzte Zustand (z. B. Position und Geschwindigkeit) für die nächste Messung vorausgesagt.
• - Innovation: Die prädizierte Messung wird mit der neuen Messung zu einem neuen Schätzergebnis kombiniert. Dabei wird Messung und Messwertprädiktion je nach Unsicherheit der Messung (Messrauschen) und Unsicherheit der Prädiktion (Modellrauschen, Kovarianz der Messwertprädiktion, Kovarianz der Schätzung) unterschiedlich gewichtet.

An example of a noise reduction algorithm is a Kalman filter. The Kalman filter is a filter applicable in state space. This allows measurements to be improved with the help of a process and measurement model, such as a marker model configuration. The Kalman filter works in the two steps of prediction and innovation:

• - Prediction: With the help of a process model (e.g. differential equation of a moving object), the last estimated state (e.g. position and speed) is predicted for the next measurement.
• - Innovation: The predicted measurement is combined with the new measurement to produce a new estimated result. Measurement and measurement value prediction are weighted differently depending on the uncertainty of the measurement (measurement noise) and the uncertainty of the prediction (model noise, covariance of the measurement value prediction, covariance of the estimate).

The Kalman filter itself is based on a set of linear mathematical equations and assumes Gaussian distributed noise. To put it simply, the Kalman filter is used in one aspect to remove the interference caused by the measuring devices, such as the digital camera. Both the mathematical structure of the underlying dynamic system, such as the movement of marker elements, and that of the measurement errors, such as e.g. B. the pixel noise to be known. In the context of mathematical estimation theory, with regard to Kalman filters, one also speaks of a Bayesian minimum variance estimator for linear stochastic systems in state space representation.

Further developments of Kalman filters can also be used, e.g. B. an Extended Kalman filter (EKF) or a linearized Kalman filter.

Another further development of a noise suppression algorithm that is preferably used in a method is the quaternion-based motion sequence filter, which can also be referred to as a quaternion-based Kalman filter. This is a non-linear filter for smoothing angular position measurements. In particular, the well-known problem of "gimbal lock" is avoided with this filter. Quaternions are particularly useful for representing rotation and solving motion tracking problems. Quaternions are also particularly compact compared to other forms of representation. These are four-parameter representations. The motion dynamics can be described as a linear model and the transition matrix, also known as Transition matrix can be called, is easy to calculate.

With three spatial coordinate axes for designating a spatial position, the transition matrix can be calculated with three angular velocities occurring around the respective coordinate axes. According to a further aspect, rotational speeds about the respective coordinate axes can be calculated in the reversal using the transition matrix from two angular positions which were determined at different points in time. The speeds of rotation are associated with a rotation that transfers an object from the first angular position to the second angular position at the respective point in time of the angular position.

The quaternion-based motion sequence filter specifically takes into account two properties of the orientation representation with quaternions. On the one hand, the motion sequence filter takes into account the property that the operator representing the rotation, in other words the rotation, namely the rotation operator, and thus a possible error associated with the orientation, is described by a multiplication and is therefore nonlinear. On the other hand, it is taken into account that the quaternion that describes the angular position of an object must always result in a unit quaternion with length 1 (considered as the absolute value of a vector).

In an embodiment of a quaternion-based additive error filter, the multiplicative rotation operation can be linearized so that it is described by an addition. In this case, the property of the unit quaternion can be preserved by normalizing the result of the filtering. In the mathematical structure of the underlying dynamic system as well as in the description of the measurement error, however, the method for determining the angular position must be viewed as a pseudo-measuring device for the four parameters of a quaternion, with each parameter being assigned a measurement error that is independent of the other parameters, what is therefore preferably to be further improved.

Another advancement of a quaternion-based motion tracking filter does not track the angular position of an object itself, but rather the change in the object's angular position. It can also be said that the multiplicative operation of the 4-parametric initial angular position with the likewise 4-parametric change in angular position gives the updated angular position that is present at the time of acquisition of a new image data pixel arrangement. In one aspect, the measurement of the new angular position from the new image data pixel arrangement may be referred to as an angular position measurement with measurement corruption. The calculation of the new angular position, which has been corrected for the measurement error, is encapsulated around the motion sequence filter. The movement sequence filter takes into account a limitation of the angular position with regard to an expected value of a movement quantity. In the movement sequence filter itself, the change in the angular position is tracked in a 3(three)-parametric form of representation and can therefore be described with a linear, additive error without distorting the rotation operation. The filtered result is converted into a 4-parameter representation with the initial angular position before the multiplicative rotation operation. The change in angular position can be measured directly, for example with angular rate sensors such as a gyroscope. It is also possible to calculate the change in the angular position, assuming uniform rotational movement in the motion process model, using the transition matrix from the initial angular position and the new measurement of the angular position. When using angular velocities collected by sensors, different data rates between sensor data and image data pixel arrangements must be compensated for. Measurements that are recorded with different data rates can be adapted by using a dynamic movement model, with which measured values in particular can be interpolated.

Through a dynamic motion model, changes in motion in position can be coupled with changes in angular attitude, resulting in improved determination of object pose, in other words, the physical location of an object.

As an alternative to a Kalman filter, a particle filter such as a condensation algorithm or a sequential Monte Carlo algorithm can also be used. These are nonlinear filters that do not require any Gaussian distributed noise processes.

The method can be used to determine the position particularly quickly and quickly if a pre-selection of a marker subgroup of the marker configuration is made as a starting point for the application of the comparison algorithm. In other words, an initial configuration of the comparison algorithm can be defined in the preselection. The marker subgroup is determined from the marker configuration. The marker subset can also be determined from a marker model configuration. The pre-selection of the marker subgroup is preferably carried out algorithmically in the computing unit, with previously known marker configuration data being used. As a marker subset, e.g. B. marker elements can be selected that have a smallest, especially average, distance. In one aspect, these are the nearest neighbors within the marker configuration. The marker subset preferably comprises three marker elements. The marker subset can also include four marker elements. The smallest mean distance is recognized in particular when a mean distance is increased by a hypothetical replacement of a marker element of the marker subgroup. The mean distance can be It can be calculated, for example, by adding up all available pairwise distances between two marker elements of the marker subgroup and dividing them by the number of distances.

According to another aspect, tests have shown that it is beneficial for the speed in determining a position or a changed position if a subgroup of marker image areas is preselected to which the comparison algorithm is applied. A first subgroup preferably includes five marker image areas. However, the first subgroup can also include three marker image areas, in particular if a numerical optimization method is to be checked. The subset is preferably selected as a nearest neighbor marker image area configuration. Here, from all the marker image areas found, the marker image areas are selected which are algorithmically evaluated as nearest neighbors to one another. The nearest neighbors can be z. B. identify that the mean distance of three marker image areas is calculated from all found marker image areas. The marker image areas with the smallest mean distance form the selected first subgroup. Further marker image areas, which have a larger mean distance than a first subgroup, can be introduced into the comparison algorithm sequentially with a larger mean distance in each case. This is preferably done when the first subgroup or a further subgroup is determined by the comparison algorithm, e.g. B. due to insufficient agreement with a marker model configuration, was discarded.

According to a further aspect, a nearest neighbor marker image area configuration can also be a first marker image area configuration from a first image data pixel arrangement and a second marker image area configuration from a second image data pixel arrangement, e.g. B. of all marker image area configurations, in particular within the marker image area configuration, have the smallest mean distance from one another.

A marker position configuration found algorithmically by means of marker image areas, which makes up the physical position, can be further refined by assigning a position and a direction of a solid angle area to the marker position configuration. In particular, it is possible to assign a changed solid angle range to a marker position configuration during a movement. A changed solid angle range is a solid angle range that is present in a second physical position. A size of the solid angle range can, for. B. be specified by two angles. The size of the solid angle range corresponds to z. B. a field of view of a human eye or a field of view of a pair of human eyes. The position can e.g. B. be the position of the head in space. The direction can e.g. B. the rotational position of the head or the rotational position of the eyes or a combination of rotational position of the head and rotational position of the eyes. The solid angle area can be converted into area image data of a virtual space. However, from image data of a real space, such as a photograph or a film, which is present in digitized form on the computing unit, it is also possible to use it to calculate area image data that is assigned to a solid angle area. In particular, image data of a three-dimensional space are used for a conversion. Area image data that can be assigned to a solid angle area can have been provided by a computer-aided design program. It is also possible to use a position and a direction in a computer-aided design program to calculate a solid angle range of a virtual space.

A system with which a visual representation can be presented preferably works with marker elements that are present in a configuration. The configuration of the marker elements can be structured by a marker carrier, in particular by surface elements of the marker carrier. The configuration of the marker elements is preferably determined by a pseudo-random distribution of the marker elements. In the case of a pseudo-random distribution, a position of a marker element is predetermined using at least one random number. Random numbers are z. B. mathematically specified by a random number algorithm. A random number can e.g. B. clearly define a coordinate or a pixel on a surface element. The term pseudo-random distribution means that the random distribution of the marker elements is subject to additional boundary conditions. For example, it is possible to specify in advance that a certain number, e.g. B. three, marker elements must be present. It can also be set that the markers elements are arranged non-linearly. In the case of a pseudo-random distribution, a boundary condition can also be specified, according to which a minimum distance of each marker element from a nearest edge is maintained in the configuration. According to another aspect, a minimum distance from a nearest edge of the marker carrier can also be required. According to yet another aspect, a minimum distance of two marker elements of the configuration can be defined in advance. With such specifications, z. B. improve the distinguishability of two marker elements. Furthermore, in a pixel data field recorded by a camera, marker elements are shown at least in regions with the background of the marker carrier, which can make it easier to find marker elements in the pixel data field. A marker carrier offers a homogeneous and even, especially dark background, which can be seen e.g. B. differs from a detectable in the spatial area background. The minimum distance is preferably at least one third of a marker element diameter. It is also possible to provide a minimum distance of half a marker element diameter or a whole marker element diameter as the minimum distance in order to further improve marker recognition in the pixel data field. All marker elements in the configuration are preferably arranged non-linearly to one another, so that the configuration can be said to be non-linear. A non-linear configuration of three marker elements is a triangular configuration. Each of these boundary conditions can further improve the determination of a position in different angular positions of the marker carrier, and the detectability of a marker element by a digital camera can be increased.

Marker elements can be found particularly well in a pixel data field if the configuration provides at least one, preferably at least four, marker image areas. Configuration also means that there is a combination of a preselected configuration of a marker carrier with marker elements. The marker image areas are provided in a recording from a digital camera. The digital camera recording also captures marker carrier image areas. At least one of the marker image areas should border on a marker carrier image area, at least in some areas. At least four marker image areas preferably border on a marker carrier image area. If additional marker image areas are contained in the pixel data field, the additional marker image areas are also identified in a two-dimensional view of the pixel data field, at least at an edge area, with a difference in brightness to a surface element of the marker carrier. A marker image area is preferably enclosed by a marker carrier image area edge.

Marker image areas can be detected particularly well with a marker detection algorithm if they are present with a high contrast in relation to an image area background. It is favorable for this if surface elements of a marker carrier have an absorption-promoting surface. A beneficial aspect of a surface is understood to be a material property that facilitates or promotes an optical effect. The marker carrier can e.g. B. to improve, ie increase, the absorption of visible light. The absorption of a surface element can also be optimized for a given spectral range. More than 50% of the light incident on a surface element is preferably absorbed. According to one aspect, an absorption-promoting surface is to be distinguished from a reflection-promoting surface. A surface that promotes reflection makes it easier for photons to leave the surface, whereas a surface that promotes absorption makes it difficult for photons to be emitted. The visible light range can e.g. B. be dimensioned between 350 and 800 nanometers. Marker elements preferably have a reflection-promoting surface. A reflection-enhancing surface is z. B. a white surface. A white surface reflects incoming light in the visible spectral range. It is also possible that the reflection takes place with a scattering of light. In other words, a reflection-promoting surface preferably does not offer any angle maintenance for incident light, in particular the lowest possible back-reflection, so that the positioning of a light source can be freely selected. A reflection-enhancing surface may, in one aspect, also be referred to as a radiation-enhancing surface. The emission may include reflection of received light. However, the emission can also include the emission of generated light. More than 50% of the light incident on a marker element is preferably reflected. Marker element surfaces that reflect broadband light are particularly advantageous. However, marker element surfaces or marker element surface coatings can also be used, which promote the emission of visible light, in particular in a spectrally restricted wavelength range. This makes it possible, for. B. by placing a suitable spectral filter in front of a digital camera to improve the findability of image areas of marker elements in an image data pixel arrangement. According to a further aspect, the ability to find marker elements in image areas can also be improved by coating them with an optical brightener. An optical brightener is a substance that B. ener converts high-energy light, such as UV light, into low-energy light, such as visible light. The optical brightener is preferably applied to the marker elements. The optical brightener can also be present as an admixture to the material that gives the shape of a marker element. Optical brighteners are z. B. particularly useful when there is daylight at the installation site of the marker carrier or at least contributes to the light of the light source. However, an additional, so-called black light source, such as a black light fluorescent lamp, can also be used in order to further improve the ability to find marker elements using a marker detection algorithm. As an alternative to passive contrast-forming marker elements, actively luminous marker elements, such as e.g. B. Marker elements, in each of which a light-emitting diode is integrated, can be used. Greater independence from available light sources can be achieved e.g. B. by integrating a light-emitting diode or an optical fiber end from a light source in at least one outwardly translucent marker element.

The computing unit of the system is preferably small and light enough to be portable. The processing unit can, for. B. be carried in one hand. It is also possible to attach the computing unit to a strap or a belt. The possible uses of the computing unit are expanded even further if the computing unit has a memory unit. The memory unit can be used, among other things, to provide data from a virtual space. However, it is also possible to transmit the data of a virtual space to the computing unit by radio, in order to make the portable computing unit as small as possible, so that it can be accommodated in an interior area of a marker carrier. The virtual space data can be used to provide positional changes or view field representations in time sequence. It is also advantageous if a control element is provided in the system. The control element can easily and quickly enable pre-selection of a spatial area positioning. A control element can be designed in such a way that it can be connected to a computing unit. For example, a control element can be in the form of a speech recognition module that evaluates a speech input via a microphone. It is also possible for the control element to be in the form of a gesture recognition unit in which one or more pixel data fields can be analyzed using algorithms. The pixel data fields are z. B. recorded optically with a digital camera and converted into algorithmically evaluable electronic data. This allows you to quickly change a field of view display without having to restart the system. The field of view representation can be further tracked by position changes in the system.

In addition to a first electronic image display, the system can also include other electronic image displays, in particular a second electronic image display. It is possible that the first electronic image display and the second electronic image display together form a field of view. A favorable arrangement is when the further electronic image display is fixedly spaced from a first electronic image display in a transverse direction. The electronic image displays are preferably arranged next to one another, with the result that an expanded field of view display is possible. According to a further aspect, the same field of view can also be presented in duplicate. This means that different viewers can concentrate on the same field of vision. In another embodiment, a first and second field of view may be presented as a stereo solid angle region. A stereo solid angle range offers an even more impressive spatial impression of a virtual environment or a virtual space for a viewer.

One embodiment of an electronic image display is video goggles. In one embodiment, video glasses have a first and a second image display. An image display is associated with an eye. Furthermore, video glasses can include at least one measuring device. Further measuring devices, such as a second measuring device, can also be provided on video glasses. A measuring device is a unit for carrying out at least one measurement of a measured variable, preferably in a chronological sequence. A measuring device includes z. B. at least one inertial sensor for measuring acceleration and rotation rates (inertial measurement unit, IMU for short), at least one eye position sensor or a visual acuity sensor. In an advantageous development, it is possible to determine a person's line of sight or a change in a person's line of sight by turning the eyes by means of video glasses, which in particular have an eye position measuring device for each eye. A viewing direction can be included in a representation, in particular of a field of view or an angular range in a virtual space. It is also possible a position sensor, the z. B. works mechanically, or to provide an acceleration sensor in the system in order to further improve an accuracy of a position determination or, according to another aspect, to further increase a speed of a position determination.

The combinations and exemplary embodiments presented above can also be used in numerous other connections and combinations consider. Individual named aspects can be freely combined with one another or combined with combinations of features as a further embodiment.

character list

• 1 ein System zur Darbietung einer Blickrichtung zeigt, wobei eine Trägerperson sich in einem Raumbereich frei bewegen kann,
• 2 einen Markerträger zeigt, in den aktivleuchtende Markerelemente integrierbar sind, und zwar in Draufsicht in 2a, in einem Durchschnitt entlang der Achse B-B in 2b, in Seitenansicht in Richtung B, in 2c sowie in einer dazu senkrechten Seitenansicht in 2d und in einer perspektivischen Ansicht in 2e,
• 3 einen Markerträger mit passiv lichtstreuenden Markerelementen zeigt, und zwar in einer Draufsicht in 3a, in einer ersten Seitenansicht, in 3b sowie in einer zweiten Seitenansicht in 3c und in einer perspektivischen Ansicht in 3d,
• 4 einen Markerträger aus 3 mit eingebauter Videobrille in Draufsicht in 4a, in erster Seitenansicht in 4b, in zweiter Seitenansicht in 4c und in perspektivischer Ansicht in 4d zeigt,
• 5 eine erste Ausführungsform eines Verfahrens zeigt und
• 6 eine zweite Ausführungsform eines Verfahrens zeigt.

The present invention can be understood even better if reference is made to the accompanying figures, which present particularly advantageous configuration options by way of example, without restricting the present invention to these, wherein

• 1 shows a system for presenting a viewing direction, with a wearer being able to move freely in a spatial area,
• 2 shows a marker carrier in which actively luminescent marker elements can be integrated, in a plan view in 2a , in an average along the axis BB in 2 B , in side view towards B, in 2c as well as in a side view perpendicular thereto in 2d and in a perspective view in 2e ,
• 3 shows a marker carrier with passively light-scattering marker elements, in a plan view in 3a , in a first side view, in 3b and in a second side view in 3c and in a perspective view in 3d ,
• 4 a marker carrier 3 with built-in video glasses in plan view in 4a , in first side view in 4b , in second side view in 4c and in perspective view in 4d indicates,
• 5 shows a first embodiment of a method and
• 6 shows a second embodiment of a method.

character description

In 1 a system 1 is shown, with which a field of view display for a wearer P can be presented. The wearer P wears a marker carrier 3 on his head at eye level, which has spherical marker elements, like the marker element 5, which are all shaped in the same way. A carrier surface element, such as the carrier surface elements 27, 27 ^v , carries at least three marker elements, such as the marker element 5. The marker carrier 3 comprises a total of thirty marker elements 5 arranged in a distributed manner by a first spatial dimension x, a second spatial dimension y, a third spatial dimension z and an orientation Θ. The first spatial dimension x, the second spatial dimension y, the third spatial dimension z and the orientation Θ can be changed individually or in combination by a movement of the wearer P. In each case there is an angular position 61 of the marker carrier 3 with respect to the camera image plane 9 of the monoocular digital camera 7 . The monoocular digital camera 7 images a cone-like spatial area 11 with the solid angle area α. The carrier P can move freely in the spatial area F, in particular when determining the position, dance, hop or do pirouettes. Even if the person P bends down or z. B. straightens or stands up from a horizontal with the upper body, the movement can be followed precisely. However, the wearer stays within the spatial area 11 . When a limit of the spatial area 11 is reached, a perceptible warning signal is automatically output by the computing unit 13 . There is a marker position configuration 46 at a point in time. A first carrier surface element 27 of the marker carrier 3 is angled to form a second carrier surface element 27 ^v . In the position of the carrier P shown, the first carrier surface element 27 is imaged by the digital camera 7 , whereas the carrier surface element 27 ^v is hidden from the digital camera 7 . The marker element 5 on the first carrier surface element is imaged by the digital camera and electronically converted into an image data field 21, which can also be referred to as an image data pixel arrangement. The image data field 21 is z. B. on an electronic image display 19 of a computing unit 13 can be displayed. Data can be sent and received via the data transmission connection 17, which is designed as a line connection from the monoocular digital camera 7 to the computing unit 13. The data transmission unit 15 is also connected to the arithmetic unit 13 and the data transmission connection 17 ′ maintains a bidirectional radio connection to a receiving unit on video glasses 23 which are located in an interior area of the marker carrier 3 .

An embodiment of a marker carrier 3' is in 2 shown, with an absorption-promoting surface 29. The marker carrier 3' is provided for the installation of self-illuminating marker elements, current-operated light-emitting diodes (LEDs) serving as the light source (not shown). The figures 2a , 2 B , 2c , 2d and 2e are different views of marker carrier 3'. The marker carrier 3 'has in the view 2a the carrier surface elements 27, 27', 27'', 27''' and 27 ^IV , which are arranged like a ring. The carrier surface elements 27, 27' have an angular position 63 of 72°. In a top view, the five surface elements 27, 27', 27'', 27''', 27 ^{IV face} a digital camera (not shown). 2 B shows one Section along the BB axis 2a , where the free interior area 46 can be seen in which an object or a head can be placed. In the side view of 2c who are heading towards B 2a lies, it is shown that the first carrier surface element 27 has an angular position 63' relative to the second carrier surface element 27 ^V , which is 80°. Also in the view along the BB axis 2a , in the 2d is present, two carrier surface elements 27', 27'' can be seen. How based on 2c , 2d and particularly 2e is illustrated in a perspective view based on individual positions of the marker carrier 3', at least two carrier surface elements, such as the carrier surface elements 27', 27'', 27''', can also always be imaged in any perspective angular position of the marker carrier 3', although individual carrier surface elements, like the carrier surface element 27 ^V 2c , in 2e are covered due to the viewing direction shown in the interior 46.

In 3 a marker carrier 3" is shown, where 3a a top view and the 3b and 3c a side view and 3d are a perspective view. The marker carrier 3 "carries marker elements like the marker element 5. The top view of 3a shows that the configurations of marker elements 41, 41', 41'', 41''', 41 ^IV provided on the carrier surface elements 27, 27', 27'', 27''', 27 ^IV differ from each other differentiate. The marker elements, such as the marker element 5, have a reflection-promoting surface 31 that promotes the emission of light. The surface elements, such as the surface element 27", each have an absorption-promoting surface, such as the absorption-promoting surface 29, which promotes the absorption of visible light. Such as e.g. 3b occupied, the configurations on the carrier surface elements differ, such as the configuration 41, 41' on the carrier surface element 27, 27', and the configurations of marker elements on the other carrier surface elements, such as the configuration 41 ^V on the second carrier surface element 27 ^V . Accordingly, there is no similar configuration of marker elements 41, 41', 41'', 41''', ^41IV , ^41V on the marker carrier 3''. The distinguishability of the configurations, such as configuration 41'', from one another is also in 3c apparent. The marker elements, such as marker element 5, 5′, 5″, 5″″, 5 ^IV , 5a, are present in a pseudo-random distribution 42 and are of the same design with a marker element diameter 39 . There is a distance 37 between the marker element 5' and the marker element 5'' which happens to be approximately twice the diameter 39 of the marker element. The distance 37' between marker element 5 and marker element 5' happens to be approximately four marker element diameters 39, a marker element 39 also being able to be referred to as a minimum distance between marker elements 5, 5', 5''. The configuration 41'' with the marker elements 5, 5' and 5'' is therefore present as a scalene triangle. The minimum distance 38 from the edge 33, which is maintained by the marker element 5, is half a marker element diameter 39. Behind the edge 33 are marker elements, such as the marker element 5a, hidden. In the perspective view of 3d the surface elements 27, 27', 27'' can be seen as trapezoidal surface elements. The surface elements 27 and 27' adjoin one another at the edge 35. The surface elements 27 and the covered surface element ^27V adjoin one another at the edge 35'. Edge 35 can also be referred to as a short-side edge and edge 35' can be referred to as a long-side edge. Configuration 41 with carrier surface element 27 cover configuration 41 ^V on carrier surface element 27 ^V . Although carrier surface element 27 ^IV can only be seen line-like in a side view, configuration 41 ^IV stands out in the perspective view of FIG 3d presented image area and can therefore be imaged and evaluated by a digital camera (not shown).

Another embodiment of a marker carrier 103 is in 4 in front. In the top view of 4a the first surface elements 127, 127′, 127″, 127′″, 127 ^IV are shown in a ring-like closed, pentagon-like arrangement. Video glasses 123 are attached to the marker carrier 103 in an interior area 146 . The video goggles 123 include the electronic image display 119 for the right eye and the electronic image display 119' for the left eye of a wearer (not shown). There is also an additional measuring device 125 . With the help of the marker elements, such as marker element 105, the position and the orientation of the marker carrier 103 can be clearly defined. The side views 4b and 4c of the marker carrier 103 show that the video glasses 123 are fully integrated into the inner area 146 and attached to the marker carrier 103 . In the perspective view in 4d the data transmission unit 150 is shown, with the measurement data from measuring devices, such as the measuring device 125 4a on the video glasses 123, can be sent to a computing unit or data representing a field of view of a virtual environment can be received and presented in the video glasses 123.

A flowchart for a method 200, 201 according to the invention, starting from the start of the method 202 in a system (not shown), is shown in 5 shown. The process scheme 200, 201 includes various options for the process sequence. The determination of a physical position 230 is provided in a method sequence 200 . In a further procedure 201 is the determination of a physical Position 230 provided after a position change. The determination of the position 230, which has changed in particular, can also be referred to as the estimation of a pose. The procedure 200, 201 of 5 includes steps to determine a physical location 230 that are used to so-called initialize the system 288. The procedure of 5 also includes method steps for determining a changed physical position, also referred to as measuring a new pose 222 . The new pose 222 was assumed in the course of a movement and can therefore also be referred to as the detection of a state of movement. Motion states can e.g. B. can be captured as snapshots or stroposcopically.

An image is recorded with a monoocular digital camera 207, from whose image data pixel arrangement 221 the marker detection algorithm 204 evaluates marker image areas 206, from which 2D (two-dimensional) coordinates 208 in the recorded image 221 of all possible markers are determined. If the system has not yet been initialized 212 , initialization steps are carried out first, with a known geometry of the marker carrier 236 and a known marker configuration 234 being included in the determination of the marker model configuration 242 . Three marker model points 264 or, if necessary, other marker model points 238 in the case of a repetition are selected from the marker model configuration 242 and compared with three marker image areas 262, with up to five pose hypotheses 266, which can also be referred to as hypotheses for the position of the three marker model points, iteratively in the two-dimensional Arrangement of the selected marker image areas 262 are projected 268. A comparison 270 is then carried out in order to find further matches between marker image areas and projected theoretically visible model points. In the iteration 272, the respective next pose hypothesis 274 is then compared and when all pose hypotheses for three marker image areas 276 have been checked, in a next iteration step 278 an ordered, in particular sequential check of random samples or variations of three marker image areas 260 takes place, taking into account other points 238, which enter into the comparison with three marker model points 264 as three marker image areas 262 . If all iterations over all variations 280 of marker image areas and model points have been checked, if no plausible hypothesis has been found 282 in the meantime, a next image 221'' is evaluated. If a suitable hypothesis for a suitable marker model configuration was found in one of the iteration steps 284 based on an accuracy value (not shown), the entire marker model with pose hypothesis is projected into the two-dimensional image with the marker image areas 268' in order to assign marker image areas and corresponding model points 286 obtain. A numerical optimization of the hypothesis for the position based on the marker model configuration or the hypothesis for the pose with regard to the pixel distances between projected marker model points and corresponding marker image areas is then carried out 220' using an algorithm such as the Levenberg-Marquardt algorithm. After completion of the numerical optimization, the system is initialized 288 and the position in question is determined using the known marker configuration 234 and the known geometry of the marker carrier 236 .

When the method initialization 288 has taken place, a starting position is known through the determination of a physical position, which can be further used to initialize the quaternion-based motion sequence filter with position and orientation 290 . A next image 221 ′ is recorded by the monoocular digital camera 207 and the two-dimensional coordinates 208 of all possible markers in the recorded image are evaluated using a marker detection algorithm 204 . If the initialization is already present in the method 210, the 2D coordinates of marker image areas 208a of the image data pixel arrangement are used from a previously evaluated image, such as the image 221, and to determine nearest neighbors 214 with the 2D coordinates from the next image 221 ' compared in a comparison algorithm. From this, hypotheses, so-called marker model configuration hypotheses, are set up about a possible correspondence for a comparison between marker image areas and points of the marker model configuration 216. In other words, a projection of the marker model configuration, which is assigned to the previous image data pixel arrangement, onto the marker image areas of the next image 221, which is also the next Image data pixel arrangement can be referred to performed. With a numerical optimization method, a comparison is carried out with regard to the projected points of the marker model configuration and the corresponding marker image areas in order to find the appropriate projection, in particular the correct projection parameters 220. A projection algorithm is used here. To improve the hypotheses about the correspondence 216, random selections of model points are checked using the algorithm for finding outliers 218, which can also be referred to as the control algorithm for eliminating error image areas (RANSAC). The determination of the physical position 222 after a position change is further improved by using a quaternion-based motion sequence filter 224, in which the movement, e.g. B. a head, 226 is taken into account with a prognosis 228 about a steady development. The pose obtained in this way with good accuracy or the changed physical position is then included in a representation or projection of a three-dimensional (3D) computer graphic 232 .

In 6 an embodiment of the method 300, 301 is shown in which, similar to the method 200, 201 according to 5 , a method 300 for determining a physical position for method initialization 388 with a method 301 for determining a physical position 322 after a position change, in particular for measuring a new pose, are combined with one another. From the start 302 to the creation or display of a three-dimensional (3D) computer graphic 332, there are numerous similarities with the method of FIG 2 before, which is why with regard to the detailed process flow for the aspects under the reference number 307, 321, 304, 306, 308, 321', 312, 321", 334, 336, 342, 338, 364, 366, 372, 374, 368, 370 , 359, 382, 384, 368', 386, 320', 388, 390, 310, 308a, 314, 316, 318, 320, 324, 328, 326, 330 to the corresponding reference numbers of 5 with reference numbers reduced by the number 100 and the descriptive text 5 is referenced. However, upon determining that the system is not yet initialized 312, a different iteration is performed. An i-th marker image area (M _i ) is iteratively selected 350 from the found marker image areas 306 and the five next neighboring marker image areas around the i-th marker image area (M _i ) are identified and combined as a group 351 . An iteration then takes place over all ten unordered random samples that can be created from a possible selection of three out of five marker image areas 352. If a combination has already been checked 353, then in the iteration of three out of five combinations a next iteration step 378' is carried out and one further combination considered. If the combination has not yet been checked 354, the three marker image regions 362 to be considered are iteratively brought together across all permutations 355 in up to five pose hypotheses 366, which are to be compared 370 with the projection of the marker model configuration. If all of the five pose hypotheses from three marker image areas have been checked 376, then a next permutation 356 in the iteration over all permutations 355 is processed. If all permutations from three marker image areas have been checked 357, then a further combination of three out of five marker image areas is subjected to the check. If all combinations of five marker image areas have been checked 358, then in a further iteration step 378" an i+1-th marker image area (M _i+1 ) is selected in order to form the next group for this i+1-th marker image area (M _i+1 ). of five adjacent marker image areas for further iterative tests. Here "i" is a natural number that is used to number different marker image areas. Tests have shown that with such an iterative nesting a considerable time gain is achieved when determining a physical layer 388 will.

However, in a further embodiment, it is also possible to carry out the method for determining a physical position (200, 300) iteratively, i. H. repeated on a different database, to determine a physical position after a change in position and thus to follow a movement. A starting position can be provided. A marker model configuration from marker image areas of further image data pixel arrangements can be improved with a quaternion-based motion sequence filter by means of a motion course prediction. The movement sequence can thus be included in a field of view representation of a 3D computer graphic.

The design options shown in the individual figures can also be connected to one another in any form. In particular, individual aspects of an exemplary embodiment can be combined with a second exemplary embodiment.

Reference List

1

System, in particular system for presenting a field of vision display

2

present situation

3, 3', 3'', 103

Marker carrier, in particular marker carrier image area

5, 5', 5'', 5''', 5IV, 105

Marker element, in particular marker image area

5a

Marker element, in particular hidden marker element

7, 207, 307

Digital camera, in particular mono-ocular digital camera

9

camera image plane

11

space area

13

unit of account

15

data transfer unit

17, 17'

Data transmission connection, in particular image data acquisition connection

19, 119, 119', 21

electronic image display, in particular display of the visual field, pixel data field, in particular an image

23, 123

video glasses

125

Measuring device, in particular acceleration sensor

27, 27'', 127

first surface element, in particular carrier surface element

27', 27v, 127'

second surface element, in particular carrier surface element

27''', 27IV, 127'', 127''', 127IV

further surface element, in particular carrier surface element

29

absorption-promoting surface

31

reflective surface

33

edge

35, 35'

edge

37, 37'

Distance, especially between two marker elements

38

Minimum distance, especially to an edge

39

Marker element diameter, especially minimum distance size

41, 41', 41'', 41''', 41IV, 41V

Configuration, in particular arrangement of marker elements on at least one carrier surface element

42

Pseudo-random distribution, in particular the configuration of the marker elements

43

Marker position configuration, especially three-dimensional arrangement

46, 146

interior

150

Data transmission unit, in particular transmitter and receiver

61

Angular position, in particular of the marker carrier

63, 63'

Angular position, in particular of the surface elements

200, 300

Method, in particular for determining a physical situation

201, 301

Method, in particular for determining an object's position after a change in position, preferably from a second image data pixel arrangement

202, 302

Start, in particular, of the procedures in the system

204, 304

Marker detection algorithm, especially bright spot filter

206, 306

marker image areas

208, 308

2D coordinates of all possible markers in the image

208a, 308a

2D coordinates from the previous image data pixel arrangement, in particular the initial position

210, 310

Process initialized, in particular system has already been initialized

212, 312

Procedure not initialized, in particular the system is not yet initialized

214, 314

Comparison algorithm, in particular nearest neighbor marker image area

216, 316

Marker model configuration hypotheses to determine correspondences to marker image areas through comparisons

218, 318

Control algorithm to find and exclude outliers, such as error image areas, especially RANSAC

220, 320

numerical optimization method with regard to projected marker model configuration and corresponding marker image areas in order to find projection parameters, in particular Levenberg-Marquardt algorithm

220', 320'

numerical optimization method with regard to projected marker model configuration and corresponding marker image areas, in particular Levenberg-Marquardt algorithm

221, 321

Image data pixel arrangement, in particular image

221', 221'', 321', 321''

Further, in particular next, image data pixel arrangement

222, 322

Determination of a changed object position, in particular new measurement of a pose

224, 324

Filter algorithm, in particular quaternion-based motion sequence filter

226, 326

Movement model of a marker carrier

228, 328

Prognosis of a course of movement

230, 330

Determination of an object position, in particular a changed one, preferably estimation of a pose

232, 332

3D computer graphics, in particular representation of the spatial position of a field of view in a solid angle range

234, 334

known marker configuration, in particular arrangement of the markers

236, 336

known geometry of the marker carrier, in particular carrier surface elements

238, 338

choose other points

242, 342

marker model configuration

350

Iteration over all marker image areas (M _i )

351

Group of the 5 nearest adjacent marker image areas around a selected marker image area (M _i )

352

Iteration over all 10 unordered samples (combinations) of 3 out of 5 marker image areas

353

combination has already been tested.

354

Combination has not yet been tested

355

Iteration over all permutations of three marker image areas

356

next permutation, specifically next iteration

357

all permutations from three marker image areas are checked

358

all combinations of five marker image areas are checked

359

all marker image areas (M _i ) are checked

260

Iteration over all ordered samples (variations) of three marker image areas

262, 362

three marker image areas

264, 364

three points of marker model configuration

266, 366

Up to five pose hypotheses, especially configuration hypotheses

268, 268', 368, 368'

Projection of the marker model configuration with pose hypothesis into the 2D image, in particular the projection algorithm

270, 370

Comparison for assignability of preferably further correspondences between marker image areas and projected, theoretically visible points of the marker model configuration, in particular comparison algorithm

272, 372

Iteration over all five pose hypotheses, especially configuration hypotheses

274, 374

next pose hypothesis

276, 376

all pose hypotheses from 3 marker image areas are checked

278, 378', 378''

next iteration step

280

End of iteration over all variations

282, 382

no pose hypothesis plausible

284, 384

suitable pose hypothesis found, in particular based on an accuracy value

286, 386

Assignment between the marker image area and the corresponding point of the marker model configuration, preferably using an accuracy value, in particular a comparison algorithm

288, 388

Procedure is initialized, in particular a starting position in the system is determined

290, 390

Initialization of the quaternion-based motion sequence filter with position and orientation

x

first spatial dimension

y

second spatial dimension

e.g

third spatial dimension

a

solid angle range

θ

orientation

B-B

axis

B

Direction, especially orientation

f

Spatial area, especially movement area

P

carrier person

Claims (5)

System (1), which presents a visual field representation (119, 119') in a physical position in a changeable solid angle range, and carries out the following steps: Determination of a first and second physical position (2, 222, 322, 288, 388), in particular after taking a Pose of a marker carrier (3, 3', 3'', 103, 236, 336), with a starting position (208a, 288, 388, 308a) as a first physical position, by capturing marker image areas (5, 5', 5 '', 5''', 5 ^IV , 105, 206, 210, 306, 310, 350, 351, 352, 355, 356, 357, 358, 359, 260, 262, 362) of a first image data pixel array (21, 221 , 221', 221'', 321, 321', 321'') in a computing unit (13), and at least one further image data pixel arrangement (21, 221, 221', 221'', 321, 321', 321'') of the computing unit, which after a movement of the marker carrier (3, 3', 3'', 103, 236, 336) or after a movement of the digital camera (7, 207 , 30 7) changed marker image areas (5, 5', 5'', 5''', 5 ^IV , 105, 206, 210, 306, 310, 350, 351, 352, 355, 356, 357, 358, 359, 260, 262, 362), wherein a position (x, y, z) and/or an orientation (Θ, B) of the second physical layer (2, 222, 322, 288, 388) can be specified using a marker position configuration (43). is, from the marker image areas (5, 5', 5'', 5''', 5 ^IV , 105, 206, 210, 306, 310, 350, 351, 352, 355, 356 defining the marker position configuration (43), 357, 358, 359, 260, 262, 362) of the further image data pixel arrangement (21, 221, 221', 221'', 321, 321', 321'') and from a marker model configuration (242, 342, 264, 364) of Initial position (208a, 288, 388, 308a) by means of a comparison algorithm (214, 314, 216, 316, 220, 320, 270, 370, 286, 386), while determining a nearest neighbor (214, 314) to a marker image area (5th , 5′, 5″, 5″″, 5 ^IV , 105, 208a, 308a) of the starting position (208a, 288, 388, 308a), further marker bi that are hidden during extrapolation ld areas (5a) with a numerical optimization method (220, 320) is calculated, the marker position configuration (43) having a plurality of marker elements (5, 5 ', 5'',5'', 5 ^IV , 105) on the at least two surface elements (27, 27', 27'', 27''', 27 ^IV , 27 ^v , 127, 127', 127'', 127''', 127 ^IV ) which have a marker carrier (3, 3 ', 3'', 103, 236, 336) for the marker position configuration (43), in a surface element-wise arrangement (41, 41', 41'', 41''', 41 ^IV , 41 ^V , 234, 334). , wherein an arrangement of marker elements (5, 5', 5'', 5''', 5 ^IV , 5a, 105) which one of the surface elements (27, 27', 27", 27''", 27 ^IV , 27 ^v , 127, 127', 127'', 127''', 127 ^IV ), of an array of marker elements (5, 5', 5'', 5''', 5 ^IV , 5a, 105) which another, neighboring one, which has surface elements (27, 27', 27'', 27''', 27 ^IV , 27 ^v , 127, 127', 127'', 127''', 127 ^IV ), differs, whereby a noise reduction algorithm in the form of a quat ion-based movement sequence filter (224, 324), which filters the position (x, y, z) and/or the orientation (Θ, B) by means of a movement course prediction (228, 328), with a portable marker carrier (3, 3' , 3'', 103, 236, 336), which comprises a plurality of marker elements (5, 5', 5'', 5''', 5 ^IV , 105, 5a), with precisely one digital camera (7, 207, 307) with which a spatial region (11) can be recorded and imaged in a two-dimensional pixel data field (21), with a computing unit (13) for executing image data processing algorithms (200, 201, 300, 301, 204, 304, 218, 318, 220 , 320, 224, 324, 268, 268', 368, 368', 270, 370, 286, 386), with at least one electronic image display (19, 119, 119', 23, 123) for presenting the visual field display (119, 119 '), with a data transmission unit (15), with the one, in particular wireless, data transmission connection (17, 17') between the digital camera (7, 207, 307) and the processing unit (13) and the image display e (19, 119, 119', 23, 123) and with at least one light source for emitting light that can be registered with the digital camera (7, 207, 307) into the spatial area (11), the visual field display (119, 119') can be provided by the processing unit (13) for the image display (19, 119, 119', 23, 123) and wherein the marker carrier (3, 3', 3'', 103, 236, 336) is free of obstructions in the spatial area (11). is movable and the marker carrier (3, 3', 3'', 103, 236, 336) has at least five surface elements (27, 27', 27'', 27''', 27 ^IV , 27 ^v , 127, 127', 127'', 127''', 127 ^IV ) which are arranged in a ring-like manner, with a surface element (27, 27', 27'', 27''', 27 ^IV , 27 ^v , 127, 127', 127'',127''', 127 ^IV ) has at least three marker elements (5, 5', 5'', 5''', 5 ^IV , 105, 5a), characterized in that the marker elements (5, 5', 5 '', 5''', 5 ^IV , 105, 5a) in a surface element-wise deviating configuration (41, 41', 41'', 41''', 411 ^IV , 41 ^V , 234, 334, 236, 336). and at least two surface elements (27, 27', 27 '', 27''', 27 ^IV , 27 ^v , 127, 127', 127'', 127''', 127 ^IV ), a first surface element (27, 27'', 127) and a second surface element (27 ', 27''', 27 ^IV , 27 ^v , 127', 127'', 127''', 127 ^IV ) can be imaged at least in regions by the digital camera (7, 207, 307), with at least four being algorithmically generated by the computing unit (13) in the pixel data field (21) detectable marker elements (5, 5', 5'', 5''', 5 ^IV , 105, 5a) are present, and by at least one comparison unit (214, 314, 216, 316, 220, 320, 270, 370, 286, 386) in the arithmetic unit (13) the four marker elements (5, 5', 5'', 5''', 5 ^IV , 105, 5a) into a marker configuration data (234, 334, 236, 336) calculated marker model configuration (242, 342, 286, 386, 220, 220', 320, 320', 230, 330), which defines a spatial position of the visual field (232, 332), can be assigned and the electronic image display ( 19, 119, 119', 23, 123) is a pair of video glasses (23, 123) which comprises at least one (125), preferably two, measuring devices, such as an eye position measuring device, a position sensor system or an acceleration sensor (125). System (1) after claim 1 , characterized in that the configuration (41, 41', 41'', 41''', 41 ^IV , 41 ^V , 234, 334, 236, 336) of the marker elements (5, 5', 5'', 5''', 5 ^IV , 105, 5a) is determined by a pseudo-random distribution (42), in particular in the configuration (41, 41', 41'', 41''', 411 ^IV , 41 ^V , 234, 334, 236 , 336) a minimum distance (37, 37', 38) of each marker element (41, 41', 41'', 41''', 411 ^IV , 41 ^V , 234, 334, 236, 336) from a nearest edge (33 ), preferably from a nearest edge (35, 35') of the marker carrier (3, 3', 3'', 103, 236, 336), and in particular a minimum distance (37, 37') from two marker elements (5, 5', 5'') to each other, preferably of at least one third of a marker element diameter (39), and preferably a non-linearity of the configuration (41, 41', 41'', 41'''', 41 ^IV , 41 ^V , 234, 334, 236, 336) is specified. System (1) according to one of Claims 1 until 2 , characterized in that the configuration (41, 41', 41'', 41''', 41 ^IV , 41 ^V , 234, 334, 236, 336) has at least one, preferably at least four, marker image areas (5, 5', 5 '', 5''', 5 ^1V , 105, 206, 306, 214, 314) which at least partially border on a marker carrier image area (3, 3', 3'', 103), preferably from a marker carrier image area (3, 3', 3'', 103) are bordered. System (1) according to one of Claims 1 until 3 , characterized in that the processing unit (13) is a one-handed portable processing unit 13) with a storage unit for providing position changes or field of view representations (119, 119') in chronological order and in particular a control element for selecting a spatial area positioning is connected to the processing unit (13). is, like a speech recognition engine or an optical gesture recognition algorithm. System (1) according to one of Claims 1 until 4 , characterized in that the field of view (232, 332) is presented on a further electronic image display (19, 119) which is spaced from a first electronic image display (119') in a transverse direction, duplicated, in particular as a stereo solid angle area.

Images