DE102014217090B4 - Method for operating an augmented reality system or a virtual reality system - Google Patents

Abstract

Method for operating an augmented reality system or a virtual reality system with a display (11) for displaying a virtual image component (VIRT) and with a tracking system for determining the alignment and/or the position of the display in relation to a real object, the Augmented reality system has a display computer (20) for such control of the display (11) that the real object (25) is superimposed by the virtual image component (VIRT) or supplemented virtually, characterized in that the augmented reality system a measuring system is assigned, which comprises a test marker (31) and a light sensor (32), the state of the test marker (31) changing, with the display (11) or a substitute display (11') showing a Light sensor (32) identifiable test image is generated when the tracking system detects the state change, and wherein at least a first latency time (τ) from the generation of the state change of the T est marker (31) is measured until the test image is recognized by the light sensor (32).

Description

The invention relates to a method for operating an augmented reality system or a virtual reality system, a method for generating an augmented reality image or a virtual reality image or for displaying a virtual image component on a transparent display.

The website www.oculusvr.com/blog/latency-tester-pre-orders-now-open discloses a so-called Oculus latency tester for determining the latency of games.

the EP 1 768 006 B1 discloses a method for setting systems in which objects are moved, with data being recorded and stored at the same time that describe movement processes and data that correspond to control-relevant signals, with data describing movement processes being recorded using a tracking system, with data , which correspond to control-relevant signals, are derived from a sensor, an actuator, a controller and a computer by means of a signal acquisition system. A time correction is provided, which corrects the time delay of the tracking system to the overall system.

the DE 10 2004 051 607 B4 discloses a method in which a digital image is processed by means of a rendering process by geometrically distorting it using a two-dimensional pixel offset field containing information about a projection surface and/or manipulating the color using a two-dimensional surface texture of the projection surface. It is provided that a test image is projected onto the projection surface and images of the projection surface are repeatedly recorded with a camera until the test field is identified in the recorded images. The time from projection to identification is measured as latency.

AZUMA, Ronald, et al. Recent advances in augmented reality. Computer Graphics and Applications, IEEE, 2001, Vol. 21, No. 6, pp. 34-47. [IEEE] doi: 10.1109/38.963459 discloses an augmented reality system with optical see-through displays, in which the real object is overlaid with the virtual image component or supplemented virtually.

JACOBS, Marco C., et al. Managing latency in complex augmented reality systems. In: Proceedings of the 1997 symposium on Interactive 3D graphics. ACM, 1997. pp. 49-ff. [ACM] doi: 10.1145/253284.253306 shows an augmented reality system with an end-to-end measurement of latency using a blinking LED.

DI LUCA, Massimiliano: New method to measure end-to-end delay of virtual reality. Presence: Teleoperators and Virtual Environments, 2010, vol. 19, no. 6, pp. 569-584 describes a method for operating a virtual reality system with a tracking system for determining the alignment of the display for the superimposition of the virtual objects. In particular, DI LUCA teaches a latency measurement system for determining the latency of the tracking system, which comprises two light sensors. A test marker's state is changed by moving it in an illumination gradient. However, the virtual representation of the tracked marker on a display is compared with a direct recording to determine the latency as a phase angle of the signal intensities.

It is the object of the invention to specify an improved augmented reality method or an improved virtual reality method, in particular in the case of dynamic processes.

The aforementioned object is achieved by a method for operating an augmented reality system or a virtual reality system with a display for displaying a virtual image component and with a tracking system for determining the orientation and/or the position of the display in relation to a real object, the Augmented reality system has a display computer for such control of the display that the real object is superimposed by the virtual image component or virtually supplemented, characterized in that the augmented reality system is assigned a measurement system that has a test marker and a Includes a light sensor, with a change of state of the test marker taking place, with a test image that can be identified by means of the light sensor being generated on the display or a replacement display (by means of the display computer) when the tracking system detects the change of state, and with (by means of the measuring system) at least a first latency from the gen ung of the state change of the test marker is measured by the light sensor until the test image is recognized.

Augmented reality in the aforementioned sense should in particular include so-called augmented reality. However, augmented reality in the aforementioned sense can also include so-called virtual reality. Augmented reality is intended to include the overlaying of a real object or a real environment with computer-generated information. Virtual reality is understood as the experience of computer-generated scenes in immersive environments.

A display computer within the meaning of the invention serves in particular to render virtual image components.

A virtual image component within the meaning of the invention can include or consist of alphanumeric characters. A virtual image component within the meaning of the invention can include alphanumeric characters and a colored background for these characters. A virtual image component within the meaning of the invention can consist of alphanumeric characters in connection with a colored background for these characters. A virtual image component within the meaning of the invention can be a drawing or include one. A virtual image component within the meaning of the invention can be a sketch or include one. A virtual image component within the meaning of the invention can be an icon or include one. A virtual image component within the meaning of the invention can be or include an animation.

A display within the meaning of the invention is in particular a transparent or a semi-transparent display. A display within the meaning of the invention is in particular a matrix display. A display within the meaning of the invention is in particular a display with a plurality or a large number of individually controllable pixels.

A test marker within the meaning of the invention is in particular a light source, in particular an infrared light source. A light source is in particular an LED or an infrared LED (infrared diode). A light source is in particular a plurality of infrared diodes. A suitable light source or infrared diode has a switch-on time of no more than 100 ns. A light sensor within the meaning of the invention is in particular a photodiode. A suitable photodiode is, for example, the SFH 205 photodiode.

A change in the state of the test marker is, in the context of the invention, in particular the change in its appearance. A status change within the meaning of the invention is in particular not a change in a position of the text marker. A state change of the test marker within the meaning of the invention is in particular the change from non-illuminating to illuminating. A state change of the test marker is, in particular, switching on a diode or an infrared diode.

A replacement display within the meaning of the invention is in particular a display that is used instead of the display to measure the latency. When measuring the latency time, however, the display cannot be replaced by a replacement display.

A test image within the meaning of the invention is in particular a monochromatic, geometric figure, for example. A test image within the meaning of the invention is in particular a white geometric figure.

The at least first latency time is measured in particular as the difference between two time stamps generated in the measurement electronics. In this case, in particular, measurement electronics are provided which bring about or initiate the state change of the test marker and generate a first time stamp at this point in time. The measurement electronics are also connected to the light sensor and generate a second or further time stamp when the light sensor recognizes the test image. In this case, the latency time is determined in particular as the difference between the second time stamp and the first time stamp. Further time stamps can be generated between the first and the second time stamp by means of the measuring electronics, by means of which latencies of individual parts of the augmented reality system, such as the tracking system or the rendering, can be determined by means of the display computer.

In a further advantageous embodiment of the invention, the display or the substitute display for measuring the latency time is placed between the light sensor and a background.

In a further advantageous embodiment of the invention, a negative test image, which can be identified by means of the light sensor, is generated (by means of the display computer) on the display or a replacement display before the tracking system recognizes the change of state. A negative test image within the meaning of the invention is in particular an image which the light sensor can (clearly) distinguish from the test image. In particular, a negative test image has a color that is complementary to the test image. In particular, it is provided that the negative test image is a black geometric figure and the test image is a white geometric figure, or that the negative test image is a white geometric figure and the test image is a black geometric figure.

In a further advantageous embodiment of the invention, the measurement system is separated from the augmented reality system.

In a further advantageous embodiment of the invention, the position of the display (relative to the real object) at a current time is replaced by an estimated position of the display (relative to the real object) at a time that corresponds to the current time plus the first latency time.

A further object of the invention is to improve the production of technical components, in particular the costs of producing motor vehicles or their repair.

The above object is - especially in connection with the aforementioned features - by a method for producing a technical component components, in particular of a motor vehicle or a motor vehicle component, whereby a component to be assembled is marked or identified by means of the virtual image component - represented according to the method described above, and that the component to be assembled is in the technical component, in the motor vehicle or in the Motor vehicle component is mounted.

The above-mentioned object is also achieved - in particular in connection with the above-mentioned features - by a method for producing a technical component, in particular a motor vehicle or a motor vehicle component, wherein the assembly location of a component to be assembled is marked by means of the virtual image component - represented according to the method described above or is marked, and that the component to be assembled is assembled in the technical component, in the motor vehicle or in the motor vehicle component at the assembly site.

The above-mentioned object is also achieved - in particular in connection with the above-mentioned features - by a method for producing a technical component, in particular a motor vehicle or a motor vehicle component, wherein an assembly process for assembling a component to be assembled by means of the virtual Image component is determined or verified, and that the component to be assembled is assembled in the technical component, in the motor vehicle or in the motor vehicle component according to the assembly process.

Motor vehicle within the meaning of the invention is in particular a land vehicle that can be used individually in road traffic. Motor vehicles within the meaning of the invention are in particular not limited to land vehicles with internal combustion engines.

• 1 ein Ausführungsbeispiel für ein Augmented-Reality-System,
• 2 ein Ausführungsbeispiel für ein virtuelles Bild, das mittels eines als transparentes Display ausgestalteten Displays dargestellt wird,
• 3 ein Ausführungsbeispiel für ein Messsystem zur Bestimmung der Latenz eines Augmented-Reality-Systems gemäß 1,
• 4 ein Ausführungsbeispiel für einen Hintergrund bei Verwendung eines Messsystems gemäß 3,
• 5 ein Ausführungsbeispiel für ein Testbild vor einem Hintergrund gemäß 4 und
• 6 ein Ausführungsbeispiel für einen Verfahrensablauf zur Herstellung bzw. Reparatur bzw. Instandhaltung eines Kraftfahrzeugs.

Further advantages and details result from the following description of exemplary embodiments. show:

• 1 an embodiment of an augmented reality system,
• 2 an exemplary embodiment of a virtual image that is represented by means of a display designed as a transparent display,
• 3 according to an embodiment of a measurement system for determining the latency of an augmented reality system 1 ,
• 4 an embodiment for a background when using a measurement system according to FIG 3 ,
• 5 an exemplary embodiment for a test image in front of a background according to FIG 4 and
• 6 an exemplary embodiment of a process sequence for the manufacture or repair or maintenance of a motor vehicle.

1 shows a system for displaying a virtual image component (augmented reality image) VIRT—stored in a memory 22 . The system includes a headset 10 worn by a user 1 with a transparent display 11 (see-through display) for displaying the virtual image component VIRT. The virtual image component VIRT is positioned at a specific point by means of a display computer 20 depending on the information from a tracking system, so that it can be displayed at this point by means of the transparent display 11 . The overlay between reality and virtual image component takes place in the eye 100 of the user 1. 2 FIG. 12 shows an exemplary embodiment of a virtual image VIRT that is displayed using the transparent display 11. FIG. In the exemplary embodiment, the real object to which the virtual image component VIRT is assigned is a motor vehicle 25.

des transparenten Displays 11 in Bezug auf das reale Objekt zu ermitteln. In Abhängigkeit dieser Position $$\left[\begin{array}{c}x\left(t\right)\\ y\left(t\right)\\ z\left(t\right)\end{array}\right]$$

des transparenten Displays 11 erfolgt das Rendering mittels des Anzeigerechners 20 und die entsprechende Positionierung des virtuellen Bildbestandteils VIRT an der Position PVIRT' des transparenten Displays 11, die der Position PVIRT des virtuellen Bildbestandteils VIRT in Bezug auf das reale Objekt entspricht. Es ist jedoch vorgesehen, dass zu einem aktuellen Zeitpunkt t nicht die Position des transparenten Displays 11 verwendet wird, sondern ein Schätzwert $$\left[\begin{array}{c}x*\left(t+\tau \right)\\ y*\left(t+\tau \right)\\ z*\left(t+\tau \right)\end{array}\right]$$

für einen in der Zukunft liegenden Zeitpunkt t + τ, wobei τ eine Latenzzeit ist, die zum Beispiel in einem Speicher 21 abgelegt sein kann. Zur Bestimmung des Schätzwerts $$\left[\begin{array}{c}x*\left(t+\tau \right)\\ y*\left(t+\tau \right)\\ z*\left(t+\tau \right)\end{array}\right]$$

kann eine einfache Extrapolation oder ein komplexer Filter, wie zum Beispiel ein Kalman-Filter, vorgesehen sein.The tracking system includes one or more cameras 27, the images are evaluated by a tracking computer 26 to the position $$\left[\begin{array}{c}x\left(t\right)\\ y\left(t\right)\\ \mathrm{e.g}\left(t\right)\end{array}\right]$$

of the transparent display 11 with respect to the real object. Depending on this position $$\left[\begin{array}{c}x\left(t\right)\\ y\left(t\right)\\ \mathrm{e.g}\left(t\right)\end{array}\right]$$

of the transparent display 11, the rendering is carried out by means of the display computer 20 and the corresponding positioning of the virtual image element VIRT at the position PVIRT' of the transparent display 11, which corresponds to the position PVIRT of the virtual image element VIRT with respect to the real object. However, it is provided that at a current point in time t it is not the position of the transparent display 11 that is used, but an estimated value $$\left[\begin{array}{c}x*\left(t+\tau \right)\\ y*\left(t+\tau \right)\\ \mathrm{e.g}*\left(t+\tau \right)\end{array}\right]$$

for a point in time t+τ in the future, where τ is a latency time that can be stored in a memory 21, for example. To determine the appraised value $$\left[\begin{array}{c}x*\left(t+\tau \right)\\ y*\left(t+\tau \right)\\ \mathrm{e.g}*\left(t+\tau \right)\end{array}\right]$$

a simple extrapolation or a complex filter such as a Kalman filter can be provided.

To determine the latency τ, an in 3 The measuring system shown is provided, which includes measuring electronics 30 which are connected in series (data technology) to the display computer 20 during the measurement of the latency time τ. The measuring system also includes an infrared diode 31, which can be switched on in less than 20 μs by means of the measuring electronics 30. In addition, the measuring system includes a photodiode 32 whose output signal can be evaluated by the measuring electronics 30 . In addition, the display 11 or a replacement display 11 ′ is placed between the photodiode 32 and a background 33 . In particular, the background 33 is intended to be a complex background to simulate normal rendering workload. An exemplary background 33 is, for example, a part of a motor vehicle or an in 4 and 5 illustrated cockpit 33 'of a motor vehicle. A polygon 35 is shown in front of this cockpit 33' as background 33 by means of display 11 or by means of substitute display 11'. The polygon 35 is initially black. If the tracking system detects that the infrared diode 31 has been switched on and sends a corresponding tracking packet to the display computer 20, this controls the display 11 or the substitute display 11' in such a way that the polygon 35 changes from black to white, as in 5 shown. This in 4 Polygon 35 shown is an exemplary embodiment for a negative test image within the meaning of the claims and in 5 The polygon 35 shown is an exemplary embodiment of a test image within the meaning of the claims.

• Schritt a: Der Anzeigerechner 20 sendet ein Startsignal S an die Messelektronik 30.
• Schritt b: Die Messelektronik 30 schaltet die Infrarot-Diode 31 an.
• Schritt c: Die Messelektronik nimmt einen Zeitstempel t0.
• Schritt d: Der Anzeigerechner 20 empfängt ein von der Infrarot-Diode 31 ausgelöstes Tracking-Paket.
• Schritt e: Der Anzeigerechner 20 sendet ein Signal T an die Messelektronik 30.
• Schritt f: Die Messelektronik nimmt einen Zeitstempel t1.
• Schritt g: Der Anzeigerechner 20 weist dem Polygon 35 die Farbe Weiß zu.
• Schritt h: Der Anzeigerechner 20 startet das Rendering und sendet zu Beginn des Rendering einen Funktionsaufruf T an die Messelektronik 30.
• Schritt i: Die Messelektronik 30 nimmt einen Zeitstempel t2.
• Schritt j: Der Anzeigerechner 20 beendet das Rendering und sendet zum Ende des Rendering einen Funktionsaufruf T an die Messelektronik 30.
• Schritt k: Die Messelektronik nimmt einen Zeitstempel t1.
• Schritt I: Das weiße Polygon 35 wird mittels des Displays 11 oder des Ersatzdisplays 11' dargestellt.
• Schritt m: Die Fotodiode 32 löst aus.
• Schritt n: Bei Erkennen des Auslösens der Fotodiode 32 nimmt die Messelektronik 30 den Zeitstempel t4.
• Schritt o: Die Messelektronik 30 sendet ein Signal D an den Anzeigerechner 20, das dem Anzeigerechner 20 mitteilt, dass die Fotodiode 32 ausgelöst hat.
• Schritt p: Der Anzeigerechner 20 sendet ein Signal E an die Messelektronik 30.
• Schritt q: Die Messelektronik 30 schaltet die Infrarot-Diode 31 aus.
• Schritt r: Der Anzeigerechner sendet ein Signal R an die Messelektronik, sobald das Trackingsystem das Ausschalten der Infrarot-Diode 31 erkennt.

In an exemplary method by which using the in 3 If the latency τ or other latency times of the system are measured or determined in the measuring system shown, the following sequential method steps are provided as examples:

• Step a: The display computer 20 sends a start signal S to the measuring electronics 30.
• Step b: The measurement electronics 30 turn on the infrared diode 31 .
• Step c: The measurement electronics take a time stamp t0.
• Step d: The display computer 20 receives a tracking packet triggered by the infrared diode 31.
• Step e: The display computer 20 sends a signal T to the measuring electronics 30.
• Step f: The measurement electronics takes a time stamp t1.
• Step g: The display computer 20 assigns the polygon 35 the color white.
• Step h: The display computer 20 starts the rendering and sends a function call T to the measuring electronics 30 at the beginning of the rendering.
• Step i: The measurement electronics 30 takes a time stamp t2.
• Step j: The display computer 20 ends the rendering and sends a function call T to the measuring electronics 30 at the end of the rendering.
• Step k: The measurement electronics takes a time stamp t1.
• Step I: The white polygon 35 is displayed using the display 11 or the substitute display 11'.
• Step m: The photodiode 32 triggers.
• Step n: When the triggering of the photodiode 32 is detected, the measurement electronics 30 takes the time stamp t4.
• Step o: The measuring electronics 30 sends a signal D to the display computer 20, which informs the display computer 20 that the photodiode 32 has triggered.
• Step p: The display computer 20 sends a signal E to the measuring electronics 30.
• Step q: The measuring electronics 30 switches the infrared diode 31 off.
• Step r: The display computer sends a signal R to the measuring electronics as soon as the tracking system detects that the infrared diode 31 has switched off.

The measurement electronics 30 also sends the time stamps t0, t1, t2, t3, t4 to the display computer 20. The latency τ is determined as the difference t4−t0. The latency of the tracking system is determined as the difference t1 - t0. The internal latency of the software of the display computer 20 when processing the tracking information (tracking packet processing) is determined as the difference t2-t1. The latency of the software of the display computer 20 for the rendering is determined as the difference t3-t2. As the difference t4 - t3, the output latency is determined, that is, by the output of the display computer 20 and the display latency of the display 11 or the replacement display 11'.

6 shows an embodiment of an exemplary process sequence for the manufacture or repair or overhaul of a motor vehicle. First, in a step 81, the latency time τ (or further latency times) is measured and the measuring system is then removed. Step 81 is followed by a step 82 in which the virtual image component VIRT is displayed by means of the transparent display 11 in such a way that it overlays a motor vehicle or part of a motor vehicle at the desired position. On the basis of the information displayed in this way, in a step 83 the motor vehicle is repaired or assembled or checked.

Claims (9)

Method for operating an augmented reality system or a virtual reality system with a display (11) for displaying a virtual image component (VIRT) and with a tracking system for determining the alignment and/or the position of the display in relation to a real object, the Augmented reality system has a display computer (20) for such control of the display (11) that the real object (25) is superimposed by the virtual image component (VIRT) or supplemented virtually, characterized in that the augmented reality system a measuring system is assigned, which comprises a test marker (31) and a light sensor (32), the state of the test marker (31) changing, with the display (11) or a substitute display (11') showing a Light sensor (32) identifiable test image is generated when the tracking system detects the state change, and wherein at least a first latency (τ) from the generation of the state change of Test marker (31) is measured until the test image is recognized by the light sensor (32). procedure after claim 1 , characterized in that the test marker (31) is a light source, in particular an infrared light source. procedure after claim 1 or 2 characterized in that the display (11) or substitute display (11') for measuring the latency (τ) is placed between the light sensor (32) and a background (33). procedure after claim 1 , 2 or 3 , characterized in that (by means of the display computer) on the display (11) or a replacement display (11 ') a means of the light sensor (32) identifiable negative test image is generated before the tracking system detects the state change. Method according to one of the preceding claims, characterized in that the measuring system is separated from the augmented reality system. Method according to one of the preceding claims, characterized in that the position of the display (11) (to the real object) at a current time is replaced by an estimated position of the display (11) (to the real object) at a time that corresponds to the current time plus the first latency (τ). Method for producing a technical component, in particular a motor vehicle or a motor vehicle component, characterized in that a component to be assembled by means of the virtual image component (VIRT) according to a method claim 6 is marked or identified, and that the component to be assembled is assembled in the technical component, in the motor vehicle or in the motor vehicle component. A method for producing a technical component, in particular a motor vehicle or a motor vehicle component, in particular a method according to claim 7 , characterized in that the mounting location of a component to be mounted by means of the virtual image component (VIRT) according to a method claim 6 is marked or identified, and that the component to be assembled is assembled in the technical component, in the motor vehicle or in the motor vehicle component at the assembly site. A method for producing a technical component, in particular a motor vehicle or a motor vehicle component, in particular a method according to claim 7 or 8th , characterized in that an assembly process for assembling a component to be assembled by means of the virtual image component (VIRT) according to a method claim 6 is determined or verified, and that the component to be assembled is assembled in the technical component, in the motor vehicle or in the motor vehicle component in accordance with the assembly process.

Images