DE102009040848B4 - Procedure for operating an augmented reality system - Google Patents

Abstract

Method for representing a virtual image component (ARB, VIRT) on a transparent display (11), with an optically time-variant marker (20) having a display for the variable representation of information being placed in a real environment to be recorded, information displayed by means of the display being changed when an element of the recorded environment assigned to the marker (20) is moved, the optically time-variant marker (20) comprising a static component (20.S) and a dynamic component (20.D), the static component comprising (20.S) of the optically time-varying marker (20) corresponds to the outline and the position of the optically time-varying marker (20) in the real environment, the dynamic part (20.D) being implemented by the display of the marker (11), a real image (RB) being recorded by means of a camera, which depicts an environment of the display (11) encompassing the optically time-varying marker (20), which an observer sees when looking through the display, with a tracking system ( 12) from the static component (20.S) the position of a virtual image component (VIRT) to the real image (RB) recorded by the camera and from the dynamic component (20.D) information relating to the movement in the recorded real image (RB) are determined, with the tracking system (12) transferring this as a corresponding synchronization signal (MOV) to a scene generator, and the virtual image component (VIRT) being displayed using the transparent display (11).

Description

The invention relates to a method for generating an augmented reality image, a method or for displaying a virtual image component on a transparent display.

The EP 0 949 513 A2 discloses an augmented reality system having a tracker system for determining the relative position between a sensor for determining a pattern of reference points on an object surface and the object surface, the augmented reality system including a processor connected to the sensor.

The WO 01/96829 A1 discloses a method for automobile production, wherein a tester is supported in carrying out a test with regard to quality defects in a motor vehicle or a motor vehicle component by an augmented reality system (augmented reality = AR), which records and/or data relevant to the test issues the tester, with the test being carried out as part of the production of the motor vehicle or the motor vehicle component to determine quality defects in the motor vehicle or the motor vehicle component in at least one predetermined manufacturing state, with a worker carrying out individual manufacturing steps preceding the test for Production of further motor vehicles or motor vehicle components via the augmented reality system, a cautionary notice based on at least one of the identified quality defects is supplied, and a post-processor when carrying out post-processing of a defective motor vehicle or a defective motor vehicle component downstream of the check error information based on the quality defects identified in each case is supplied via the augmented reality system.

According to the WO 01/96829 A1 the term augmented reality is defined as the superimposition of a real environment with a computer-generated environment, ie a virtual world. Augmented reality is thus a type of human-machine interaction that displays information in the user's field of vision and thus expands his perception.

The DE 10 2005 045 973 A1 discloses a device with an image capture unit for capturing a real environment and generating images of a surrounding area captured within a time interval and with processing means for determining movement information by evaluating the images, the movement information indicating a change in the position of an object within the images that has occurred within the time interval and wherein transmitting means are provided for transmitting the movement information to a data processing device and receiving means for receiving pose information generated on the basis of the movement information, with the pose information identifying a position and an orientation of the object.

The EP 1 507 235 A1 discloses a method for generating a combined image or video that includes virtual objects. It is provided that a marker is equipped with position detection means, in particular a GPS.

The article by Kishino, Yasue [et al.]: Realizing a visual marker using LEDs for wearable computing environment. In: 23rd International Conference on Distributed Computing Systems Workshops, 2003. Proceedings. IEEE, 2003. P. 314-319 assumes that in order to make the real world usable via computer using camera images in augmented reality systems, the concise coordinate information of the systems must be determined. Image-based approaches are promising for this, as they enable direct mapping of the camera images to the coordinate system for virtual objects and therefore no camera calibration is necessary. However, traditional image-based markings are typically stickers and the information they provide is limited and prone to dynamic changes. In contrast, this article discloses a location tagging method called VCC (Visual Computer Communication) that uses smarter markers. In this method, a marker consisting of 16 LEDs (Light Emitting Diodes) flashes continuously to provide both coordinate information and attached information such as an address or a URL.

It is the object of the invention to improve an augmented reality system with regard to moving images or to operate it in an improved manner.

The aforementioned object is achieved by a method for generating an augmented reality image according to claim 1. It is provided in particular that an optically time-variant marker is placed in a real environment to be recorded, with a camera being used to record a real image that depicts the area around the display that includes the optically time-variant marker, which an observer sees when looking through the display, wherein the position of a virtual image component relative to the real image recorded by the camera is determined, and wherein the virtual image component is represented by means of the transparent display.

An optically time-variant marker within the meaning of the invention is in particular a marker whose appearance changes (over time) or whose appearance can be changed. It can be provided in particular that the shape, the size, the brightness, the color and/or the fine structure of the marker change. In particular, it can also be provided that binary (on/off) changeable elements of the marker change their binary state.

A marker within the meaning of the invention is placed in particular physically. However, it can also be provided that a marker within the meaning of the invention is placed immaterially, for example by projection.

A virtual image component within the meaning of the invention(s) can include or consist of alphanumeric characters. A virtual image component within the meaning of the invention(s) can include alphanumeric characters and a colored background for these characters. A virtual image component within the meaning of the invention(s) can consist of alphanumeric characters in connection with a colored background for these characters.

The marker has a display for the variable representation of information. In a further advantageous embodiment of the invention, the display is an LCD, LED or OLED display or is designed as an LCD, LED or OLED display. In addition, information displayed by means of the display is changed when an element of the recorded environment that is associated with the marker moves, or when an element of the recorded environment that is associated with the marker moves. Provision can be made for the optically time-variant marker to be connected to the movable or moving element. However, it can also be provided that the optical time-variant marker is arranged statically or independently of the movement of the element.

The aforementioned object is also achieved by a method according to claim 3, with provision being made, inter alia, for a component to be assembled to be displayed using the virtual image component and for the component to be assembled to be assembled in the technical component, in the motor vehicle or in the motor vehicle component .

The aforementioned object is also achieved by a method according to claim 4, whereby it is provided, among other things, that the assembly location of a component to be assembled is marked or identified by means of the virtual image component, and that the component to be assembled is in the technical component, in the motor vehicle or in of the motor vehicle component is assembled at the assembly site.

The aforementioned object is also achieved by a method according to claim 5, whereby it is provided, inter alia, that an assembly process for the assembly of a component to be assembled is determined or verified by means of the virtual image component, and that the component to be assembled is in the technical component, in the motor vehicle or is mounted in the motor vehicle component according to the assembly process.

Motor vehicle within the meaning of the invention is in particular a land vehicle that can be used individually in road traffic. Motor vehicles within the meaning of the invention are in particular not limited to land vehicles with internal combustion engines.

• 1 ein Ausführungsbeispiel eines Systems zur Darstellung eines Augmented-Reality-Bildes,
• 2 ein Ausführungsbeispiel einer aufgenommenen bzw. aufzunehmenden realen Umgebung bzw. das entsprechend aufgenommene reale Bild,
• 3 ein Ausführungsbeispiel für einen Informationsfluss innerhalb eines Augmented-Reality-Systems gemäß 1,
• 4 ein Ausführungsbeispiel für einen Informationsfluss in einem gegenüber dem Augmented-Reality-System gemäß 1 abgewandelten Augmented-Reality-System,
• 5 ein Ausführungsbeispiel für einen Informationsfluss in einem weiteren gegenüber dem Augmented-Reality-System gemäß 1 abgewandelten Augmented-Reality-System,
• 6 ein Ausführungsbeispiel für einen Informationsfluss in einem weiteren gegenüber dem Augmented-Reality-System gemäß 1 abgewandelten Augmented-Reality-System,
• 7 Darstellungen eines optisch zeitvarianten Markers gemäß 2 zu neun unterschiedlichen Zeitpunkten,
• 8 Darstellungen eines optisch zeitvarianten Markers gemäß 2 zu weiteren neun unterschiedlichen Zeitpunkten,
• 9 Darstellungen eines optisch zeitvarianten Markers gemäß 2 zu weiteren neun unterschiedlichen Zeitpunkten,
• 10 Darstellungen eines optisch zeitvarianten Markers gemäß 2 zu weiteren neun unterschiedlichen Zeitpunkten,
• 11 Darstellungen eines alternativen Ausführungsbeispiels eines optisch zeitvarianten Markers zu neun verschiedenen Zeitpunkten,
• 12 Darstellungen des alternativen Ausführungsbeispiels eines optisch zeitvarianten Markers gemäß 11 zu weiteren neun verschiedenen Zeitpunkten,
• 13 Darstellungen des alternativen Ausführungsbeispiels eines optisch zeitvarianten Markers gemäß 11 zu weiteren neun verschiedenen Zeitpunkten und
• 14 Darstellungen des alternativen Ausführungsbeispiels eines optisch zeitvarianten Markers gemäß 11 zu weiteren neun verschiedenen Zeitpunkten.

Further advantages and details result from the following description of exemplary embodiments. show:

• 1 an embodiment of a system for displaying an augmented reality image,
• 2 an exemplary embodiment of a recorded or to be recorded real environment or the correspondingly recorded real image,
• 3 according to an embodiment of an information flow within an augmented reality system 1 ,
• 4 according to an embodiment of an information flow in a relation to the augmented reality system 1 modified augmented reality system,
• 5 according to an embodiment of an information flow in a further compared to the augmented reality system 1 modified augmented reality system,
• 6 according to an embodiment of an information flow in a further compared to the augmented reality system 1 modified augmented reality system,
• 7 Representations of an optically time-varying marker according to 2 at nine different times,
• 8th Representations of an optically time-varying marker according to 2 at another nine different points in time,
• 9 Representations of an optically time-varying marker according to 2 at another nine different points in time,
• 10 Representations of an optically time-varying marker according to 2 at another nine different points in time,
• 11 Representations of an alternative embodiment of an optically time-varying marker at nine different points in time,
• 12 Representations of the alternative embodiment of an optically time-varying marker according to 11 at another nine different times,
• 13 Representations of the alternative embodiment of an optically time-varying marker according to 11 at a further nine different times and
• 14 Representations of the alternative embodiment of an optically time-varying marker according to 11 at a further nine different times.

1 shows a system for displaying an augmented reality image ARB. The system includes a headset 10 worn by a user 1 with a display 11 for showing the augmented reality image ARB. A camera assigned to the display 11 is installed in the headset 10 to record a real image of the environment seen by the user 1 or a part of this environment. The real image RB output by the camera is an input signal to a tracking system 12, which outputs the real image RB and the alignment POS of the real image RB. The real image RB and the alignment (position signal) POS of the real image RB are input signals to a scene generator 13.

The system for displaying an augmented reality image ARB also includes a database 14 with virtual image components or another source of virtual image components. A virtual image component VIRT is supplied to the scene generator 13 from this database 14 or the other source. The scene generator 13 combines this with the real image RB to form an augmented reality image ARB, which is shown using the display 11 . It can be provided that the display 11 is designed as a transparent display. In this case, the augmented reality image ARB is not an overlay of the real image RB with the virtual image component VIRT, but rather the virtual image component VIRT, which is positioned at a specific point so that it can be displayed at this point using the transparent display. The superimposition between reality and the virtual image component takes place in the eye of the user 1.

2 shows an embodiment of the environment seen by the user or the real image RB. In this case, a workpiece 22 is moved by a robot arm 21 . An optically time-varying marker is placed on the robot arm 21 . Alternatively, it can be provided that instead of the marker 20 a corresponding marker 20 ′ is placed at a non-moving point in the recorded environment and/or a point in the recorded environment that is independent of the movement of the robot arm 21 .

It is provided that the optically time-varying marker 20, 20 'has an LCD display, by means of which a pattern, as exemplified in the 7 until 14 is shown. When the robot arm 21 moves, the pattern shown is changed. For this purpose, the optically time-variant marker includes an input for receiving an in 3 external control signal denoted by reference EXT. This control signal EXT is used to transmit the information to the optically time-variant marker 20 that the robot 21 is moving.

In an embodiment as described in 3 shows the flow of information shown, the optically time-variant marker 20 includes a static portion 20.S and a dynamic portion 20.D. The static portion 20.S of the optically time-varying marker 20 corresponds to the outline and the position of the optically time-varying marker 20 in the real environment. The dynamic part 20.D is implemented by the LCD display. Both the static portion 20.S of the optically time-variant marker 20 and the dynamic portion 20.D of the optically time-variant marker 20 are evaluated by the tracking system 12, the tracking system 12 from the static portion 20.S the position POS for the virtual image component determined. From the dynamic portion 20.D, the tracking system determines information relating to the movement in the recorded real image RB and transfers this to the scene generator 13 as a corresponding synchronization signal MOV.

4 shows an alternative exemplary embodiment with an alternatively designed tracking system 12' for alternative use instead of the tracking system 12. The tracking system 12' comprises a plurality of tracking systems 12.1', 12.2', 12.3', . . . 12.n'. It is provided that the position signal POS is generated by means of the tracking systems 12.1', 12.2', 12.3', . . . , 12.n' and thus the positioning of the virtual image component VIRT. In contrast, the time synchronization is carried out solely by the tracking system 12.1', which is the only one of the tracking systems 12.1', 12.2', 12.3', . . . , 12.n' that evaluates the dynamic component 20.D.

5 shows an alternative embodiment with an alternatively configured tracking system 12" for alternative use instead of the tracking system 12 or the tracking system 12'. The tracking system 12" includes several tracking systems 12.1", 12.2", 12.3", ..., 12.n". It is provided that the tracking system 12.1" generates the control signal EXT for controlling the dynamic component 20.D or the display. The tracking systems 12.2", 12.3", .

6 12 shows an alternative exemplary embodiment with an alternatively configured tracking system 12''' for implementing so-called marker-less tracking. The static component 20.S of the optically time-variant marker 20 is not taken into account. For local synchronization, on the other hand, image components such as an edge 25 of the robot arm 21 and/or an edge of a non-movable portion in the real image denoted by reference number 26 are evaluated.

7 , 8th , 9 and 10 show the marker 20 in the display of nine different patterns. The change in the pattern can symbolize a movement in the real image and can be used for synchronization or for generating the synchronization signal MOV.

11 , 12 , 13 and 14 each show nine alternative configurations of an optically time-varying marker for alternative use instead of the optically time-varying marker 20.

Reference List

1

user

2

marker

10

Headset

11

screen

12, 12', 12'', 12''', 12.1', 12.2', 12.3', 12.n', 12.1'', 12.2'', 12.3'', 12.n''

tracking system

13

Advanced scene generator

14

database

20, 20'

Optically time-varying marker

20th D

dynamic portion

20.p

static part

21

robotic arm

22

workpiece

25, 26

edge

ARB

Augmented reality image

EXT

Signal for controlling an optically time-varying marker

MOV

synchronization signal

POS

Alignment of a real image, position signal

RB

real picture

VIRT

Virtual Image Component

Claims (5)

Method for representing a virtual image component (ARB, VIRT) on a transparent display (11), wherein an optically time-variant marker (20) having a display for the variable representation of information is placed in a real environment to be recorded, changing information displayed by means of the display is when an element of the recorded environment assigned to the marker (20) is moved, the optically time-variant marker (20) comprising a static component (20.S) and a dynamic component (20.D), the static component (20 .S) of the optically time-varying marker (20) corresponds to the outline and the position of the optically time-varying marker (20) in the real environment, the dynamic part (20.D) being implemented by the display of the marker (11), using a camera captures a real image (RB) that depicts the area surrounding the display (11) that encompasses the optically time-variant marker (20), which a viewer sees when looking through the display, with a tracking system (12) being used from the static component (20.S) the position of a virtual image component (VIRT) in relation to the real image (RB) recorded by means of the camera and information relating to the movement in the recorded real image (RB) are determined from the dynamic component (20.D). , wherein the tracking system (12) transfers this as a corresponding synchronization signal (MOV) to a scene generator, and the virtual image component (VIRT) is represented by means of the transparent display (11). procedure after claim 1 , characterized in that the display comprises an LCD, LED or OLED display or is designed as an LCD, LED or OLED display. Method for producing a technical component, in particular a motor vehicle or a motor vehicle component, wherein a virtual image component by means of a method claim 1 or 2 is displayed, characterized in that a component to be assembled is marked or identified by means of the virtual image component (VIRT), and that the component to be assembled is assembled in the technical component, in the motor vehicle or in the motor vehicle component. Method for producing a technical component, in particular a motor vehicle or a motor vehicle component, wherein a virtual image component by means of a method claim 1 or 2 is shown, characterized in that the assembly location of a component to be assembled is marked or identified using the virtual image component (VIRT), and that the component to be assembled is assembled in the technical component, in the motor vehicle or in the motor vehicle component at the assembly site. Method for producing a technical component, in particular a motor vehicle or a motor vehicle component, wherein a virtual image component by means of a method claim 1 or 2 is shown, characterized in that an assembly process for assembling a component to be assembled by means of the virtual image component (VIRT) according to a method according to one of Claims 1 or 2 is determined or verified, and that the component to be assembled is assembled in the technical component, in the motor vehicle or in the motor vehicle component in accordance with the assembly process.

Images