EP2467833A2

EP2467833A2 - Method for representing virtual information in a real environment

Info

Publication number: EP2467833A2
Application number: EP10744693A
Authority: EP
Inventors: Peter Meier; Frank Angermann
Original assignee: Metaio GmbH
Current assignee: Apple Inc
Priority date: 2009-08-18
Filing date: 2010-08-13
Publication date: 2012-06-27
Also published as: US8896629B2; CN102473324A; WO2011020793A2; DE102009037835B4; CN102473324B; CN104731337A; US20150015611A1; WO2011020793A3; US20200242846A1; US20200193712A1; EP3675067A1; US20230290083A1; US11562540B2; DE102009037835A1; CN104731337B; US20120176410A1

Abstract

The invention relates to a method for ergonomically representing virtual information in a real environment, comprising the following steps: providing at least one view of a real environment and a system construction for showing virtual information for overlaying over the real environment in at least one part of the view, wherein the system construction comprises at least one display device, determining a position and orientation of at least one part of the system construction relative to at least one portion of the real environment, subdividing at least one part of the view of the real environment into a plurality of regions comprising a first region and a second region, wherein objects of the real environment are located closer to the system construction within the first region than objects of the real environment within the second region, and showing at least one item of virtual information on the display device in at least one part of the view of the real environment, considering the position and orientation of the at least one part of the system construction, wherein the virtual information is shown differently in the first region than in the second region with respect to the type of overlay in the view of the real environment.

Description

Method for presenting virtual information

in a real environment

The present invention relates to a method for displaying virtual information in a real environment. The method is particularly suitable for the ergonomic representation and marking of points of interest in the world by means of augmented reality technology.

Background of the invention

Augmented Reality (AR) is a technology that overlays virtual data with reality, facilitating the association of data with reality. In the prior art, the use of mobile AR systems is already known. In recent years, powerful mobile devices (e.g., smartphones) have been found to be suitable for AR deployment. They now have comparatively large color displays, built-in cameras, good processors and additional sensors, such as orientation sensors and GPS. In addition, the position of the device can be approximated via radio networks.

In the past, various projects were performed on mobile devices using AR. First, special optical markers were used to determine the position and orientation of the device. Concerning AR, which is also widely used, also called large area AR, hints for the meaningful representation of objects have also been published in connection with HMDs [2,4,5,6,7,8,9]. Recently, there have also been approaches to using the GPS and the orientation sensors of more modern devices ([1, 3, 10, 15]). However, these approaches with video sea-through-AR on small, mobile devices have been no innovative methods for increasing the usability or usability published.

[1] AR Wikltude. http://www.mobilizy.com/wikitude.php.

[2] B. Bell, S. Feiner, and T. H ^'ollerer. View management for Virtual and augmented reality. In Proceedings of the 14th ACM Symposium on

User Interface Software and Technology, pages 101-110, 2001.

[3] Enkin. http://www.enkin.net.

[4] S. Feiner, MacIntyre example, T. H ^'ollerer, and A, Webster. A touring machine:

Prototyping 3d mobile augmented reality Systems for exploring

the urban environment. In Proceedings of the Ist International Symposium

on Wearable Computers, pages 74-81, 1997.

[5] J.L. Gabbard, J.E. Swan II, D. HIx, S.-J. Kim, and G. Fitch. Active

text drawing styles for outdoor augmented reality: A user-based study

and design implications. In Proceedings of the IEEE Virtual Reality

Conference, pages 35-42, 2007.

[6] TH H ^' ollerer. User Interfaces for Mobile Augmented Reality Systems.

PhD thesls, Graduate School of Arts and Sciences of Columbia

University, 2004.

[7] S. Julier, Y. Baillot, D. Brown, and M. Lanzagorta. Information filtering

for mobile augmented reality. IEEE Computer Graphics and

Applications, 22 (5): 12-15, 2002.

[8] G. Reitmayr and D. Schmalstieg. Collaborative augmented reality

for outdoor navigation and information browsing. In proceedings

of the Symposium Location Based Services and TeleCartography-

Geoscientific Communications, pages 31-41, 2004.

[9] J. Rekimoto. The magnifying glass approach to augmented reality systems.

In Proceedings of the International Conference on Artificial Reality

and Tele-Existence and Conference on Virtual Reality Software

and Technology, pages 123-132, 1995.

[10] Sekal Camera, http://www.tonchidot.com/product-lnfo.html.

[11] J. Wither, p DiVerdl, and T. H ^'ollerer. Evaluating display types for ar

selection and annotation. In IEEE International Symposium on Mixed

and Augmented Reality, 2007.

[12] J. Wither and T. H ^' Ollerer. Pictorial depth cues for outdoor augmented

reality. In Proceedings of the 9th IEEE International Symposium on

Wearable Computers, pages 92-99, 2005. [13] From video imageb. (Automatic Fog Detection and Estimation of Visibility Distance through the Onboard Camera)

Magazine Machine Vision and Applications Publisher Springer Berlin / Heidelberg ISSN 0932-8092 (Print) 1432-1769 (Online) Volume 17, Number 1 / April 2006 Pages 8-20

[14] Marko Heinrich, Bruce H. Thomas, Stefan Mueller, "ARWeather," Mixed and Augmented Reality, IEEE / ACM International Symposium on, pp. 187-188, 2008 7th IEEE / ACM International Symposium on Mixed and Augmented Reality, 2008.

[15] layar.com

The Applicant has come to the realization that certain system characteristics should be met for the use of large-area-AR: (1) straightforward, speedy use and direct access to relevant information. (2) The accuracy of mapping virtual information to reality is important. (3) The user interface should be clean and tidy.

Disadvantage of previous methods: (1) Frequently, the so-called "Birdview" (a kind of bird's-eye view) is used as an overview of points in the environment.It is used, for example, to display to the user through visualized virtual information, such as where interesting in the real world With limited screen size, it has a limited view of distant elements, or the resolution becomes too small and the elements are indistinguishable / visible, complicating the surface through simultaneous display two views. (2) If the size of the objects is not scaled correctly, this reduces the distance perception of the users and thus the ability to allocate. (3) If the size of the objects is scaled, they become small and unreadable at long distances. The surface looks unclean and untidy.

Summary of the invention

The aim of the invention is to provide a method for displaying virtual information in a real environment, with which an ergonomic representation of points of interest in the real world can be achieved without restricting the field of view of the user too much and the user with too much information overwhelm.

A first aspect of the invention relates to a method for presenting virtual information in a real environment, comprising the following steps:

Providing at least one view of a real environment and a system layout for overlaying virtual information for superimposition with the real environment in at least part of the view, the system structure comprising at least one display device,

Determining a position and orientation of at least a portion of the system structure relative to at least a portion of the real environment,

Subdividing at least part of the view of the real environment into a plurality of areas comprising a first area and a second area, wherein objects of the real environment within the first area are placed closer to the system construction than objects of the real environment within the second area,

Displaying at least one virtual information on the presentation device in at least part of the view of the real environment, taking into account the position and orientation of the at least one part of the system structure,

wherein the virtual information regarding the type of insertion in the view of the real environment in the first area is displayed differently than in the second area. The at least one part of the system structure may, for example, be a camera whose position and orientation (pose) is determined, the camera not having to be permanently connected to the display device. In certain cases, no camera is needed at all for the entire system, if, for example, the pose of at least part of the system setup is determined only by GPS and orientation sensors. Basically, the determination of the position of each part of the system structure is suitable, provided that conclusions about the viewing direction of the user can be made.

For example, the first area may be a near area, while the second area may be a far area. However, it is also possible that the first area represents a location area, while the second area represents a proximity area. An embodiment with short range, far range and location range will be explained in more detail below with reference to the figures.

According to a further aspect of the invention, which is also independently applicable to the above aspect of the invention of area division of the view, weather data can be taken into account in the display of virtual information in a view of a real environment, for example, via the Internet ("online") queried In order to increase the degree of reality of displayed virtual information in relation to the real environment and thus to improve the assignment, different degrees of complexity are conceivable for processing, eg fixed lighting models or materials can be assigned on the basis of the weather conditions (eg textures). In addition or alternatively, depending on weather data (such as cloud cover, solar radiation, etc.) and / or other data (such as time of day, season, etc.), shadow throws or lighting conditions may be calculated virtual information is an interesting point (also commonly referred to as a "point of interest", POI for short, especially in conjunction with navigation devices) in relation to the real environment. The invention offers the advantage that one obtains an ergonomic representation of virtual information, in particular of interesting points, in the real world, without restricting the field of vision of the user too much and overburdening the user with too much information. At the same time, many different virtual information can be displayed, but this does not lead to an excessive demand of the user due to the ergonomic presentation of the virtual information. In addition, the assignment of information can be strengthened by considering the human perception mechanisms. Further advantageous embodiments of the invention can be found in the dependent claims.

BRIEF DESCRIPTION OF THE DRAWINGS The invention will be explained in more detail below with reference to the figures shown in the drawing.

FIG. 1 is a plan view showing a schematic arrangement of exemplary system structures in relation to a real environment that are usable to perform a method according to the invention; FIG.

2 shows, in a schematic arrangement, an exemplary division of points of interest (surrounding POIs) into different radii ranges,

3 shows, in a schematic arrangement, a possible approach for the calculation of the radii over POIs in sectors, wherein the radii per sector may differ. FIG. 4 shows possible elements of an exemplary view of the real environment with visualized virtual information (here : POI objects) for forming an embodiment of a user interface according to aspects of the invention, 5 shows possible elements of the user interface according to FIG. 4, when the operating focus is in the location area,

6 shows an exemplary preview ("Preview") for a virtual information (here: POI) in the location area of the view according to FIG. 4,

FIG. 7 shows an exemplary possibility of masking out surface elements from the view according to FIG. 4, if these are not needed; FIG. 8-1 1 show different display options for embodiments of virtual information (here: POI objects),

12 is a flowchart showing an overview of the overall flow of an embodiment of a method according to the invention;

13 shows by means of an embodiment a possible procedure for calculating the radius 0 (see FIG. 2) of the location area, as shown by way of example in the view according to FIG. 4. FIG. 14 shows by way of an embodiment a possible procedure for calculating the radius 1, as shown by way of example in FIG. 2,

15 shows by way of an embodiment a possible procedure for calculating the radius 2, as illustrated by way of example in FIG. 2, FIG.

16 shows a schematic view to illustrate exemplary possibilities for the aggregation of virtual information (here: POI objects), FIG. 17 shows an exemplary, possible procedure for assigning shading and material effects (shading) to the virtual information (FIG. here: POI objects), FIG. 18 shows exemplary possible interactions with virtual information (here: POI objects).

Description of embodiments of the invention

Figure 1 shows in a plan view a schematic arrangement of exemplary system structures with respect to a real environment, which are usable to perform a method according to the invention. In particular, Figure 1 shows various possibilities of a system structure.

In the illustration of Fig. IA, the user carries as a display device a head-mounted display system (HMD for short) with a display 21 which is part of the system structure 20. The display 21 can, for example, comprise a generally known semipermeable data glasses ( "Optical See-Through Display"), in the virtual information provided by a computer 23, can be displayed. The user then sees in a view of the real world 40 through the semitransparent data goggles 21 real-world objects enriched with visualized virtual information (such as POI objects related to the real world). In this way, the system assembly 20 forms one embodiment of a well known Augmented Reality (AR) system.

Additional sensors 24, such as rotation sensors and an optical tracking camera 22 may be attached to the display 21. The display 21 may be semipermeable or powered by a camera image with images of reality. If the display 21 is semitransparent, calibration between eye 25 and display 21 is necessary. For this purpose, various methods are documented in the prior art and known to the person skilled in the art. Advantageously, on the display 21 or anywhere on the user's body or in the arithmetic unit 23, position sensors can also be installed, such as GPS sensors (GPS: Global Positioning System), for determining the geographical location of the system structure 20 (eg, according to lengths and latitude) in the real world 40. 1B, another exemplary system structure 30 is shown which can often be found eg in modern mobile phones (so-called "smartphones") .The display device 31 (eg in the form of a screen or display), computer 33, Sensors 34 and camera 32 form a system unit which is accommodated approximately in a common housing of a mobile telephone The view of the real environment 40 is provided by the display 31, which represents a camera image of the real environment 40, which is from the camera 32 For augmented reality applications, the camera image may be displayed on display 31 and supplemented with additional virtual information (such as POI objects related to the real world.) In this manner, the system assembly 30 forms another embodiment of the present invention well-known Augmented Reality (AR) system.

Basically, this invention can be used sensibly for all forms of AR. For example, it does not matter if the visualization is performed in the so-called optical-see-through mode with semi-transmissive HMD or in the video sea-through method with camera and screen. If the camera is referred to below, this may be a camera or the optical system which results from the combination of eye and display screen (see display 21 of FIG. Both have camera properties relevant to the insertion of virtual information, such as the opening angle and the main image point.

In principle, the invention can also be used in connection with stereoscopic displays, whereby advantageously two cameras each record a video stream for an eye in the video see-through approach. In any case, the virtual 3D information can be calculated individually for each eye. The processing of the different substeps described below can basically be distributed to different computers via network. So it is a client / server architecture or purely client-based solution possible. For example, the client could send an image to a server, which based on the image statements on the 3D position and 3D orientation of the system structure (see. Fig. 1) or a part thereof in relation to the real world (hereinafter called pose) and provides the client with visibility. Furthermore, the client or server may also include multiple computing devices, such as multiple CPUs or specialized hardware components, such as well-known FPGAs, ASICs, GPUs, or DSPs. Multiple clients can also exchange information with each other, which is generated, for example, in terms of visibility at this location or if a client creates a POI. This information exchange can take place via a server, but it would also be possible direct connections via Bluetooth or WLAN.

In order to realize AR, the pose (position and orientation) of the camera in the room is needed. This can be realized in many different ways. For example, you can only detect the pose in the world with GPS and an orientation sensor with electronic compass (as used in some modern mobile phones, for example). However, the uncertainty of the pose is then very high. Therefore, other methods such as optical initialization and tracking or the combination of optical methods with GPS and orientation sensors may be used. WLAN localization can also be used or RFIDs or optical markers can help localization. Again, as already mentioned, a client-server based approach is possible. In particular, the client may request location-specific information that it requires for optical tracking from the server. These may be, for example, reference images of the environment with pose information and depth information. The invention can improve the information presentation for the client. It can also be used in a remote scenario (remote view scenario). Here, for example, a maintenance expert in a control room sees the image of the client transmitted via the data network and the correspondingly prepared information on his screen. He could then give the client instructions or just watch. In a similar scenario, it is conceivable for a person to view recorded image or video material with interactive additional information presented according to the invention and, if possible, navigate through the material similarly to the Internet-based application "Google Streetview". Furthermore, the invention can also be installed or carried in vehicles, aircraft or ships as a monitor, HMD or by means of a head-up display. Basically, an interesting Point of Interest (POI) can be created for a wide variety of information types, examples are given below, images of places with GPS information can be displayed, and information from the Internet can be extracted automatically Companies or restaurants can be websites with addresses or pages on which evaluations can be made, users can deposit texts, pictures or 3D objects in places and make them accessible to others, information pages such as Wikipedia can be searched for geoinformation and The sites can be made accessible as POIs, POIs can be generated automatically from the search or browse behavior of mobile device users, and other interesting places can be displayed such as subways or bus stations, hospitals, police stations, doctors, real estate ads or fitness clubs.

In the following aspects and embodiments of the invention with reference to the flow charts, which are shown from Figure 12, in conjunction with the other figures 1 to 11 explained in more detail.

2 shows in a schematic arrangement an exemplary distribution of points of interest (surrounding POIs) into different radii ranges. FIG. 3 shows in a schematic arrangement a possible approach for calculating the radii over POIs in sectors, the radii differing per sector 4 shows possible elements of an exemplary view of the real environment with displayed virtual information (here: POI objects) to form an embodiment of a user interface according to aspects of the invention.

As shown in more detail in FIG. 4, at least part of the view 1 is related to the real environment (which is the view through a semitransparent HMD 21 or on-screen HMD 21 of FIG. 1A, or a screen shot of a camera image on the display 31 of FIG Fig. IB) into several areas. Splits. In this embodiment, these comprise a first region 3, which in this embodiment represents a proximity region 3, a second region 4, which in this embodiment represents a far region 4, and a third region 5, which in this embodiment represents a location region 5. In this case, objects of the real environment 40 within the proximity area 3 are placed closer to the system structure 20 or 30 than objects of the real environment within the far area 4. In the location area 5, in turn, objects of the real environment 40 are closer to the system structure 20 or 30 Placed as objects of the real environment within the proximity 3.

At least one virtual information is displayed in the view 1 according to FIG. 4 (in the present exemplary embodiment, a plurality of virtual information POI1-POI4, POIl 1-POI14 and POI21-POI23 is displayed). In this embodiment, these represent points of interest (POIs) in relation to reality, which are visible in view 1. For example, a virtual information POIl is displayed for a visible in the view 1 building (not shown in FIG. 4), which points to this building or is optically associated with this and possibly allows to retrieve more information about the building. Here, the information POIl is superimposed on the display device 21 or 31 in at least one part of the view 1 of the real environment, taking into account the position and orientation of the corresponding system structure 20 or 30 or a part thereof. As will be explained in more detail below, the visual information displayed with respect to the type of insertion in the view 1 is displayed differently in the proximity area 3 than in the distance area 4. In other words, the same virtual information (say POI1) in the proximity area 3 , eg if it is associated with a real object in the near zone 3, shown differently than if it were faded in the far-end region 4, e.g. if it is associated with a real object in the far-end area 4 (a POI is simultaneously present in only one area here).

As illustrated in FIGS. 2 and 3, the proximity region 3 is separated from the far region 4 and the location region 5 by a respective boundary, each having a radius (radius 0 to radius 3) in the direction of a viewing angle of the display device in the present embodiment (Fig. 2 shows beyond that, further limitations or radii, as explained in more detail below). The definition and calculation of the radii according to Figures 2 and 3 will be described in more detail below. The boundaries have no fixed shape, but are preferably rotationally symmetric with respect to the camera or the viewer. Also, the boundaries need not be circular, but may, for example, also have elliptical or other shapes.

OVERVIEW

FIG. 12 shows in a flow chart an overview of the sequence of an embodiment of a method according to the invention in conjunction with FIGS. 1-11. In a first step 1.0, the system is initialized or, if a run has already taken place, old values can optionally be kept. In a next step, data is loaded, if not existing or if the situation has changed. The data source may be located on one or more servers or stored locally on the device or encoded as information in the environment (e.g., via RFID or QR codes). If a very large number of POIs are stored in the data source, the number can be limited by the position and the definition of a radius 3. For example, modern database systems with spatial features or properties ("spatial features") offer such functionality, and in principle, POIs can be filtered out with filters already defined in this step by the user, for example, only catering or no catering. POIs which carry corresponding information are then sorted out, and it is clear to the person skilled in the art that a corresponding data structure can be stored for a POI, for example its position, its 2D or 3D shape, an image and / or or further meta-information, such as the category, for example, the POIs are then organized according to the invention in steps 3.0 - 6.0, after which they can be visually processed and displayed, after which interaction with the illustrated POIs is possible explained in more detail by way of example.

DISTRIBUTION IN AREAS In the following, steps 3.0 - 6.0 of FIG. 12 will be described in more detail in conjunction with the following FIGS. 13-18 and FIGS. 1 to 11. Here, FIGS. 13-18 relate to possible sub-steps of the general method steps shown in FIG. In principle, these steps can be performed per run or in the background, at certain intervals, depending on the performance of the system. A fundamental aspect of the invention is that superimposed virtual information (here: POIs) in tightly configured or automatically calculated, different areas of a view are classified in the real world.

Figure 2 shows a view of a possible scenario from a bird's eye view. Different POIs are spatially arranged around the camera. If all these POIs were simply displayed, the screen would quickly become crowded with many objects. Therefore, a division into different areas. The location area 5 is restricted by radius 0. Regarding the radii, it should be said that they do not have to be constant over all angles, but can change depending on the angle (see Fig. 3). The location area 5 shows objects that are in the immediate vicinity, so that it is difficult for the user to retrieve them with the device. The findability is particularly dependent on the accuracy of the detectable pose. Correspondingly, the radius 0 can advantageously be set larger or smaller as a function of the (uncertainty of) the pose determination (FIG. 13).

4 shows possible elements of an exemplary view of the real environment with inserted virtual information (here: POI objects) for forming an embodiment of a user interface according to aspects of the invention. FIG. 4 shows an advantageous distribution of the user interface. In this case, POIs in the location area 5 are fixedly arranged in the lower area of the view 1 and can be selected, for example, by mouse click or touch by means of a touchscreen. FIG. 6 shows a possible indication as a result of the activation of an exemplary POI23. If present, an image P23 of the exact location of POI23 may be displayed for guidance. If available, an additional preview V23 ("Preview") of the stored information can be displayed POI objects that are outside of the location area 5 but are within sight are placed in close proximity 3. In this area, the correct assignment of virtual information to reality is found to be particularly important. Thus, it can be relevant for the personal planning of the path whether a POI object lies in front of or behind a street intersection. The near zone 3 is bounded by the radii 0 and 1 (FIG. 2).

FIG. 14 shows a possibility for calculating the radius 1. On the one hand, the meaningful visibility of POI objects plays a role. If the representation of a POI were only 2 mm in size (sample value), it would bring little benefit, but only cause disorder in the display. For the calculation in step 4.1, therefore, the resolution of the display, the size of the POI (the representation of, for example, a dinosaur allows a larger radius 1 than the representation of a large ball), and the aperture angle of the camera. In the further steps, the radius 1 can be further adjusted if too many POIs fill the screen (referred to as a global approach). The calculation can refer to the total aperture angle of the camera or even a larger area. It is also possible to set smaller sectors individually. The sectors may e.g. be fixed or formed on the basis of clusters, where clusters help to divide sectors (see step 6.2 in Fig. 16). An example method could be performed as follows: The center of gravity is calculated for each cluster. This will draw a straight line. Now straight lines, which have a small angle to each other, summarized. Each resulting line is aligned to integer angles and assigned a sector. The size of the sectors is then rounded and iteratively magnified in integer angular increments until they come in contact with a neighboring sector. If the sectors are determined by clusters or by fixed settings, the number of POIs of a sector is counted in step 4.2. If this is above a certain configurable limit value, the radius is reduced until it falls below the limit value (step 4.3).

Alternatively, individual POIs can also be moved into the far-end area 4 without adjusting the radius (referred to as a local approach) if POIs in the Overlap display. For this purpose, the 2D position and extent of the POIs, corresponding to the rendering pipeline, are calculated in step 4.4. As a rendering pipeline, a person skilled in the art understands the projection of objects in 3D space onto a smaller-dimensional display. In the near range, this happens, for example, depending on the position, orientation, opening angle of the camera and resolution of the 2D display area that is available to the near range. For the far-end, the rendering pipeline projects to a line. This can be done, for example, by means of a standard 3D-2D projection in which the objects for the renderer are communicated very far away in order to determine the position of the objects on the possibly rotated display in their vanishing point. If they overlap, the rear POI for the far range 4 is marked (step 4.5). This designation may be preserved for a rendering run (step 1.0-9.0 of FIG. 12) or may be retained longer (for example, 3 seconds or 5 rendering runs). The system can continue with steps 4.4 and 4.5 until no overlaps occur or fall below a certain value. There can also be a tolerance as to when an overlap is considered as such (eg more than 10% of the sum of the area of both objects as intersection). Advantageously, the global can also be combined with the local approach.

Although in the drawings, for illustrative purposes, the sectors and POIs are shown from a bird's eye view, all of these methods are also optionally applicable to the 3D case. The far-end area 4 contains POIs which have been displaced from the near area or which lie outside the radius 1 in the sector. But even here it does not necessarily make sense to represent all existing POI objects. Therefore, advantageously, a radius 2, similar to radius 1, can be calculated, as shown in FIG. Since radius 2 should in particular still display POI objects which lie within the range of the user, advantageously the current speed or the average speed of the user or the distance which the user can travel by public transport, vehicle or bicycle or the like Time can be used to calculate radius 2. The ability to calculate which depends on these factors Objects can be represented by AR, can also be used independently of the other claims of the invention.

In an exemplary embodiment of the invention, one can also mark individual POIs from one area to another (e.g., via a touch screen), such as for near-field display. The POI object is then permanently stored, e.g. displayed in close range 3.

It is also conceivable for the manual adjustment of the radii, superimposed on the user superimposed on the reality of the user. The user can then manually change the radii by means of a touchscreen,

PRESENTATION OF THE OBJECTS

FIG. 4 shows a possible exemplary representation for POIs in different areas. In this case, POIs in the location area 5 below are displayed in a uniform size immobile and initially independent of the viewing direction. POI objects in the vicinity 3 are superimposed according to the pose, the camera parameters and the model properties three-dimensionally and in perspective correctly the environment 40. POI objects in the far-end area 4 are advantageously displayed in a uniform size and advantageously move in accordance with the orientation of the device (in particular the presentation device) so that a vertical line downwards would hit the corresponding assigned real location. Advantageously, symbols can still be displayed on the right and left edge 6, which point to POI objects outside the opening angle of the camera. An arrow 8 can give an indication of direction, so that if you move the device along the direction of the arrow around the viewer, the POI object would enter the field of view. These edges 6 may exist for each area (FIG. 4) or, for example, only for the near area 3 (FIG. 5). Short range 3, far range 4, and / or location range 5 may optionally be hidden, especially if no virtual information is presented therein. This can also be done automatically if there is no POI in these areas (Figure 7). Basically, a point of interest (POI) may be represented by a symbol, an image, an SD object, or the like. For the presentation is the use of a 3D rendering technology (like the well-known OpenGL or DirectX) with 3D objects, the rendering of so-called billboards (2D objects, which are always facing the viewer) or so-called 2D overlays with 2D Rendering techniques possible whose projection is calculated independently. The presentation can be based on the category of the POIs (eg a globe for the representation of a website) or determined by the user (eg the placement of a dinosaur with additional information). In particular, the POIs can be colored in a high contrast.

As shown in FIGS. 10 and 11, POIs may be provided with additional information that is differently detailed (hereinafter exemplified by POIl). The lowest level of detail is just the POI. The next step is to display labels (see Label Ll) (Figure 10, left), which, for example, display a descriptive text about the associated POI object. The next stage is the so-called preview (see Vl or picture Pl in Fig. 11) (e.g., an image or a rendering of the website or an info text). Some POI objects can then be examined in more detail in a next step (see Information II). This can also trigger the activation of a separate program of the operating system, such as starting an Internet browser or a media player. Depending on the configuration or on automatically evaluable criteria, such as the resolution of the display, the POIs can be displayed simultaneously with their additional information or only by activation. It can also advantageously be determined which POI object is closest to one another, and only the foremost POI objects first display labels at a certain distance, and previews automatically when approaching.

Alternatively, the display can also be controlled by so-called eye tracking. The POIs that the user views are displayed with additional information. The additional information can advantageously be anchored immovably in the view and be connected by means of a dynamic connection with the mobile POI representation. This increases the readability of the information. The use of eye-tracking to activate additional information about a POI in AR can also be used independently of the method according to claim 1.

In order to improve the assignment of the POI objects to real locations, various procedural steps according to the invention can be combined, as shown in FIG. 17. In principle, the following method steps can also be carried out independently of the previously described idea of subdividing the view of the real environment into several areas (such as near, far and location area) and displaying the displayed virtual information differently depending on the respective area as described above.

In principle, a perspectively correct 3D rendering of the POI object takes place (in particular in the proximity region 3 when a region is divided). Furthermore, the standard POI objects have a fixed size in order to be able to estimate the distance to them continuously. In order to increase the degree of reality and thus improve the allocation, weather data can advantageously be queried online. Here, different levels of complexity are conceivable for processing. Thus, based on the weather conditions (for example, according to the Google weather service "mostly cloudy", "occasionally stormy", "occasional rain", etc.) permanent lighting models or materials are assigned (eg textures), which are adapted to the weather However, in the highest degree of complexity one could also use a current cloud or rain-satellite or radar image to dynamically create an approximate model of the cloud cover and calculate the shadows and optionally the detailed lighting conditions (see also Figure 9) This can be done by a server, which makes the data available to the client in a location-specific way.Also helpful for the perception of the distance is the determination of the visibility through fog, rain or haze, which can be automatic (see "From video image eg (Automatic Fog Detection and Estimation of Visibility Distance through the Onboard Camera) ", Journal Machine Vision and Applications Publisher Springer Berlin / Heidelberg ISSN 0932-8092 (Print) 1432-1769 (Online) Issue Volume 17, Number 1 / April 2006, pages 8-20) or also be queried using current weather data. The simplest technical implementation, among other well-known, is the setting of the fog settings in OpenGL. In particular, for the use of the visibility to the realistic representation of the distance, a part of a virtual object can be changed by fog, while another part is clearly displayed. This is to prevent, for example, when using the technology in a vehicle important POI information disappear completely through fog. Technically, this can be realized, for example, by a second rendering pass, which takes into account only certain materials but does not represent fog. As described in step 7.2 of FIG. 17, the state of the sun or of the moon can additionally be calculated on the basis of the position, the date and the time, and set to set the light sources. This affects in particular the shadows (compare shadows S1-S4 for the POI1-POI4 in Figures 4, 10 and 11), which help humans to better determine the position of the POI. The shadows can be precomputed textures (advantageously with a transparency value) located below the POI on the ground level, depending on the position of the sun or moon, where the line between the sun or moon and POI intersects the ground plane (exceptions, if that not the case). If the sun or moon are visible at the same time, the sun is used for the calculation.

However, the shadows may also be dynamically calculated as known in the art. This may advantageously involve the mutual shading of POIs. If a 3D model of the environment exists (see also step 7.4 in FIG. 17, eg in the form of a model for masking a real object, so-called "occlusion geometry"), this can be used in addition to the realistic calculation of the shadow situation, for example by shadows POIs raises (see also Figure 9) In step 7.3, the reality level of the overlay can be further increased by the materials are enriched by images of the environment of the POI.The use of so-called environment maps (environment maps) is known in the art that these are dynamically extracted locally, for example, from Google Streetview and taken into account. In step 7.4, another step is taken to strengthen the depth perception of the viewer. By loading occlusion models (so-called "occlusion geometry"), it can be determined whether one POI is visible to the viewer, or, for example, disappears behind another building POIs, when obscured, can be pushed directly into the far area 4 or more For example, the hidden part may be displayed semi-transparently, with dashed lines or other colors, and advantageously also the shadows are not calculated and displayed.The depth model may be deposited or generated dynamically by SLAM algorithms, stereo cameras or a Time Of Flight camera In this case, depth information per pixel is sufficient In step 7.5, the camera parameters are generated for a correct overlay in the near range (this need not be constant), which can be generated dynamically, for example by means of a SLAM mechanism or according to the device name of server be retrieved or stored in the program. In the case of a see-through HMD or HUD, the results of the sea-through calibration or a dynamic measurement of the position of the eye are used for the display.

In step 7.6, advantageously, the camera image can be processed so that only the most important image components are displayed in high contrast. This only makes sense in video sea-through mode and should help users make the mapping, especially in very bright ambient light. The video image can be revised, for example, by means of the Sobel operator for edge extraction. Advantageously, this mode can be switched on and off in bright outdoor light, for example when the device contains a brightness sensor. In step 7.7, the parameters of the pose are now additionally made available to the rendering system if this was not already necessary for the calculations in steps 3 to 6 of FIG. Now, depending on the hardware capabilities of the system in step 7.8, everything can be represented and calculated. If the hardware of the system is weak, the calculation of the correct material surfaces can also be done on the server side (Step 7.8B) or the overall picture on the server side. With a strong system, a modern GPU (Graphic Processor Unit) can do much of the work (step 7.8A). For this purpose, many possibilities are known to the expert.

AGGREGATION OF POIs

At a point in time in the sequence (FIG. 12), advantageously before step 4.0 or after step 5.0, POIs can also be combined, as shown in FIG. 16, bottom left. The points of a cluster form a new, differently shaped POI. This can be done individually in the close range 3, in the location area 5 and in the distance range 4 or can already be done before the allocation. Advantageously, this can only be done with POIs of a category (e.g., Web sites only). Advantageously, POIs in the forefront can be excluded from this process. INTERACTION WITH POIs

FIG. 18 shows possible interaction possibilities with POIs according to aspects of the method according to the invention. As usual in augmented reality, the user can already change the section to the virtual information and the reality by changing his location or the viewing direction. This can also trigger automatic interaction, as already described in "Appearance of Objects." If the device is equipped with a touch screen or a kind of mouse control (eg a trackball), it is basically possible to select POIs directly POI object, the individual POIs are now arranged to be selectable and the labels are displayed (step 9.8) Otherwise, if not yet displayed, the label is displayed (step 9.9) By reactivating the POI (step 9.1 or keypress) the info text or preview is displayed if it is not already displayed (step 9.11) Re-activation triggers the detail display, step 9.13 for POIs that have stored the information (eg playing a movie, music or viewing a website). By touching the close button or advantageously by shaking the device and registering by motion sensors n this display can be closed. Alternatively, you could possibly run this in the background. Optionally, the user can also The next stored audio or video file will be executed nearby and will run in the background.

Instead of direct selection by touch, the interaction can also be carried out by means of the target cross 7 shown in FIG. 7 (optional), as described in FIG. 18 in steps 9.5-9.7. If the user points the crosshair 7 (or a similar target object) in the camera to a POI, the next detail level is activated there (label if only POI visible, preview, if label visible). Advantageously, the system can also activate the POI closest to the user interface to the crosshairs. Long aiming at a POI or pressing a button activates the next level of detail. To get into the far-end area 4, the user points the camera upwards until he has exceeded the highest POI in the vicinity 3 by a threshold (e.g., 10 degrees). There he can then navigate by right and left turning between the POIs. Similarly, by pointing the camera down, he can enter the location area 5. Here he can also navigate by turning right and left through the otherwise fixed POIs. A special feature in the location area 5 is that a POI, if specified or automatically created, there in addition to the POI detail information is given an image of the place to facilitate finding (see preview V23 and Figure P23 for the POI23 in Figure 6). The image can be entered manually or automatically created using an image database containing pose information.

The selection of a POI can optionally also be done by voice control. To do this, the user activates control over a particular set (e.g., "activate voice control"), whereupon each POI is identified with a number, and then, by pronouncing the number, that POI can be activated.

Advantageously, the system can be activated by further pointing down a card mode. Advantageously, an acoustic signal is triggered immediately when changing the POI or when triggering an interaction. Furthermore, a haptic signal (for example a slight vibration) can also take place. Advantageously, in a particular interaction with a POI (eg, three clicks within a short time), a map mode may be opened centering the map at the POI location. Alternatively, a navigation to this POI can be started by a special interaction.

Furthermore, the invention also includes the following aspects and embodiments which may be applied in connection with what has been described so far: The virtual information (e.g., POIs) may be superimposed on the real-world view in the remote area 4 in a uniform size. With regard to the type of insertion in the view of the real environment in the location area 5, the virtual information is displayed differently than when it is displayed in the proximity area 3. In particular, the virtual information can be included in the view of the real environment in the location area 5 immovable, especially in a uniform size and / or regardless of the orientation of the display device, are displayed.

It is also possible to combine several different virtual information into one virtual object and to display the virtual object instead of the multiple virtual information in the view. In this case, the plurality of virtual information can be selected from a group of virtual information that together form a cluster, wherein the virtual object is shaped differently compared to the virtual information.

Furthermore, a geographical map and / or a bird's-eye view can be superimposed on the display device if the display device is held approximately horizontal to the earth's surface or if more than one specific angle is kept below the lowest POI in the near zone or if this function is already used to achieve the Location area is occupied, is held further below.

A respective boundary, in particular its radius, between the areas can be changed if virtual information from the user of one of the areas rich, for example, the near area, in another of the areas, such as the long-range, and / or vice versa is transferred.

The boundary, in particular its radius, can also be calculated as a function of the number of multiple virtual information within a particular sector of the view. The boundary, in particular its radius, can also be calculated as a function of the two-dimensional density of a plurality of virtual information within a particular sector of the view. Furthermore, the limitation, in particular its radius, can also be calculated as a function of a plurality of virtual information items which together form a cluster.

In the near area, a shadow in the vicinity of a ground plane represented in the presentation device, which corresponds to the position of the virtual information, can be displayed in the display device below the virtual information.

As the display device, a video display device may be used in which the view of the real environment is enriched by an edge image or replaced to enhance the contrast.

The user may be given acoustic and / or haptic feedback in an input device used for selection in a user action to select virtual information or switch between multiple virtual information.

By determining the position and orientation of the at least part of the system structure 20 or 30 relative to the real environment, depth information relating to at least one real object included in the view may be calculated or loaded, wherein the depth information may be used Masking model for concealing a real object in the display device when virtual information is obscured by the real object to be hidden in the view. Such depth information can also be used to calculate a boundary between areas, eg between near and far. Furthermore, a number can be assigned to a plurality of virtual information displayed in the display device, wherein the corresponding virtual information can be selected by voice recognition of the number or selection of the number on a keyboard or a touch-sensitive input field.

In addition, multiple virtual information can each be assigned to one of several categories, whereby the virtual information can be shown and / or hidden depending on the category.

Also, in the display device, an edge 6 can be displayed, which indicates a range of the proximity area 3, wherein a limitation of the near area can be changed by a user action, in particular by dragging the boundary.

For example, the virtual information can be displayed in at least three stages. In this case, a first stage has a body (eg a 2D body or SD body)) as merely a local indication of the virtual information (cf. POIl in FIGS. 10, 11), a second stage has an indication of the virtual one Information in the form of a labeled label (see Label L1 in Fig. 11), and a third stage has an abstract-like preview of the virtual information (see Preview V1 and picture P1 in Figures 10, 11), which are particularly shown becomes when a user action selects the dot-like display or the label. A fourth stage may represent the full length information (see Information II of Fig. 10).

One embodiment may provide that in a first part of the long range 4 virtual information is displayed only in the first stage, in a second part of the long range 4 with real objects placed closer to the display device than in the first part of the long range and in a first part of the short range 3 virtual information is displayed in the second stage, and in a second part of the short range 3 with closer than in the first part of the short range at the Representation device placed real objects the virtual information is displayed in the third stage.

Furthermore, it can be provided that a boundary between the near range and the far range, in particular a radius of the limit, depending on a size of the displayed virtual information, a resolution of the display device and / or a resolution of a camera, which is used to generate the view is calculated , Furthermore, the boundary of the location area can become larger with an increased measurement uncertainty of the position detection.

Furthermore, the limitation, which may be, in particular, a radius and determines which objects are to be displayed at all, may be based on the current speed or the average speed of the user or the distance which the user uses by public transport, vehicle or bicycle or the like certain time can depend.

Furthermore, independently of the other disclosures, the system may overlay virtual information with reality, taking into account, in the display of virtual information in a view of a real environment, weather data, e.g. queried over the Internet ("online") to increase the level of realism of visual information displayed in relation to the real environment and thus to improve the allocation The features and embodiments described above in connection with weather data and the like can also be found in Connection with this aspect can be applied independently of other aspects described.

Claims

claims

5 1. A method of presenting virtual information in a real environment, comprising the following steps:

- Providing at least one view (1) of a real environment (40) and a system structure (20, 30) for superimposing virtual information (POI) to overlay the real environment in at least part of the view, wherein the o system structure at least one display device (21, 31),

Determining a position and orientation of at least part of the system structure (20, 30) relative to at least one component of the real environment (40),

- subdivision of at least part of the view (1) of the real environment into a plurality of areas comprising a first area (3) and a second area 5 (4), wherein objects of the real environment (40) within the first area (3) closer to the System structures (20, 30) are placed as objects of the real environment (40) within the second area (4),

Displaying at least one virtual information item (POI) on the display device (21, 31) in at least part of the view (1) of the real environment, taking into account the position and orientation of the at least one part of the system structure (20, 30),

- The virtual information (POI) is displayed differently in terms of the type of insertion in the view (1) of the real environment in the first area (3) than in the second area (4).

2. The method of claim 1, wherein

the at least one virtual information item (POI) is superimposed in the view of the real environment in the first area (3) in perspective correctly corresponding to the position and orientation of the presentation device (21, 31) relative to the real environment, in particular in different size depending on the perspective positioning of the virtual information in the view.

3. The method according to claim 1 or 2, wherein at least one virtual item of information (POI) is displayed in the view of the real environment in the second area (4) in uniform size.

4. The method according to any one of claims 1 to 3,

wherein the plurality of areas into which the at least part of the view of the real environment is subdivided include, in addition to the first area (3) and the second area (4), a third area (5) within which objects are closer to the real environment System structure (20, 30) are placed as objects of the real environment within the first area (3), and wherein the at least one virtual information (POI) regarding the type of insertion into the view (1) of the real environment in the third area ( 5) is shown differently than in the first area (3).

5. The method according to claim 4,

wherein the at least one virtual information item (POI) is superimposed on the view (1) of the real environment in the location area (5) immovably, in particular uniformly and / or independently of the orientation of the presentation device (21, 31).

6. The method according to any one of the preceding claims,

in which several different virtual information (POI) are combined into one virtual object (POI_C) and the virtual object is displayed instead of the multiple virtual information in the view (1).

7. The method according to claim 6,

wherein the plurality of virtual information (POIs) are selected from a group of virtual information that together form a cluster (Cl), wherein the virtual object (POI_C) is shaped differently in comparison to the virtual information (POI).

8. The method according to any one of the preceding claims,

in which a geographical map and / or a bird's-eye view is superimposed on the display device (21, 31) if the display device has more than one specific angle below one of the regions (3, 5) or a virtual information item (POI) shown below is inclined in one of the areas.

9. The method according to any one of the preceding claims,

in which the at least one virtual information item (POI) can be transferred from one of the areas (3, 4, 5) to another of the areas (3, 4, 5) by selecting the virtual information by the user and by a transfer action, in particular by Dragging the virtual information is transferred.

10. The method according to any one of the preceding claims,

wherein the first region (3) is separated from the second region (4) by a boundary (radius 1) which in particular has a radius in the direction of a viewing angle of the display device (21, 31), the boundary, in particular its radius, being dynamic is calculated.

11. The method of claim 10, wherein

the boundary (radius 1), in particular its radius, is changed when virtual information (POI) is transferred by the user from one of the areas (3, 4, 5) to another of the areas (3, 4, 5).

12. The method according to claim 10 or 11, wherein

the boundary (radius 1), in particular its radius, is calculated as a function of the number of multiple virtual information (POIs) within a particular sector of the view.

13. The method according to any one of claims 10 to 12, wherein

the boundary (radius 1), in particular its radius, is calculated as a function of the two-dimensional density of several virtual information (POI) within a specific sector of the view.

14. The method according to any one of claims 10 to 13, wherein

the boundary (radius 1), in particular its radius, is calculated as a function of a plurality of virtual information (POI), which together form a cluster (Cl).

15. The method according to any one of the preceding claims,

wherein in the first area (3) in the display device (21, 31) below the at least one virtual information item (POI), a shadow (S 1- S4) is displayed in the vicinity of a floor level represented in the display device, which is aligned with the position of the virtual Information corresponds.

16. The method according to any one of the preceding claims,

wherein as the display device (21, 31) a video display device is used in which the view (1) of the real environment is enriched by an edge image or replaced to enhance the contrast.

17. The method according to any one of the preceding claims,

wherein in a user action for selecting virtual information (POI) or switching between multiple virtual information (POI), the user is given an acoustic and / or haptic feedback in an input device used for selection.

18. The method according to any one of the preceding claims,

wherein, by determining the position and orientation of the at least one portion of the system structure (20, 30) relative to the at least one constituent of the physical environment (40), depth information is computed or loaded relative to at least one real object included in the view; the depth information is used to display a masking model for masking a real object in the display device (21, 31) when virtual information is obscured by the object to be hidden in the view.

19. The method according to any one of the preceding claims,

wherein, by determining the position and orientation of the at least one portion of the system structure (20, 30) relative to the at least one constituent of the physical environment (40), depth information is computed or loaded relative to at least one real object included in the view; the depth information is used to calculate a boundary (radius 1) between the first area (3) and the second area (4).

20. The method according to any one of the preceding claims,

wherein a plurality of displayed virtual information (POI) is assigned a respective number, and by voice recognition of the number or selection of the number on a keyboard or a touch-sensitive input field, the corresponding virtual information is selectable.

21. The method according to any one of the preceding claims,

wherein in the display device (21, 31) an edge (6) is displayed, which indicates a range of the first area (3), wherein a boundary of the first area by a user action, in particular by pulling the boundary, changeable is.

22. The method according to any one of the preceding claims,

wherein the at least one virtual information (POI) is representable in at least three stages, wherein a first stage comprises a body (POIl) as merely a local indication of the virtual information, a second stage an indication of the virtual information in the form of a labeled label ( Ll) and a third stage has an extract-like preview (Vl) of the virtual information, which is shown in particular when the point-like representation (POIl) or the label (Ll) is selected by a user action.

23. The method of claim 22, wherein

in a first part of the second area (4) virtual information (POI) is displayed only in the first stage, in a second part of the second area (4) with closer than in the first part of the second area on the display device

(21, 31) and in a first part of the first area (3) virtual information (POI) is displayed in the second stage, and in a second part of the first area (3) with closer than in the first part of the first Area on the display device (21, 31) placed real objects the virtual

Information (POI) is displayed in the third stage.

24. The method according to any one of the preceding claims, wherein the second area (4) or third area (5) is hidden from the view when no virtual information (POI) in the second area or third area is displayed.

25. The method according to any one of the preceding claims,

wherein a boundary (radius 1) between the first area (3) and the second area (4), in particular a radius of the boundary, depending on a size of the displayed virtual information (POI), a resolution of the display device (21, 31) and / or a resolution of a camera (22, 32) used to generate the view (1) is calculated.

26. The method according to any one of the preceding claims,

wherein dynamic weather information in relation to the real environment is taken into account in the insertion of the at least one virtual information (POI) in the view of the real environment.

27. Method according to one of the preceding claims,

wherein the at least one virtual information represents an interesting point (POI) with respect to the real environment.

28. The method according to any one of the preceding claims, wherein

a boundary of the third area (5) becomes larger with an increased measurement uncertainty of a position detection.

29. The method according to any one of the preceding claims, wherein

a limit, in particular with a radius, which determines which virtual information (POI) is to be displayed at all, the current speed or the average speed of the user or the distance which the user with public transport, vehicle or bicycle or the like in a particular Time can reach, depends.

30. A method for overlaying virtual information with reality, wherein the insertion of virtual information in a view of a real environment, weather data are taken into account, in particular via the Internet ("online") be queried to increase the level of reality of displayed virtual information in relation to the real environment and thus to improve the assignment.