EP2489013A1

EP2489013A1 - Method for representing virtual information in a view of a real environment

Info

Publication number: EP2489013A1
Application number: EP10768226A
Authority: EP
Inventors: Peter Meier; Michael Kuhn; Frank Angermann
Original assignee: Metaio GmbH
Current assignee: Apple Inc
Priority date: 2009-10-12
Filing date: 2010-10-11
Publication date: 2012-08-22
Also published as: US10074215B2; CN102598064A; US20220375123A1; DE102009049073A1; JP2013507677A; US20150154803A1; US11880951B2; CN105046752A; US20200160605A1; US20240127561A1; EP3410405A1; US20190005734A1; CN102598064B; US10453267B2; US11410391B2; US20120218263A1; WO2011045276A1; JP5582548B2; US9001154B2; CN105046752B

Abstract

The invention relates to a method for representing virtual information in a view of a real environment, comprising the following steps: providing at least one virtual object (10) having a global position and orientation with respect to a geographic global coordinate system (200), together with first position data (PW10) that allow conclusions about the global position and orientation of the virtual object, in a database (3) of a server (2), recording at least one image (50) of a real environment (40) by means of a mobile device (30), and providing second position data (PW50) that allow conclusions about the position and orientation at which the image was taken with respect to the geographic global coordinate system (200), representing the image (50) onto a display (31) of the mobile device, accessing the virtual object (10) in the database (3) of the server (2) and positioning the virtual object (10) in the image (50) shown on the display on the basis of the first and second position data (PW10, PW50), manipulating the virtual object (10) or adding a further virtual object (11) by corresponding positioning in the image (50) shown on the display, and providing the manipulated virtual object (10) together with modified first position data (PW10) corresponding to the positioning in the image (50) and the further virtual object (11), together with third position data corresponding to the positioning in the image (50) in the database (3) of the server (2), wherein the modified first position data and third position data each allow conclusions about the global position and orientation of the manipulated or further virtual object. Instead of with an image of the real environment, the method can also be performed analogously, using a view of an HMD for example.

Description

Method for presenting virtual information

in a view of a real environment

The present invention relates to a method for displaying virtual information in a view of a real environment.

Background of the invention

Augmented Reality (AR) is a technology that overlays virtual data with reality, facilitating the association of data with reality. In the prior art, the use of mobile AR systems is already known. In recent years, high-performance mobile devices (e.g., smartphones) have been found to be suitable for AR deployment. They now have comparatively large color displays, built-in cameras, good processors and additional sensors, such as orientation sensors and GPS. In addition, the position of the device can be approximated via radio networks.

In the past, various projects were performed on mobile devices using AR. First, special optical markers were used to determine the position and orientation of the device. Concerning AR, which can also be used extensively, also called large area AR, hints for the meaningful representation of objects have also been published in connection with HMDs (Head Mounted Displays) [3]. Lately, there are also approaches, the GPS and to use the orientation sensors of more modern devices ([1, 2, 4,5]).

[1] AR Wikitude. http://www.mobilizy.com/wikitude.php.

Enkln. http://www.enkin.net.

[3] S. Feiner, MacIntyre example, T. H ^'ollerer, and A. Webster. A touring machine:

Prototyping 3d mobile augmented reality Systems for exploring

the urban environment. In Proceedings of the Is International Symposium

on Wearable Computers, pages 74-81, 1997.

[4] Sekai Camera, http://www.tonchidot.com/product-lnfo.html.

[5] layar.com

However, the previously published approaches have the disadvantage that they do not allow the simple inclusion of other users in the AR scenes. Furthermore, the Most systems based on GPS and compass have the disadvantage that these devices must be available and high inaccuracies may occur.

US 2009/0179895 A1 describes a method for fading in three-dimensional notes or quotations into an image of a real environment ("street view"), wherein a user selects by means of a selection box in the picture at which point a note is to be faded in Next, the selection box is projected onto a three-dimensional model to determine a position of the indicia in relation to the image, and location data corresponding to the projection on the three-dimensional model is determined and assigned to the user-entered memo stored in a database of a server and can be displayed in a different image of the real environment according to the location data.

"Tagging" generally and in the following refers to augmenting the reality with additional information provided by a user. Previous tagging approaches include placing objects on map views (eg, Google Maps), photographing location points, and storing those images with additional ones It is disadvantageous that remote viewers and users can no longer have AR access to interactive scenes in the world, only so-called screenshots of the AR scene can be viewed but not changed any more ,

It is an object of the present invention to provide a method for displaying virtual information in a view of a real environment that allows users to interactively view augmented reality AR-image scenes created by other users while ensuring high accuracy and user-friendliness ,

Summary of the invention

In a first aspect of the invention, there is provided a method of displaying virtual information in a real-world view, comprising the steps of: providing at least one virtual object having global position and orientation with respect to a geographic global coordinate system with first pose data, which allow conclusions to be drawn about the global position and orientation of the virtual object, on a database of a server, recording of at least one image of a real environment by means of a mobile device, and provision of len of second Posedaten that allow a conclusion about the position and orientation with respect to the geographic global coordinate system, the image was taken, displaying the image on a display of the mobile device, access to the virtual object on the database of the server and positioning the virtual object in the image displayed on the display based on the first and second pose data, manipulating the virtual object or adding another virtual object by appropriate positioning in the image displayed on the display, and providing the manipulated virtual object along with modified ones first pose data corresponding to the positioning in the image or the further virtual object together with third pose data corresponding to the positioning in the image on the database of the server, wherein the modified first pose data and third pose data each have a conclusion f allow the global position and orientation of the manipulated or further virtual object. In this case, the image can be provided, for example, together with the second pose data on the server.

In another aspect of the invention, there is provided a method of presenting virtual information in a real-world view, comprising the steps of: providing at least one virtual object having global position and orientation with respect to a geographic global coordinate system on a database of a server, providing at least one view of a real environment by means of data glasses (for example a so-called optical see-through data goggles or a video camera) with first pose data which permit a conclusion on the global position and orientation of the virtual object See-through data goggles) along with second pose data allowing conclusions about where and with what orientation with respect to the geographic global coordinate system the goggles are positioned, access to the virtual object on the server's database and posit ionizing the virtual object in the view based on the first and second pose data, manipulating the virtual object or adding another virtual object by correspondingly positioning in the view, and providing the manipulated virtual object together with modified first pose data corresponding to the positioning in the view; of the further virtual object together with third pose data corresponding to the positioning in the view on the database of the server, wherein the modified first pose data and third pose data each allow a conclusion about the global position and orientation of the manipulated or further virtual object. In one embodiment of the invention, the mobile device or the smart glasses has or is connected to a device for generating the second pose data.

For example, the pose data may include respective three-dimensional values in position and orientation. In addition, an orientation of the image of the real environment can be defined independently of the earth's surface.

According to a further embodiment of the invention, a storage location on the server records in which image of a plurality of images of a real environment or in which view of several views of a real environment which virtual object has been provided with pose data by a plurality of virtual objects.

If the position of the mobile device is determined, for example, by means of a GPS (Global Positioning System) sensor, it may happen that the position of the mobile device is only determined relatively inaccurately due to a sensor inaccuracy or a GPS-inherent inaccuracy. This can have the result that superimposed virtual objects in the image relative to the geographic global coordinate system are also positioned with a corresponding inaccuracy, so that in other images or views with different angles, the virtual objects displayed there are displayed out of place in relation to reality.

In order to improve the accuracy of the representation of virtual objects or their position in the image of the real environment, an embodiment of the method according to the invention comprises the following steps: providing a reference database with reference views to a real environment together with pose data containing a Infer at which position and with which orientation with respect to the geographic global coordinate system the respective reference view was taken by a camera, comparison of at least one real object depicted in the image with at least one part of a real object, included in at least one of the reference views and matching the second pose data of the image with the pose data of the at least one reference view, and modifying at least a portion of the second pose data based on at least a portion of the pose data of the at least one reference view of the Abgle I ... it. In a further embodiment, at least a portion of the first pose data of the virtual object positioned in the image is also modified as a result of matching the second pose data of the image with the pose data of the at least one reference view.

Further embodiments and embodiments of the invention are specified in the subclaims.

Aspects and embodiments of the invention are explained in more detail below with reference to the illustrated figures.

Brief description of the drawings

1A is a plan view showing a schematic arrangement of a first exemplary embodiment of a system structure usable to carry out a method according to the invention;

Fig. 1B is a plan view showing a schematic arrangement of a second exemplary embodiment of a system structure usable to carry out a method according to the invention;

FIG. 1C shows, in a schematic arrangement, a possible data structure of an embodiment of a system in order to carry out a method according to the invention, FIG.

2 shows a schematic representation of an overview of participating coordinate systems according to an embodiment of the invention,

3 shows an exemplary sequence of a method according to an embodiment of the invention,

4 shows an exemplary sequence of a method according to a further embodiment of the invention, in particular supplemented by optional measures for improving the image pose,

FIG. 5 shows an example scene of a real environment with virtual objects placed therein, without a pose improvement taking place, FIG. FIG. 6 shows an exemplary scene of a real environment with virtual objects placed therein after a pose improvement has taken place.

7A shows an exemplary real world map view on which a virtual object has been placed;

FIG. 7B shows an exemplary perspective view of the same scene as in FIG.

7A.

Description of embodiments of the invention

Fig. 1A shows in plan view a schematic arrangement of a first exemplary embodiment of a system structure that is usable to carry out a method according to the invention.

In the illustration of Fig. 1A, the user carries as a display device a head-mounted display system (HMD) with a display 21 which is part of the system body 20. At least parts of the system body 20 can be considered as a mobile device The component may be connected to one another by wire connections and / or also wirelessly, and it is also possible for individual components, such as the computer 23, to be stationary The display 21 may, for example, a generally known data glasses in the form of a so-called optical see-through data glasses ("Optical See Through Display", in which the reality by the semipermeability the data glasses is visible) or in the form of a so-called video-see-through data glasses ("Video See Through-display ", in which the reality is played on a worn in front of the user's head screen) may be in the virtual information provided by a computer 23, can be displayed in a known manner. The user then sees in a view 70 visible to the real world within a viewing angle or opening angle 26 by the display 21 or on the display 21 objects of the real environment 40 that are displayed with virtual information 10 (such as so-called point-of-interest Objects, in short, POI objects that are related to the real world) can be enriched. The virtual object 10 is displayed such that the user perceives it in a manner as if it were located in the real environment 40 at an approximate position. This position of the virtual object 10 may also be referred to as a global position with respect to a geographic location global coordinate system, such as an earth coordinate system, are stored, as explained in more detail below. In this way, the system assembly 20 forms a first embodiment of a well-known augmented reality system that can be used for the method of the present invention.

Additional sensors 24, such as rotation sensors, GPS sensors or ultrasonic sensors, and an optical tracking camera 22 and the recording of one or more images (so-called "views") may be attached to the display 21. The display 21 may be semitransparent or through If the display 21 is semitransparent, calibration is required between the user's eye 25 and display 21. This procedure, known as the Seethrough Calibration, is well known to those skilled in the art from eye to camera 22. The camera can be used to take views to make them accessible to other users, as explained in more detail below As a pose, the position and orientation of an object or object is generally related to a reference coordinate system To determine poses are in the art various methods are documented and known to those skilled in the art. Advantageously, on the display 21 or anywhere on the body of the user, or in the computer 23 position sensors may be installed, such as GPS sensors (GPS: Global Positioning System) to a geographical location of the system structure 20 (eg by latitude and longitude and altitude) in the real world 40. Basically, the determination of the position of each part of the system structure is suitable, provided that conclusions about the position and viewing direction of the user can be made.

In the illustration of Fig. 1B, another exemplary system assembly 30 is shown, e.g. A display device 31 (eg in the form of a screen or display), computer 33, sensors 34 and camera 32 form a system unit which is accommodated approximately in a common housing of a mobile phone At least portions of the system assembly 30 may be considered as a mobile device comprising one or more of said components, the components may be disposed in a common housing or (partially) distributed and interconnected by wire connections and / or wirelessly be.

The view of the real environment 40 is provided by the display 31, which displays an image 50 of the real environment 40, which can be viewed by the camera 32 in a glance. and was recorded with an opening angle 36. For augmented reality applications, the camera image 50 may be displayed on the display 31 and augmented with additional virtual information 10 (such as POI objects related to the real world) that have a particular position relative to reality as described with reference to FIG. 1A. In this way, the system assembly 30 forms another embodiment of a well known Augmented Reality (AR) system.

A similar calibration is used as described with respect to Figure 1A to determine the pose of virtual objects 10 relative to the camera 32 in order to make them accessible to other users, as further explained below. To determine poses, various methods have been documented in the prior art and known to the person skilled in the art. Advantageously, on the mobile device (in particular, if the system assembly 30 is formed as a unit) or anywhere on the user's body, or even in the computer 33 position sensors may be installed, such as GPS sensors 34, a geographic location of the system structure 30th (eg, latitude and longitude) in the real world 40 to allow. In certain cases, no camera is needed to determine the pose, for example if the pose is determined only by GPS and orientation sensors. Basically, the determination of the position of each part of the system structure is suitable, provided that conclusions about the position and viewing direction of the user can be made.

In principle, the invention can be usefully used for all forms of AR. For example, it does not matter if the display is performed in the so-called optical see-through method with semipermeable HMD or in the video see-through method with camera and screen.

In principle, the invention can also be used in connection with stereoscopic displays, whereby advantageously two cameras each record a video stream for an eye in the video see-through approach. In any case, the virtual information for each eye can be calculated individually and stored as a pair on the server.

The processing of the different substeps described below can basically be distributed to different computers via network. So it is a client / server architecture or a more client-based solution possible. Furthermore, the client or the server can also have several computing units, such as multiple CPUs or specialized hardware components, such as well-known FPGAs, ASICs, GPUs or DSPs include.

In order to realize AR, the pose (position and orientation) of the camera in the room is needed. This can be realized in many different ways. One can e.g. only with GPS and an orientation sensor with electronic compass (as used for example in some more modern mobile phones) determine the pose in the real world. However, the uncertainty of the pose is then very high. Therefore, other methods such as optical initialization and tracking or the combination of optical methods with GPS and orientation sensors may be used. WLAN localization can also be used, or RFIDs (markers or chips for "Radio Frequency Identification") or optical markers can support the localization. Again, as already mentioned, a client-server-based approach is possible the client may request site-specific information required for optical tracking from the server, such as, for example, reference images of the environment containing pose information and depth information An optional embodiment of this invention provides, in particular, the ability to enhance the pose of a view on the server and also to improve the pose of the placed virtual objects in the world based on this information.

Furthermore, the invention can be installed or carried in vehicles, aircraft or ships using a monitor, HMDs or a head-up display.

Basically, virtual objects, such as an interesting point (Point of Interest, POI), can be created for a wide variety of forms of information, examples are given below: Images of places with GPS information can be displayed - information from the Internet These can be, for example, company or restaurant websites with addresses or pages on which evaluations are made, users can deposit texts, images or 3D objects in specific locations and make them accessible to others Wikipedia, for example, searches for geo-information and makes the pages accessible as POI, and POIs can be automatically generated from the search or browse behavior of mobile device users, or other interesting places, such as subways or bus stations, hospitals , Police stations, doctors, real estate ads or fitness clubs are presented , Such information can be stored in the image 50 or in the view 70 (see Figures 1A and 1B) as virtual objects 10 at specific locations in the real world 40 by a user and made accessible to others with the position corresponding to the respective location. The other users can then, for example, manipulate the information shown in their position in an accessible view or a picture of the real world, or they can also add further virtual objects. This will be explained in more detail below.

First, FIG. 1C shows data structures used in accordance with one embodiment of the invention and briefly explained below.

A view is a fixed view of the real world, in particular a view (see view 70 according to FIG. 1A), an image (see image 50 according to FIG. 1B) or a sequence of images (a film). The view (image 50 / view 70) is associated with camera parameters which describe optical properties of the camera 22, 32 (for example with regard to the aperture angle, main point shift or image distortion) and to the image 50 or the view 70. Similarly, pose data are also associated with the view, which describe the position and orientation of the image 50 or the view 70 with respect to the earth. For this purpose, the earth is assigned a geographically global coordinate system in order to obtain a geographic global position determination in the real world, e.g. longitude and latitude, to allow.

A placed model is a graphically representable virtual object (compare object 10 according to FIGS. 1A, 1B), which also has pose data. For example, the placed model may represent an instance of a model from a model database, that is, refer to it. It is advantageously deposited by means of which views 50 or 70 the respective virtual model 10 has been placed in the world 40, if this is the case. This can be used to improve the pose data, as explained in more detail below. A scene represents a link from a view 50, 70 with 0 to n placed models 10 and optionally includes a creation date. All or part of the data structures may still be linked to metadata. For example, the creator, the date, the frequency of images / views, ratings and keywords can be dropped.

In the following, aspects of the invention will be explained in more detail with reference to the embodiment according to FIG. 1B, in which an image 50 is taken by a camera 32 and displayed by the user on the display 31 together with displayed virtual objects. 10 is considered. The relevant explanations are, however, readily analogously transferable by the person skilled in the art to the embodiment with HMD according to FIG. 1A.

FIG. 2 gives an overview of participating coordinate systems according to an embodiment of the invention. On the one hand, an earth coordinate system 200 is used (which in this embodiment represents the geographic global coordinate system), which is a connecting element. The earth's surface is indicated in Figure 2 by reference numeral 201. To define a geographic global coordinate system, such as an earth coordinate system 200, various standards are defined that are known to those skilled in the art (eg WGS84; National Imaging and Mapping Agency: Department of Defense World Geodetic System 1984; Technical Report, TR 8350.2; 3rd edition, January 2000). Furthermore, a camera coordinate system represents a connection between displayed virtual objects 10 and images 50. Conversions known to the person skilled in the art can pose P50_10 ("pose model in picture") of the pose of camera 32 and image 50 in the earth coordinate system 200 of an object 10 relative to the image 50. The global image pose PW50 ("Pose image in the world"), for example, becomes calculated via GPS and / or orientation sensors. From the poses PW50 and P50_10, the global pose PW10 of the virtual object 10 ("pose model in the world") can then be calculated,

Similarly, from the pose of a second image 60 with another global pose PW60 in the earth coordinate system 200, the pose P60_10 ("pose model in image 2") of the object 10 relative to the image 60 can be calculated. Pose picture 2 in the world ") becomes eg also calculated via GPS and / or orientation sensors.

In this way, it is possible to place a virtual object 10 in a first image (image 50) and view it in a second image (image 60) at a nearby position on the earth but from a different angle. The object 10 is placed, for example, by a first user in the first image 50 with the pose PW10. If a second user then generates a view according to FIG. 60 with his mobile device, the virtual object 10 placed by the first user is automatically displayed in the image 60 at the same global position corresponding to the pose PW10, provided that the image 60 is acquired in one Aperture angle or angle of view is a part of the real world that includes the global position of the pose P Wl 0. Aspects and embodiments of the invention will be explained in more detail below with reference to the flow diagrams of FIGS. 3 and 4 in conjunction with the other figures.

FIG. 3 shows an exemplary sequence of a method according to an embodiment of the invention. In a first step 1.0, world related data is generated. These may for example be extracted from the Internet or generated by a first user by means of a camera (FIG. 1B) or an HMD with camera (FIG. 1A). To this end, in step 1.0, it takes up a view (image or view) for which position and orientation (pose) in the world are determined (step 2.0). This can be done for example by means of GPS and compass. Optionally, information regarding uncertainty of the generated data may also be included.

If the view (the image or the view) exists, the user can advantageously place a virtual object in the view directly on his mobile device (step 3.0). Advantageously, the object is placed and manipulated in the camera coordinate system. In this case, in step 4.0, from the global pose of the view and pose of the object in the camera coordinate system, the global pose of the virtual object (or objects) in the world (e.g., with respect to coordinate system 200) is calculated. This can be done on a client 1 or done on a server 2.

A client is a program on a device that makes contact with another program on a server in order to use its services. The underlying client-server model allows tasks to be distributed to different computers in a computer network. A client does not solve one or more specific tasks itself, but has them done by the server or receives corresponding data from the server, which offers a service for it. Basically, most steps of this system can be done either on the server or the client. For high-performance clients, for example, it is advantageous to have them perform as many calculations as possible, thus relieving the load on the server.

In step 5.0, this information from step 4.0 is then stored in a database 3 of the server 2, advantageously as described with reference to FIG. In step 6.0, the same user or other user on another client then takes a picture of the real environment (or views a particular part of the environment via an HMD) and then loads data stored in step 5.0 relative to a location of the considered one Real environment from the server 2, The loading and display of location-related information by means of augmented reality and advantageously with Geospatial functionalities equipped database is known in the art. The user now sees the previously stored information from the previously stored or a new point of view and is able to make changes (manipulations of existing and / or adding new virtual information), which in turn are stored on the server 2. In this case, the user does not necessarily have to be on-site, but can use the previously, advantageously stored view as a window on the reality while sitting in his office, for example on an Internet-enabled client.

In the example of FIGS. 1 and 2, in this way, a virtual object 10 is provided or generated by a user on the database 3 of the server 2, which has a global position and orientation with respect to a geographic global coordinate system 200, together with the pose data (Pose PW10), which allow a conclusion on the global position and orientation of the virtual object 10. This user or another user takes at least one image 50 of a real environment 40 by means of a mobile device 30 together with the pose data (pose PW50), which allow conclusions to be drawn as to which position and with which orientation with respect to the geographic global coordinate system 200 picture 50 was taken. The image 50 is displayed on the display 31 of the mobile device. Access is made to the virtual object 10 on the database 3 of the server, and the virtual object 10 is positioned in the image 50 displayed on the display based on the pose data of the poses PW10 and PW50. The virtual object 10 can then be manipulated (eg displaced) by corresponding positioning (compare arrow MP in FIG. 1B) in the image 50 shown on the display, or another virtual object 11 can be positioned by positioning it in the one shown in FIG Display 50 can be added.

Such a manipulated virtual object 10 together with the modified pose data (modified pose PW10) corresponding to the positioning in the image 50 or such another virtual object 11 together with its pose data corresponding to the positioning in the image 50 is then stored on the database 3 of the server 2, wherein the modified pose data PW10 and the pose data of the new virtual object 11 each allow a conclusion on the global position and orientation of the manipulated object 10 or other virtual object 11 with respect to the coordinate system 200. In certain cases, it may happen that the server is not reachable and that, for example, saving the new scene is not possible. In this case, advantageously, the system can respond and cache the information until the server is reachable again. In one embodiment, in the event of failure of the network connection to the server, the data to be stored on the server is cached on the mobile device and transmitted to the server as soon as the network connection is available again.

In another embodiment, the user may retrieve a collection of scenes in an area of a real environment (e.g., in its surrounding area or vicinity) that are sorted by proximity in a list, or provided on a map or by augmented reality for selection.

In another embodiment, the image or virtual information has unique identifying characteristics (eg, unique names), and image or virtual information already present on a client or mobile device (these may be virtual model data or views). , are no longer downloaded from the server, but are loaded from a local datastore.

FIG. 4 shows an exemplary sequence of a method according to a further embodiment of the invention, in particular supplemented by optional measures for improving the image pose. The method comprises steps 1.0 to 6.0 of FIG. In addition, according to FIG. 4, in steps 7.0 and 8.0 the pose of the view (image or view) is subsequently improved, for example by means of optical methods, and the advantageous storage of the information by means of which views has placed which virtual information is the pose of the information also corrected. Alternatively, the pose of the view can be improved immediately after the creation of the view on the client 1 by providing optical tracking reference information for this view or a view with a similar pose to the client 1 from a reference database 4 of the server 2. Alternatively, the accuracy of the view can also be done before calculating the pose of the placed virtual objects (step 4.0) and saved directly correctly. However, the advantage of the subsequent procedure is that reference data do not have to be available for all locations and a correction can be made for such views as soon as reference data is available.

Of course, other views can also be used as reference data, especially if there are many views for a location. This was called a bundle adjustment The method is known to the person skilled in the art, as for example in the publication of MANOLIS IA LOURAKIS and ANTONIS A. ARGYROS: SBA: A Software Package for Generic Sparse Bundle Adjustment. In ACM Transactions on Mathematical Software, Vol. 1, Article 2, Publication date: March 2009. In this case, the 3D position of the point correspondences, the pose of the views, and advantageously also the intrinsic camera parameters could be optimized. Thus, the approach according to the invention also offers the possibility of creating a separate model of the world in order to use this data in general. For example, for masking models to support depth perception or for real-time optical tracking.

FIG. 5 shows an exemplary scene of a real environment with virtual objects placed therein without a pose improvement having taken place so far. FIG. 5 shows a possible situation before a correction. A virtual object 10 (for example, a restaurant's rating) is placed from a view of a mobile device 30 in an image displayed on the display 31 of the device 30 relative to real objects 41, 42 (e.g., representing the building of the restaurant). On the basis of faulty or inaccurate GPS data, both the image and the object 10 with erroneous world coordinates are stored corresponding to incorrect or inaccurate particular camera pose data P30-2. This leads to a corresponding incorrectly stored object 10-2. This is not a problem in this captured image itself. The error, however, comes to light when the virtual object 10 is viewed, for example, on a map or in another image.

If the image had been generated along with true camera pose data P30-1, the virtual object 10 would be displayed at a position in the image as shown by the representation of the virtual object 10-1 and viewed by the generating user as well , However, the erroneously stored virtual object 10-2 is displayed shifted in an image other than the true position of the virtual object 10 by a degree by which the defective camera pose P30-2 is shifted from the true camera pose P30-1 , The representation of the erroneously stored virtual object 10-2 in the image of the mobile device 30 thus does not correspond to the true positioning by the generating user in a previous image.

In order to improve the accuracy of the representation of virtual objects or their position in the image of the real environment, an embodiment of the method according to the invention comprises the following steps: A reference database 4 with reference views to a real environment together with pose data is provided make it possible to infer at which position and with which orientation with respect to the geographic global coordinate system 200 the respective reference view was taken by a camera. Then, at least a portion of a real object depicted in the image is compared to at least a portion of a real object included in at least one of the reference views, and an alignment of the pose data of the image with the pose data of the at least one Reference view made. Thereafter, at least a portion of the pose data of the image is modified based on at least a portion of the pose data of the respective reference view as a result of the matching.

In a further embodiment, at least part of the pose data of the virtual object positioned in the image is also modified as a result of matching the pose data of the image with the pose data of the relevant reference view.

FIG. 6 shows an exemplary scene of a real environment similar to that of FIG. 5 with a virtual object 10-1 placed therein after a pose improvement has taken place. FIG. 6 shows on the one hand the mechanism of the recognition of image features in the image and on the other hand the corresponding correction of image pose and object pose. In particular, image features 43 (e.g., distinctive features of the real objects 41 and 42) are compared and matched with corresponding features of reference images of a reference database 4 (known as "matching" image features).

Now, the virtual object 10 would also be displayed correctly in other images (which have a correct pose), or a placement correction could take place. Placement correction means that the user could misjudge the height of the placed object above the ground when placing a virtual object in perspective. By means of two images overlapping in parts of the recorded reality, it may be possible to extract a floor plane and to relocate the placed objects so that they stand on the ground, but almost in the image in which they were originally placed to stay in the same place.

Figure 7A shows an exemplary real-world map view on which a virtual object has been placed, while Figure 7B shows an exemplary perspective view of the same scene as in Figure 7A. FIGS. 7A and 7B serve in particular to explain the user-assisted determination of the camera pose. For example, in the case of using mobile devices that are not equipped with compass, it is useful to obtain a rough estimate of the viewing direction. For this purpose, the user, as in Figure 7B represented as usual take a picture 50 and place a virtual object 10 in relation to a real object 41. Following this, the user may be prompted to re-display the position of a placed object 10 on a map 80 or world virtual view 80, as shown in Figure 7A. From the connection between the GPS position of the image 50 and the object position of the object 10 on the map 80, the orientation (heading) of the image 50 in the world can then be calculated or corrected. If the mobile device also has no GPS, the method can also be carried out with two virtual objects or one virtual object and the indication of the location. Further, as shown by way of example in Figure 7A, the user may also be presented with the "field of view" (see indicator 81 of the image detail) of the last image, and the user may select the "field of view" on the map for correction interactively move and reorient. In this case, the opening angle of the "Field of Views" can be shown according to the intrinsic camera parameters.

In particular, according to this embodiment, the method includes the following steps: providing a map view (see map view 80) on the display of the mobile device and providing a choice for the user to select a viewing direction when capturing the image. In this way, it is possible to select the viewing direction in which the user is currently looking with the camera on the map.

According to a further embodiment of the invention, the method also comprises the following steps: placing the virtual object in the image of the real environment and in a map view provided on the display of the mobile device, and determining an orientation of the image from a determined position of the image and the position of the virtual object in the provided map view. This makes it possible for virtual objects to be placed on the map and also in the perspective image of the real environment, from which it is possible to draw conclusions about an orientation of the user.

In order to enable other users to remotely view and edit a virtual objects-enriched image of a real-world environment (for example, on a client communicating with the server, for example, over the Internet), in one embodiment of the invention, that the method has the following further steps: At least one image of the real environment is provided along with its pose data on the server's database. Thereafter, access is made to the image of the real environment on the server and transmitted to a client device for displaying the image on the client device. The user manipulates the virtual object or adds another virtual object by appropriate positioning in the real environment image displayed on the client device. The manipulated virtual object, together with its modified pose data corresponding to the positioning in the image displayed on the client device, or the other virtual object is along with its (new) Posedaten according to the positioning in the image displayed on the client device provided the database of the server, wherein the modified Posedaten or new Posedata each allow a conclusion on the global position and orientation of the manipulated or further virtual object in the image displayed on the client device. Thus, in a 'remote access' to a client device, the AR scene in the image can be modified or enriched with additional virtual information and written back to the server. Due to the newly stored global position of the manipulated or new virtual information, this information can in turn be retrieved by other users via an access to the server and viewed in an AR scene corresponding to the global position.

Based on this, in yet another embodiment, the method comprises the following steps: accessing the image of the real environment on the server and transmitting it to a second client device for viewing the image on the second client device and accessing the server provided on the server virtual objects, wherein in the view of the image on the second client device those virtual objects are displayed whose global position lies within the real environment, which is shown in the view of the image on the second client device. In this way, a viewer on another client device can view a scene representing those virtual objects that other users have previously positioned at a corresponding location (whose global location is within the real environment in the view the image is displayed on this client device). In other words, the observer sees from his viewing angle those virtual objects that other users have previously placed in the visible field of view.

Claims

claims

1. A method for presenting virtual information in a view of a real environment, comprising the following steps:

Providing at least one virtual object (10) having a global position and orientation with respect to a geographic global coordinate system (200), together with first pose data (PW10), which inferences the global position and orientation of the virtual object on a database (3) of a server (2),

- Recording at least one image (50) of a real environment (40) by means of a mobile device (30) and providing second Posedaten (PW50), which allow a conclusion about what position and with which orientation with respect to the geographic global coordinate system (200) the picture was taken,

Displaying the image (50) on a display (31) of the mobile device,

Accessing the virtual object (10) on the database (3) of the server (2) and positioning the virtual object (10) in the image (50) displayed on the display based on the first and second pose data (PW10, PW50) .

Manipulating the virtual object (10) or adding another virtual object (11) by appropriate positioning in the image (50) displayed on the display,

Providing the manipulated virtual object (10) together with modified first pose data (PW10) corresponding to the positioning in the image (50) and the further virtual object (11) together with third pose data corresponding to the positioning in the image (50) on the Database (3) of the server (2), wherein the modified first Posedaten and third Posedaten each allow a conclusion about the global position and orientation of the manipulated or further virtual object.

The method of claim 1, wherein the image (50) is provided along with the second pose data (PW50) on the server.

3. A method of presenting virtual information in a view of a real environment, comprising the steps of:

Providing at least one virtual object (10) having a global position and orientation with respect to a geographic global coordinate system (200), on a database (3) of a server (2) together with first pose data (PW10), which permit a conclusion on the global position and orientation of the virtual object,

Providing at least one view (70) of a real environment (40) by means of data goggles (20) together with second pose data which allow conclusions to be drawn as to which position and with which orientation with respect to the geographic global coordinate system (200 ) the data glasses are positioned,

Accessing the virtual object (10) on the database (3) of the server (2) and positioning the virtual object (10) in the view on the basis of the first and second pose data,

Manipulating the virtual object (10) or adding another virtual object (11) by appropriate positioning in the view (70),

Providing the manipulated virtual object (10) together with modified first pose data (PW10) corresponding to the positioning in the view (70) or the further virtual object (11) together with third pose data corresponding to the positioning rank in the view (70) on the database (3) of the server (2), wherein the modified first Posedaten and third Posedaten each allow a conclusion on the global position and orientation of the manipulated or further virtual object.

4. The method according to any one of the preceding claims, wherein

the mobile device (30) or the data glasses (20) has a device (24, 34) for generating the second pose data or is connected to such a device.

A method according to any one of the preceding claims, wherein the pose data (PW10, PW50) include respective three-dimensional values of position and orientation.

6. The method according to any one of the preceding claims, wherein an orientation of the image (50) of the real environment is defined independently of the earth's surface (201).

7. The method according to any one of the preceding claims, wherein in a storage location on the server (2) is recorded, in which image (50) of a plurality of images of a real

Environment or in which view (70) of several views of a real environment which virtual object (10) was provided by several virtual objects with Poss data.

8. The method according to any one of the preceding claims, with the following further steps: - Providing a reference database (4) with reference views to a real environment together with Poss data, which can draw conclusions about the position and orientation with respect to the geographic global coordinate system (200) the respective reference view of a camera was shot,

Comparing at least one real object (41, 42) depicted in the image with at least a part of a real object included in at least one of the reference views and matching the second pose data (PW50) of the image with the Pose data of the at least one reference view,

Modifying at least a portion of the second pose data (PW50) based on at least a portion of the pose data of the at least one reference view as a result of the alignment.

9. The method of claim 8, wherein

In addition, at least a portion of the first pose data (PW10) of the virtual object (10) positioned in the image is modified as a result of matching the second pose data (PW50) of the image with the pose data of the at least one reference view.

10. The method according to any one of the preceding claims, with the following further steps:

Providing the at least one image (50) of the real environment together with the second pose data (PW50) on the database (3) of the server,

Accessing the image (50) of the real environment on the server (2) and transmitting it to a client device (1) for displaying the image on the client device,

Manipulating the virtual object (10) or adding another virtual object (11) by appropriate positioning in the image of the real environment displayed on the client device,

- Providing the manipulated virtual object (10) together with modified first Posedaten (PW10) according to the positioning in the image displayed on the client device or the other virtual object (11) together with third Posedaten according to the positioning in the at the client Device displayed on the database (3) of the server, wherein the modified first Posedaten and third Posedaten each allow inference to the global position and orientation of the manipulated or further virtual object in the image displayed on the client device.

The method of claim 10, further comprising the steps of:

Accessing the image (50) of the real environment on the server (2) and transmitting to a second client device to view the image on the second client device and Access to virtual objects (10) provided on the server, wherein in the view of the image on the second client device are represented those virtual objects whose global position lies within the real environment, that in the view of the image on the second client Device is shown.

12. The method according to any one of the preceding claims, with the following further steps:

- Providing a map view (80) on the display of the mobile device and

- Providing a choice for the user to select a viewing direction when recording the at least one image of the real environment.

13. The method according to any one of the preceding claims, comprising the following steps:

Placing at least one virtual object (10) in the image of the real environment and in a map view (80) provided on the display of the mobile device,

- Determining an orientation of the image (50) of the real environment from a determined position of the image and the position of the at least one virtual object (10) in the provided map view (80).

14. The method according to any one of the preceding claims, wherein in case of failure of the network connection to the server (3) to be stored on the server data to be stored on the mobile device and transmitted to the server as soon as the network connection is available again.

A method according to any one of the preceding claims, wherein the user can retrieve a collection of scenes in an area of a real environment sorted by proximity in a list, or on a map, or augmented reality for selection.

16. The method according to any one of the preceding claims, wherein the image or the virtual information have uniquely identifying properties, and an image or virtual information that already exists on a client or on the mobile device are no longer downloaded from the server, but be loaded from a local data store.