DE102016116582A1 - Method and apparatus for displaying augmented reality - Google Patents

Abstract

Thus, there is provided a computer-implemented method of presenting augmented reality based on images of a real three-dimensional scene, wherein, by a first AR device, based on an image, a section of the image is defined as a target and of the section deriving information that is descriptive of the target, and wherein at least one virtual three-dimensional container object is initialized and provided with a position indication, wherein the position statement has a geometric relation to the target and wherein the information describing the target, a Information indicative of the at least one virtual container object and the location indication for the AR-like display of the container object is provided by the first AR device and, via a communication link, for the AR display of the container object for provided at least a second AR device becomes.

Description

The Augmented Reality technique gives a user the technical possibility of displaying virtual objects in real images on a display device (smartphone, tablet, ...) so that a mixed image of virtual reality can be seen in the visual perception of the viewer and real elements, called virtual or AR scene arises. It can also be said that the virtual objects act as a kind of hologram, which can only be viewed through a specific "window" (in this case the display of, for example, a smartphone). From the prior art similar or related applications are already known and there are already various approaches. In most cases, a target previously defined in the real image is used for this, which contains particularly well machine-recognizable visual features (for example a QR code). Virtual objects are then "placed" on this target. The target thus serves as a reference point for the virtual objects within the virtual scene.

In addition, there is already the possibility to let the user define his own target according to the current environment. He takes a photo of a physically existing object that he wants to use as a target and then takes the target directly as a fixed point for his virtual objects. Depending on the framework used, this technique is referred to as a "user defined target" (UDT) or "target on the fly". This approach is taken up in the present invention.

The virtual objects in known applications are mostly any static objects that enhance the environment, e.g. for additional information or illustrations.

As multi-user or multiplayer AR applications, there are games like e.g. Scavenger hunts (e.g., "Pokémon Go") through cities where game participants must find certain locations, focus them with the camera of the smartphone and then, e.g. Treasures get displayed in the form of virtual objects. Often the virtual objects are simply placed over the current camera image as soon as the player has reached a certain geographic location.

Instead of a smartphone, any other device may be used that has a display, a camera, and a data communication interface (e.g., tablets or wearables such as the "Google Glass" goggle or the Project Tango). In the following, all these devices are commonly referred to as "AR devices".

However, with previous AR applications, users can not communicate interactively with virtual user-related objects (player object) in the AR scene. Each user is only shown in relation to his current position (determined by GPS) an image that makes sense from its position and perspective.

Other users also see only the virtual object displayed in their camera image. However, if, for example, turning the virtual object toward a player or otherwise moving it has no direct effect on what the other users see on their AR devices.

Also, the "virtual physics" is slightly different on each device, if e.g. virtual light (or wind or similar) is used for the virtual scene.

Also, the current environmental conditions (such as time of day, weather, ...) and lighting conditions are not considered there.

Thus, on the whole, it is not possible to influence the elements in augmented reality by a user on their AR device so that other users can see them directly (in real time) on their AR devices.

The object is thus to provide methods and devices which at least partially overcome these disadvantages.

This object is achieved by the method and apparatus according to the independent claims. Advantageous developments are defined in the dependent claims.

Accordingly, what is provided is a computer-implemented method for presenting augmented reality based on images of a real three-dimensional scene, wherein, defined by a first AR device, on the basis of an image, a section of the image as a target and derived from the section of the image information that is descriptive of the target, and wherein at least one virtual three-dimensional Container object is initialized and provided with a location specification, wherein the location statement has a geometric relation to the target and wherein the information describing the target, information indicative of the at least one virtual container object, and the location indication to the AR The container device is provided by the first AR device and is provided, via a communication link, for the AR display of the container object for at least one second AR device.

Also provided is a computer-implemented method of presenting augmented reality based on images of a real three-dimensional scene, wherein, by a second AR-device, information that is descriptive of a target, and information that is suitable for at least one The position statement has a geometric relation to the target, is received via a communication connection coming from a first AR device and, based on the received information, the target in images, recorded by the second AR. Device, is focused and the virtual container object according to the relation AR-moderately displayed. Advantageous developments of the invention may include the following features.

By means of an AR device, at least one virtual three-dimensional user object can be initialized and provided with a position specification, the position specification having a geometric relation to the position specification of the container object, wherein the at least one three-dimensional user object is detected by the AR device is controlled and information indicative of the at least one virtual user object, and the location indication for AR-moderate display is provided by the AR device and provided for displaying the user object for at least one further AR device.

It is advantageous if the provision is made in real time.

It is also advantageous if the container object is provided with a root property of a predetermined hierarchical relation and at least one virtual object is provided with the predetermined hierarchical relation to another virtual object, wherein the root is arranged at the highest hierarchical level such that the virtual objects provided with a hierarchical relation together with the root object represent a hierarchical tree structure.

The predetermined hierarchical relation may include "is subordinate".

Each location indication may include at least one of location, orientation, and scale of the object in the scene, most preferably all three.

The three-dimensional object may include a description for representing the article by an AR device.

The AR representation can be obtained by optical superimposition of the virtual objects represented by a virtual camera function and the image of the scene acquired by means of a real camera function, the virtual camera function being similar to the real camera function in terms of focal length and perspective.

It is also advantageous if, by means of a second AR device, the target is identified on the basis of the provided target information in an image and focused with the real camera function.

Each virtual object in the scene can be moved (in the AR scene) by changing the position indication by means of an AR device.

Advantageously, not only the movement but also any other interaction (e.g., color change of an object) may be effected by the AR device.

In this case, both the user-related object and the shared object can be controlled.

It is particularly advantageous that during a movement of a virtual object each object is moved, which is arranged in the hierarchical structure from the root behind / below the moving object.

It is advantageous if, by the first AR device, a first server is initialized which defines for how many AR devices the scene is retrievably kept available, provides network information as well as information which is descriptive of a game for which the Objects are provided.

The first server may provide information for synchronizing the locations and / or other application-relevant information of all objects of the scene for retrieval by at least one AR device.

It is also advantageous that a second server is initialized by the first AR device, which describes the information describing the target, the information that is indicative of the at least one virtual container object, and the position indication for the AR-moderate display of the container Object for at least one AR device for retrieval by at least one AR device. Advantageously, the second server can run on the first AR device.

It is advantageous if positional differences and / or changes of further game or application-relevant information are provided in real time.

It is also advantageous if, by the at least one further AR device, first broadcast data are received, which contain information to find the first AR device in the network, and to connect the AR device with the servers.

It is also advantageous if a third server, which broadcasts broadcast data, is initialized by the first AR device.

If the servers on one and the same AR device (eg on the first device), they can be transferred together during a game / during operation on another AR device on the network, the one device should fail or the user from the Want to retire.

It is advantageous if the target information is retrieved by the at least one further AR device and the target object is displayed based thereon.

It is also advantageous if a container is created, game objects are generated, further objects are generated locally and displayed in the local scene and synchronized with the server.

In a particularly advantageous embodiment it is provided that, if the target in the image optically out of focus of an AR device, for the purpose of invisible the player assigned to the AR device player object this player object assigned by the AR device assigned a location indication, which one for all remaining AR devices corresponds to invisible position in the scene.

It is also provided that when the target in the image optically out of focus of an AR device, the container object receives a position indication assigned on the AR device, which corresponds to an invisible to the AR device position in the scene.

It is also advantageous if, in particular during the recording of the image, the target is brightened by means of a light source.

The invention also includes a program, in particular an app, which, when executed by an electronic processing device, controls the electronic processing device to carry out the method according to the invention as described above.

Also included is a data carrier or signal containing instructions that, when loaded into an electronic processing device, control the processing device to execute the method according to the invention as described above.

The invention also includes an AR device, in particular a smartphone or tablet computer, wearable device, which is programmed to carry out the method according to the invention as described above.

The communication connection between the AR devices can be wired or wireless, in particular via infrastructure WLAN, AdHoc WLAN of the first AR device or a second AR device, Bluetooth, mobile, are handled.

The information and data may be used as records e.g. in XML, JSON, bytes are communicated between devices.

The interaction or control of the shared container object by several AR devices is also possible.

Movement of the target in the real world can also be detected on all devices in real time by the movement of the container associated with the target.

The present invention thus makes it possible to use a target in augmented reality at the same time from several different perspectives on different AR devices. To use means to have the target displayed in an AR scene by the respective AR device, to be able to view and also additionally to be able to interact with this object and possibly other virtual objects in the AR scene by means of virtual objects. It goes without saying that the various AR devices display or display the AR scene from different positions and perspectives. These positions and perspectives correspond to the positions and perspectives of the users associated with the respective AR devices.

For this purpose, according to the invention, a virtual user is set up by a first user, by means of his AR device, to identify the remaining users (called second users), e.g. with techniques of multiplayer programming (e.g., server-client approach). This virtual scene contains a "root object" in which further virtual objects are placed as "child elements". This root object is also called (virtual) "container object" in the following. The virtual scene is imaged by an AR device onto a target defined by the first user and is then, as it were, in the real scene, ie "in" the ("real") reality. This product is referred to herein as an AR or virtual scene. In addition, the first user initializes one or more servers to which additional users can connect to interact in the AR scene.

When a second user connects to the server of the first user, he gets transferred from the server, the information on the current target on his AR device and can thus, without further action on his part, in the (already existing) AR scene "latch "And with the first Users or other users in the AR scene.

This approach is in principle expandable for any number of users.

Thus, the present invention is a technique of using, in the realm of augmented reality, a user-defined and physically existing object (hereafter referred to as "target") as the common origin or reference point for multiple-user applications, if all Users are physically in the same location with their AR devices. In addition, each user can directly control, interact, or manipulate (virtual) objects, and these manipulations and interactions can be perceived in real time on all participating AR devices from different perspectives. Furthermore, the user can control not only the virtual object but also the physical one.

If he has the target e.g. physically moved with the hands, this (real) movement is also visible on the other AR devices.

Unlike other solutions, you are completely free of fixed targets, and the solution can be used completely mobile, even without a communication infrastructure. What is needed is only an object in the real world (as a target) and the app implementing the inventive method for the respective AR devices.

The invention will be described in more detail with reference to embodiments and the drawing. In the drawing shows

1 an overview of the complete structure of the technology exemplary for two users;

2 the step of detecting the target by the first AR device;

3 the step of creating the container;

4 the insertion of player-related virtual objects into the AR scene;

5 the step of connecting to the (target) server and retrieving information therefrom;

6 a diagram showing the components;

7 a flow chart of the individual steps;

8th an example application; and

9 the visual view of the 3D scene on an AR device.

The in the 1 to be recognized surface 10 is merely any surface on which the target (to be defined) is located, eg a table or the floor.

In the first step, using the camera function of the AR device of the first user, an image of a physical object is created, which serves as a target 7 should be defined. For example, an object is selected that is on a table 50 lies. This should - advantageously - stand out as well as possible (eg in contrast, brightness, color, texture, patterning or the like) from its surroundings, so that the physical object is well recognizable automatically. In addition, it should have a smooth or even surface, so that it is easily recognizable from different perspectives. For example, it should not reflect too much or shine or be partially obscured by certain perspectives.

• • Buch/Zeitschrift/Prospekt
• • Geldschein/Kreditkarte
• • Handy-Display (eingeschaltet)
• • Bierdeckel/Zigarettenschachtel
• • eigene Zeichnung/Skizze

Possibilities for (good) targets 7 are for example:

• • Book / Magazine / Brochure
• • Bank note / credit card
• • Mobile phone display (switched on)
• • Beer coaster / cigarette box
• • own drawing / sketch

For the creation of the image, the (first) AR device (eg smartphone or tablet) 1 the first user as parallel as possible to the surface of the target 7 brought so that the optical axis A of the lens of the AR device 1 as perpendicular to the surface of the target 7 is oriented. Then a picture is taken. This illustrates 2 , When the image is recreated for each application (that is, each "game session"), it can be ensured that the current weather, light and environmental conditions are also taken into account. The above-mentioned aspects are optional to observe, so that the target is particularly easy to identify in a camera image. Depending on the quality and capabilities of the camera or image processing software of the AR device, these considerations may not necessarily be respected.

In an application running in the background (advantageously on the first AR device), this image is examined (optionally) for certain particularly salient or striking features and a data record is calculated. Once this has been calculated, by the first AR device of the first user, a server initialized and started and the virtual scene, so a coordinate system for, related to the target. This is now the "container" object 8th for more virtual objects 5 . 6 . 22 , The server can run on the first AR device. The container object is thus assigned to the target in a specific (spatial) relation to the target. It is then also in the virtual scene (so to speak, in the coordinate system mentioned here). The container object provides a virtual "playing field" on which users can move and interact with each other through the player objects they control. The definition of the virtual container is advantageously contained in a program (app) running on each of the AR devices. Thus, at the beginning of a session, it is merely "positioned" at the target by the first Ar device and initialized (started).

• • Ein Spielfeld (z.B. für Brettspiele oder eine virtuelle Arena/Stadion);
• • eine Autorennbahn;
• • eine Bühne;
• • ein Haus, z.B. für Architekten, ein Auto, z.B. für Autohäuser.

3 illustrates this. This container object can represent, for example:

• • A playing field (eg for board games or a virtual arena / stadium);
• • a racecourse;
• • a stage;
• • a house, eg for architects, a car, eg for car dealerships.

In addition, the user can now own, controlled by him virtual object 5 . 22 into the AR scene, cf. 4 , This step is optional or individually customizable, because it can already be enough, only with the virtual container 8th to work or even just display control elements in the display, through which one with the virtual container 8th can interact directly. In addition, it can also be that you bring several own objects into play (eg in a strategy game or a virtual board game) or even one or more objects 5 . 22 which are then used by several users simultaneously or alternately (eg dice in a dice game)

The virtual user object controlled by him can be his play person. This virtual user object, like all other virtual objects also, is also provided with a position specification, the position specification having a geometric relation to the position specification of the container object. The virtual user object is thereby subordinated to the container object. The information indicative of the user object as well as the location indication are also provided for the AR-based display for the relevant AR device and at least one further AR device.

The second user (who is physically in the same place as the first user but not in the same position) now has nothing to do but talk about his AR device 2 to connect to the server, cf. 5 , As a result, he automatically gets information that is descriptive of the target, and provides the location specification of the container object, so that he now the target using the camera function of his AR device 2 can focus. Also on his AR device 2 the virtual container 8th created and displayed and all previous (player) objects) that are already in the virtual scene are displayed. In addition (also again optional) his own (player) object is added to the scene. Optionally, there is a certain position on the field, a so-called "spawn point". This step can (exactly) be carried out by each additional user as long as the application is designed for the corresponding number of users and the limit has not yet been reached. The complete structure then turns as in 1 represents.

• • Möglichkeit 1: Das Bild des Target selbst wird übertragen, so dass es auf dem Client auch wieder berechnet werden kann.
• • Möglichkeit 2: Das fertig berechnete bzw. analysierte Target (also ein Datensatz bzw. das Bild mit Metadaten) wird an den zweiten Spieler (Client) übertragen.

There are several possibilities for transmission: an extra (parallel) server, Bluetooth, WLAN, etc., and also the data to be transmitted differ depending on the AR-Framework used:

• • Option 1: The image of the target itself is transmitted so that it can be calculated again on the client.
• • Option 2: The finished calculated or analyzed target (ie a data set or the image with metadata) is transmitted to the second player (client).

In any case, a record is transferred that contains the information about the target that can be used by the client.

Each player gets on his AR device 2 Apart from the AR view, a GUI (Graphical User Interface) adapted to the respective application is also displayed, with the help of which he can control or interact with his object remotely, eg via buttons, sliders or touch gestures.

A second way with the virtual objects 5 . 6 . 8th . 22 To interact, for example, is to use other physically existing objects / objects in the real world as a "button". In general, all other common interaction techniques with this idea are easily transferable or augmented in augmented reality. Other additional information or common items such as different menus can be used as in any other "normal" application. So you can use all input and output methods or interaction forms that would be used for other normal software. Examples are visual interaction (with menus, or special adaptation for the visually impaired), interaction via sound (input and output), haptic interaction (eg mobile phone vibrate as feedback).

More generally, all techniques used in previous AR applications can be improved to work in multi-user applications as well. And on the other hand, common techniques used in multi-user applications or multi-player games can also be augmented to augment reality.

Another optional feature is that a user on all other AR devices 1 . 2 is hidden (or displayed in a "disconnected" state / pose) if the user focuses the camera on the target 7 should lose. This always gives you a consistent view on all devices. This may be useful for applications (eg "action-heavy" so as not to be distracted by "non-existent" players) and for others rather less (eg strategy games or board games). For this purpose, the complete virtual container (on the local AR device) is "pushed" into an invisible area for all users in the virtual scene, ie hidden, and on the remaining AR devices, only the player becomes an invisible "pool" pushed, so that the other users still see the scene, but without the user who has lost the camera focus, so to speak, has moved away from the virtual scene ... Because the technology for any number of users is extensible, it is also very good suitable, for example, to view a specific object from several perspectives (from multiple viewers) at the same time.

Another advantage is that the target is not bound to the rest of the real scene environment. If the target is e.g. If a book is on a table, then you can move the book freely on the table (real), and the container placed on it moves in the AR scene. You can even take the book to a very different location. The target itself is thus easily transportable, not tied to a location and even mobile / mobile targets, which are thus constantly moving, are possible. This works because the position specification of the container has a geometric relation to the target, that is, related to the target.

Another approach, namely that the players each define their own, different targets or even located in different locations, is also supported by the inventive technique. In this case, you simply let each client create their own target when connecting to the game server.

In poor visibility, when e.g. When the scene is dark or the user casts shadows, a user may optionally turn on a light (e.g., the camera function flash or an external light source) dynamically and aim at the target to really only track the target, not the shadows. This light can be turned on and off dynamically, even in the scene / game itself as needed. For this purpose, the light source of the AR device (for example, the smartphone) can be addressed directly.

The applications are designed for multiplayer. However, you can use the applications that use this technique, even without problems in single-player mode and, for example. play against an AI (Artificial Intelligence System).

Another optional feature of the invention is that one can send sound from the virtual object and associate it with a virtual microphone (called in Unity, for example, AudioListener) to the exact point in reality only by listening. For this purpose, a corresponding audio output system should be connected to the AR device, e.g. a surround headphone.

The technique of the invention may e.g. be developed or implemented as an app or mobile game and is thus platform-independent on many devices usable.

In the following, individual components are described by way of example.

Involved components: (see also Fig. 6)

First AR device ("main" device) 1 : This is the device used by the first user. It can be any (mobile) device, a camera to capture the environment and a display to represent the AR scene 9 owns and has enough power to run AR applications. Also running (software side) on the AR device 1 an "application server" 18 synchronizing all game items, the AR application / app itself and another parallel server 19 with which the information about the target is exchanged. Optionally, there will be another, third server 20 used to advertise the open game (via broadcasts) in a WLAN, and a virtual router to build an ad hoc WLAN (instead of an infrastructure WLAN). However, how this is implemented in detail is up to the developer. In any case, all components can be mapped via software, so that apart from the AR devices, no further hardware components are needed.

Second AR device (s) ("client device (s)") 2 : This AR device can be any (mobile) Device that meets the same requirements as the first AR device. Here, only the AR application / app needs to run, and depending on the architecture, a permanent (TCP) connection needs to be established by a client, or it is only communicating with the server ("stateless via UDP").

Connection: All AR devices 1 . 2 have to be connected to each other. The easiest way is, if they are all in the same WLAN. Since they are physically in the same place, that makes the most sense. If no (infrastructure) WLAN currently exists, you can also set up an ad hoc WLAN using the first AR device. Other connection techniques such as Bluetooth or ZigBee are just as possible.

In the following, some implementation details are described by way of example

In the example implementation the game engine Unity3D and the programming language C # as well as the AR framework EasyAR are taken as basis. However, the invention can also be implemented with other game engines or completely without an engine, in all possible programming languages and with other AR frameworks (such as Vuforia, Metaio or ARToolkit).

It should be noted that these engines are usually constructed so that there is a 3D space (a so-called "scene") in which 3D elements are placed. In the memory these are mapped as (from a computer science point of view) tree with n children. Here, the children "inherit" certain characteristics of the parents. Elements that are positioned directly and in any position in the 3D scene are the first elements of a tree, ie the root elements (referred to here as "root elements"). Objects that are placed as children, and thus in a relative position to their parent object, are called child objects. Each object always has a certain position (= transform (position, rotation, scale)) in space. In addition, it can be equipped by various so-called components with all sorts of other information. These may be in Unity e.g. are written in C # and are therefore arbitrarily complex, ie from a single variable to a complete computer program. In the example implementation, the components are e.g. used to let the user interact with the objects or to have synchronous objects (GameObjects) in all scenes. This will be described below.

The virtual container object according to the invention 8th is considered such a root game object, in which all other objects 5 . 6 . 22 implemented as a child. The child objects 5 . 6 . 22 thus always have the same relative position to / in the container 8th , The container 8th now gets exactly to the target via AR 7 placed in the real world. In each case a virtual camera is used for the view on the virtual scene. In the case of augmented reality, it gets or changes its position and rotation, so to speak, by movements of the real device camera. This task and the merging of the two images is done by the AR-Framework.

Thus, the VR scene is "laid over" the real world so to speak, so that the target in the real world (as long as it is not moving) as a fixed point always exactly in the same place (ie the container) in the virtual World lies. And all points / objects in both "worlds" have exactly the same relative position to the fixed point / target as when creating and placing the container object 8th ,

The various users can now interact with each other through the multiplayer support of Unity. The server takes over the "management" and the game logic and manages the scene and interactions (such as collisions) of individual GameObjects. The client has the exact same 3D scene on his AR device and keeps his AR scene in sync (keeping the view fluid, up to 20-30 times a second, depending on the application) with the server's scene. How this works in detail is known to the person skilled in the art and in the special case Unity3D can be found here ( https://docs.unity3d.com/Manual/UNet.html ). Of course you can also reprogram such a server-client-principle from the ground up in your own implementation or you can use a corresponding technique in the (game) engine of your choice.

Do you now with an AR device 1 . 2 an image (eg in EasyAR TargetOnTheFly, or in Vuforia UserDefinedTarget), this is either directly as image or previously evaluated and, converted as a record, stored at a specific location in the memory of the AR device. The location and also the type of data set vary according to the AR framework.

For EasyAR on an Android device, for example, it is stored in the tmp data in the user context of the app as a jpg.

In addition to this game server, another server is running on the first AR device 1 , In our example, this is implemented in C # using the HTTPListener class from the .NET Framework. This target server searches for the created object and delivers it to the clients (ie the second AR devices 2 ) as a byte array as soon as a new client connects to the game server. In the client will This image or record then converted back and stored exactly as if it had just been created on this device, eg for Android and EasyAR again in the context of the app.

Thus one has the exact same picture (or the same data set) under the same conditions (for example light conditions) and thereby the exact same fixed point in the real world.

On the basis of this image / data set, the container GameObject is now also created on the client in the Unity scene and placed on the target in the real world.

The visual impression of the different perspectives results from the fact that each player (user) moves his (virtual) camera function through which he sees the virtual (and augmented) reality as he moves the (real) camera of his AR device Relation to the target moves. Because every player is now on his AR device 1 . 2 the exact same AR scene sees and the positions are always synchronized with the "original" scene on the server, creates the impression of shared and influenced Augemted Reality or in other words: the same scene is then from each player through the perspective of his Camera (per device adapted to position and rotation to the target in the real world). The respective image is rendered in each case on the user's AR device by Unity.

The example implementation has an extra broadcast server so players can find each other (or the clients find an open game). Again, this is programmed in C # (as a customized version of the NetworkDiscovery class from the Unity API) and will be launched as soon as a game is open.

The following functional steps are in addition to the text described here also in the flowchart of 7 to see. There is also the correlation between the components drawn.

On the first AR device (server device at the same time) 1 :

Select target:

The user launches the built into the app function to create an image. On this a section is marked on which the container is to be placed later. He now holds the camera parallel to the target, sees that the area on which the container should be placed is as far as possible in the marked area and waits until the camera has focused this point. This is usually done on the software side by all cameras. Once the camera has focused on this point, the user can click on a trigger button and the image is created.

An optional additional feature is the activation of a light source. If the user in the real scene cast a shadow that obscures the target, he can directly connect the integrated into the AR device camera flash as a "flashlight". This is not always favorable from AR aspects, because then the lighting conditions are changed. However, it is often useful in practical use, but then the light should remain switched on throughout the game.

Create image: Now a photo is created and converted into a certain image format (here jpg) and stored in memory.

Evaluate the image and save it as a dataset: The image is examined for distinctive points / features (special contrast / color, etc.). These points / features are stored as a record about the target (the information describing the target) so that they can be used later.

Target server 19 Create: Initializes a server listening for a specific port. For example, here is where and under what name he can find the record with the specific target details (the information describing the target) and how he should provide it (here by byte array HTTP response to a specific request). This can be started here directly, but does not have to be started yet. To save resources, it can only be dynamically started when a new client enters the main game server and wants to retrieve the record.

Game server 18 create:

First of all, elementary information (such as the size of the room, how many user clients can connect to the scene at most), network-specific details (such as which frequency to communicate with or definition of the IP address and ports) and game-specific precautions (eg initialization of various variables and objects and creation of the GUI) set and made.

Later, this server is also responsible for ensuring that all connected user clients have a consistent view of objects in the scene, such as allowing players to be moved and synchronizing the new location with all users. He does this by always sending the delta to the current position via TCP / IP connection to all connected devices, which gives a change of the game objects in real time. Furthermore It also manages possible colliding accesses from multiple clients to the same (shared) virtual object simultaneously. (see below)

Create the virtual container 8th : It becomes a virtual scene 9 created and a virtual container 8th placed in it. For this purpose, an image (later invisible to the user) of the previously recorded image (or the analyzed version of the image) is placed in the scene and then the container 8th posed.

This container 8th is then placed by an optical overlay of the virtual camera, which captures the virtual scene from a certain position and the device camera, which focuses the target in the real world, so that it looks as if the container is on the target /stands. As a result, the entire virtual scene is placed over the real world. From this point on, the position and rotation of the virtual camera is then always synchronized with the position and rotation of the device camera, in each case relative to the container or to the target, which respectively form the common reference point of the scenes. The user thus always sees the section of both "worlds" which is determined by the position and rotation of the device camera and which can be influenced by movements of the AR device (eg smartphone).

Broadcast server 20 and start broadcasting broadcast data: To let other users know under which IP address the server (s) are running / running and how the server (s) can be addressed and addressed over the network. This should be done as a last step so that all preparations for the AR scene and the multiplayer game could be made before the first additional user client enters the scene.

Typically, all three (software) servers run in parallel on different ports or are started dynamically when needed.

On the user client devices, ie the AR devices 2 of the second and the following users:

The following steps are almost exactly the same on the "server" device 1 off, because this AR device 1 at the same time provides a server, but runs the user-related actions as a "client". This AR device 1 of the first user thus simultaneously assumes the function of a server and a client.

However, it is done here right after creating the scene, so this client does not need to receive broadcast data and retrieve the specific target details via "network", but it can simply address it as a client in the same location as the server.

Receipt of broadcast data: The packets sent via UDP broadcast are intercepted by the server. These packages contain the name of the server and its IP address as well as the ports and possible meta-information (eg if the "room" is already filled, ie the maximum client number (which can be defined by the server depending on the area of use) has already been reached. In that case, this room would simply not be displayed to him and he would not be able to connect).

Connection to the game server: Now a (TCP / IP) connection to the server is established on the basis of the received information. A UDP / IP connection would also be possible. In addition, here too a game is set up / created / started and e.g. Initialized various variables and objects, so that later defined in the game that this user, for example, a client and not a server. He has fewer rights as a result; e.g. he can only influence his objects and not those of other players. This is also called authority over an object. In addition, e.g. Also, a customized GUI (e.g., in color or by individual inputs such as displaying the name) is created and displayed.

Automatically retrieve the specific target details (information that is descriptive of the target) from the TargetServer ( 19 ): This is a function on the game server ( 18 ) (eg via remote procedure call), which opens the target server on a specific port. Now this can be addressed with normal HTTP methods (here by GET request). The target server now delivers the saved target data record. Depending on the server or protocol used, it can be transmitted as a byte array, JSON file, XML, JPG, etc. In the example implementation, it is transmitted as a byte array and then this is converted back to a jpg image file on reception on the client and stored in the memory.

The example implementation uses WLAN / Wifi for wireless communication. Other variants of the technique could use the byte array (or the record used for transmission) but also e.g. via Bluetooth or other (wireless) channels.

Create the virtual container ( 8th ): This works exactly as it was done on the server device.

Place the personal GameObjects (hierarchically assigned per user) in the container 8th : As child objects, the personal GameObjects are now placed in the scene and told the server that from now on they should be synchronized with those in the local scene. In addition, objects already existing (by other users on the server) are created locally and synchronized with the location on the server. In the server scene, the shared objects are additionally placed. Often it is also useful to highlight the personal objects especially (eg color stand out or a player marker above the player, etc.).

Interaction and race conditions (or conflicting access):

If a user wants to move his GameObject (ie the virtual object he controls), he moves his GameObject into his local AR scene, which is a mirrored / synchronized version of the AR scene on the server. As a result, he gets interactions that he himself executes, displayed directly in the scene without delay. Now all changes that are "multiplayer-moderately" relevant (eg position, rotation, scaling, type of representation (eg changed color), game-related information (eg lost life, scored goal, shot fired, purchased item) - ie everything that is also relevant for the other users as well as the management of the game logic), this GameObjects transferred to the server, which in turn synchronizes the new position (and all other data to be synchronized) with all other users. This happens so fast that for the users with "human comprehension" no difference of the scenes on the individual devices would be recognizable. In the example implementation e.g. 1/30 s + WLAN latency = max. 7 ms.

However, if the user wants to interact with an object shared with other users, he must first have the server authorize the action. After the action, it is then withdrawn. This ensures that multiple users can not interact with the same object at the same time.

Another feature implemented in the sample implementation is the ability to show on all clients when a particular user loses focus on the target. That he either moves the camera away or the camera image is blurred at short notice, which can happen again and again. To avoid here, for example In games to give the "still present" users an unfair advantage, the missing user is "hidden" so that the others can not interact with him anyway or may be distracted by him. For this purpose, a new root object is created in virtual space at a far-away (and hence "invisible") point by the AR device of the associated user and the associated user object is placed therein as a child. As a result, this child inherits the distant location of the new root object and the associated user is no longer visible to any other user, but is still in the scene so that the associated user can still interact, e.g. he can call a user-related menu.

Of course, there is also the possibility to inform the user only e.g. to take a certain "I am offline" pose, or take it e.g. with a "disconnected" icon or similar (hovering over the head, for example). This is left to the respective developer.

Instead of using an existing infrastructure WLAN, it is also possible to create an ad hoc network from the server device and to let all clients connect to it. Then one is independent of the physical location, since the complete game or the communication between each other and the AR-creation etc. completely without infrastructure and also without Internet access works.

You can play a (multiplayer) game thus also in places, where one does not have a mobile Internet and infrastructure WLAN, which does not work with other solutions, since one always there either a predefined target on a server (on the Internet) has or interacts with elements over one.

Thus, mobile AR devices (e.g., smartphones) can be used completely independently of any infrastructure, e.g. everywhere in the great outdoors. Together with the ability to choose a target freely and for each game session, there is a greater freedom in the selection of the game environment and thus an extended experience.

Below is a brief example of an application based on a simple game. She is referring to the 8th and 9 described.

Two users (players) with AR device 1 respectively. 2 sit opposite each other. Between them is a table with a book on it. player 1 Now uses the technique described to create a virtual playing field (container object 8th ) on the book (Target 7 ) and places his game character (User Object 21 ) on the field. player 2 now connects with players 1 and thereby also places his game character (User Object 22 ) on the field. Now every player can his character with buttons on his smartphone (AR device 1 respectively. 2 ) Taxes.

Subsequently, a game can be started in which both players mutually eg a (virtual) ball (virtual object 22 ) to play. The virtual objects are shown in the figure as dotted lines. Like the players or any other virtual object, the ball is a 3D model. Like the players, it is already in the app in this form and only needs to be initialized. It is also possible to load the associated data record eg from a server or to exchange it between the individual devices

On his AR device, each player would have the following view (cf. 9 ) With the outer two buttons, he can then each move his character to the right or to the left and shoot with the middle the ball on the opponent's goal.

Another example is: An architect shows his customers his designed house. He places it on the "playing surface" (land) and then all customers can see the house from their perspective. The architect can now, for example, alternately mark different parts of his house, which are also visible on all customer equipment, in order to see the e.g. to draw attention to a specific part of the house. Or he can dynamically show or hide certain elements (roof, furniture, etc.) of the house.

Other examples are virtual board games; a virtual racecourse; a virtual band playing on a virtual stage; a product presentation (e.g., a new toy or a car); a piece of furniture should be checked before installation to see if it also fits into the real environment (and several people can look at it at the same time and thus have a better basis for discussion); a virtual theater; many other applications are conceivable.

LIST OF REFERENCE NUMBERS

1

AR device of user 1

2

AR device of user 2

3

Visibility range of users 1

4

Visibility range of users 2

5

Virtual object of users 1

6

Virtual object of users 2

7

 Target

8th

Virtual container (container object)

9

AR space

10

surface

11A

1. Connect to the server

11B

2. Get information about the goal

12

Connection (e.g., WLAN)

13

"Main" device

14

"Client" device

15

Display / Screen

16

camera

17

AR application / app

18

Application / game server

19

Server for exchanging the target information / target server

20

Broadcast server

21

Application / Game Client

22

shared virtual object (here a ball)

23

Left button

24

Shoot button

25

Right button

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited non-patent literature

• https://docs.unity3d.com/Manual/UNet.html [0093]

Claims (17)

A computer-implemented method for presenting augmented reality based on images of a real three-dimensional scene, wherein, by a first AR device ( 1 ), on the basis of an image, a section of the image as a target ( 7 ) and from the section of the image information is derived, which for the target ( 7 ) are descriptive, and wherein at least one virtual three-dimensional container object ( 8th ) is initialized and provided with a position specification, wherein the position specification has a geometric relation to the target ( 7 ) and where for the target ( 7 ) descriptive information, information that is indicative of the at least one virtual container object, and the location indication for AR-moderate display of the container object ( 8th ) by the first AR device ( 1 ) and, via a communication link, for the AR-like display of the container object ( 8th ) for at least a second AR device ( 2 ) provided. A computer-implemented method for presenting augmented reality based on images of a real three-dimensional scene, wherein, by a second AR device ( 2 ), Information for a target ( 7 ) are descriptive and information indicative of at least one virtual container object provided with a location indication, the location indication being a geometric relation to the target ( 7 ) are received via a communication link coming from a first AR device and, based on the received information, the target in pictures taken by the second AR device ( 2 ) and the virtual container object ( 8th ) according to the relation AR-moderately. Method according to claim 1 or 2, wherein, by an AR device ( 1 . 2 ), at least one virtual three-dimensional user object ( 5 . 6 ) which is a description of how the article is represented by an AR device ( 1 . 2 ), and provided with a position specification, wherein the position statement has a geometric relation to the position specification of the container object ( 8th ), wherein the at least one three-dimensional user object ( 5 . 6 ) through the AR device ( 1 ) and information indicative of the at least one virtual user object and the location indication for AR-type display by the AR device ( 1 . 2 ) and to display the user object ( 5 . 6 ) for at least one further AR device ( 1 . 2 ), preferably each in real time. Method according to one of the preceding claims, wherein the container object ( 8th ) is provided with a root property of a predetermined hierarchical relation and at least one virtual object ( 5 . 6 ) is provided with the predetermined hierarchical relation to another virtual object, wherein the root is arranged at highest hierarchical level, such that the virtual objects provided with a hierarchical relation together with the root object represent a hierarchical tree structure, so that during a movement a virtual object of each object is moved, which is arranged in the hierarchical structure from the root behind the moving object. Method according to one of the preceding claims, wherein the AR representation is obtained by optical superimposition of the virtual objects represented by a virtual camera function and the image captured by a real camera function image of the scene, wherein the virtual camera function of the real camera function in terms of focal length and perspective. Method according to one of the preceding claims, wherein application-relevant information of each virtual object in the scene can be influenced by means of at least one AR device and, preferably in real time, provided for further devices. Method according to one of the preceding claims, wherein, by the first AR device, a second server ( 19 ) initializing the information describing the target, the information indicative of the at least one virtual container object, and the location indication for AR-like display of the container object for at least one AR device for retrieval by at least one AR device running on the first AR device. The method of claim 1, wherein, by the first AR device, a first server is defined that defines for how many AR devices the scene is retrievably held, provides network information as well as information descriptive of a game, for which the objects are provided, providing information for synchronizing the positions and / or other application-relevant information of all the objects of the scene for retrieval by at least one AR device, and running on the first AR device. Method according to one of the preceding claims, wherein, by the first AR device, a third server ( 20 ), which broadcasts broadcast data containing information to find the first AR device in the network and to connect the AR device to the servers initially received by the at least one other AR device. Method according to one of the preceding claims, wherein, when the target in the image optically out of focus of an AR device, for the purpose of invisibility of the player device assigned to the AR device by the AR device assigned a location indication, which one for all AR devices invisible position in the scene. Method according to one of the preceding claims, wherein, when the target in the image visually gets out of focus of an AR device, and wherein, on the AR device, the container object is assigned a position indication that is an invisible position for the AR device in the scene corresponds. Method according to one of the preceding claims, wherein, in particular during the recording of the image, the target is lightened by means of a light source. A program, in particular an app, which, when executed by an electronic processing device, controls the electronic processing device to carry out the method according to one of the preceding claims. Method according to one of the preceding claims, wherein the communication connection is wired or wireless, in particular via infrastructure WLAN, AdHoc WLAN of the first AR device or a second AR device, Bluetooth, mobile phone is handled. Method according to one of the preceding claims, wherein a physical change of the target affects the representation of the virtual objects in the AR scene on all AR devices, preferably in real time. A program, in particular an app, data carrier or signal, containing instructions which, when executed by an electronic processing device, controls the electronic processing device to carry out the method according to one of the preceding claims. AR device, in particular smartphone, tablet computer or wearable device, which is programmed to carry out the method according to one of claims 1 to 15.

Images