US11762453B2 - Method performed by a computer system for creation of augmented reality experiences and connection of these to the real world - Google Patents

Method performed by a computer system for creation of augmented reality experiences and connection of these to the real world Download PDF

Info

Publication number
US11762453B2
US11762453B2 US17/399,389 US202117399389A US11762453B2 US 11762453 B2 US11762453 B2 US 11762453B2 US 202117399389 A US202117399389 A US 202117399389A US 11762453 B2 US11762453 B2 US 11762453B2
Authority
US
United States
Prior art keywords
data
nearby object
nearby
computer system
physical position
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US17/399,389
Other versions
US20220050519A1 (en
Inventor
Morgan Fredriksson
Jürgen Königsmann
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nagoon AB
Original Assignee
Nagoon AB
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nagoon AB filed Critical Nagoon AB
Publication of US20220050519A1 publication Critical patent/US20220050519A1/en
Application granted granted Critical
Publication of US11762453B2 publication Critical patent/US11762453B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/29—Geographical information databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • G06F16/44—Browsing; Visualisation therefor
    • G06F16/444—Spatial browsing, e.g. 2D maps, 3D or virtual spaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations

Definitions

  • the present invention relates to a method performed by a computer system for creation of augmented reality experiences and connection of these to the real world.
  • Augmented reality is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information.
  • AR can be defined as a system that fulfils three basic features: a combination of real and virtual worlds, real-time interaction, and accurate 3D registration of virtual and real objects.
  • the overlaid sensory information can be constructive (i.e. additive to the natural environment), or destructive (i.e. masking of the natural environment).
  • Augmented reality is used to enhance natural environments or situations and offer perceptually enriched experiences.
  • advanced AR technologies e.g. adding computer vision, incorporating AR cameras into smartphone applications and object recognition
  • the information about the surrounding real world of the user becomes interactive and digitally manipulated. Information about the environment and its objects is overlaid on the real world.
  • AR augmented reality
  • the feedback loops retrieving and transmitting data between different parts of the computer system handling the AR experience are however not very well developed. If the computer system could handle augmented reality experiences and connect these to the real world, by using more advanced feedback loops retrieving and transmitting data between different parts of the computer system, it would be highly valuable.
  • An objective of the present invention is thus to accomplish a method performed by a computer system for creation of augmented reality experiences and connection of these to the real world, that generates a value-added description of the surrounding environment, that provides an efficient interaction between different users and the surrounding environment, and that manages the behaviour of different users or objects present in the surrounding environment.
  • the invention concerns a method performed by a computer system for creation of augmented reality experiences and connection of these to the real world.
  • the computer system is arranged to communicate with a first database.
  • the method comprising the steps of: A. receiving descriptive data of a first object from a first client, B. receiving a physical position of a positioning device being external to and in close proximity to or internal the first object, C. retrieving from the first database a data set of a first nearby object having a physical position being related to the physical position of the positioning device, D. creating a description of the surrounding environment by combining: the retrieved data set of the first nearby object with the descriptive data of the first object, the physical position of the positioning device, and the physical position of the first nearby object, E.
  • An advantage of the solution is that it handles and improves augmented reality experiences and connects these to the real world. Furthermore, it specifically generates in an intelligent manner a value-added description of the surrounding environment based upon retrieved and processed information/data.
  • the method above may be configured according to different optional embodiments.
  • the method may further comprise the steps of: A. retrieving a first instruction from the optimised sub data, and B. sending the first instruction to the first client for triggering an activity of the first object, or C. sending the first instruction to a second client for triggering an activity of a second object, wherein the second object being related to the first object, or D. sending the first instruction to a third client for triggering an activity of the first nearby object, or E. sending the first instruction to a fourth client for triggering an activity of a second nearby object, wherein the second nearby object being related to the first nearby object.
  • An advantage of the solution is that it provides the possibility to send specific instructions to the first or the second object or to the first or the second nearby object.
  • the first or the second object being e.g. a human/humans/a vehicle/vehicles using the system
  • the first or the second nearby object being e.g. a building or any other object found in the surrounding environment.
  • it provides the possibility to trigger corresponding activities of the first or the second object or of the first or the second nearby object.
  • interesting interactions between the system and e.g. real humans/vehicles using it and/or nearby objects found in the real world could occur. These interactions are based upon the provided value-added descriptions of the surrounding environment.
  • the method may further comprise the step of illustrating the final description of the surrounding environment on a user interface of the first client.
  • An advantage of the solution is that it generates a value-added description of the surrounding environment and specifically depicts it on a user interface.
  • the method may further comprise receiving a first behavioural data set of the first object from the first client by means of a first behavioural device being external to and in close proximity to or internal the first object.
  • the method may further comprise receiving a second behavioural data set of the second object from the second client by means of a second behavioural device being external to and in close proximity to or internal the second object.
  • a behavioural device could collect information/data about for instance the real world activities of a first and a second object being e.g. a human/humans or a vehicle/vehicles using the system.
  • the method may further comprise the steps of: A. retrieving a second instruction from the optimised sub data as a response of the received first and second behavioural data set of the first and the second object, and B. sending the second instruction to the first client for triggering an activity of the first object, or C. sending the second instruction to the second client for triggering an activity of the second object, wherein the second object being related to the first object, or D. sending the second instruction to the third client for triggering an activity of the first nearby object, or E. sending the second instruction to the fourth client for triggering an activity of a second nearby object, wherein the second nearby object being related to the first nearby object.
  • An advantage of the solution is that it provides the possibility to send specific instructions to the first or the second object or to the first or the second nearby object.
  • the first or the second object being e.g. a human/humans/a vehicle/vehicles using the system
  • the first or the second nearby object being e.g. a building or any other object found in the surrounding environment.
  • These instructions are partly being based upon information/data collected by the first and the second behavioural device about for instance the real-world activities of the first and the second object.
  • it provides the possibility to trigger corresponding activities of the first or the second object or of the first or the second nearby object. In this manner even more interesting interactions between the system and e.g. real humans/vehicles using it and/or nearby objects found in the real world could occur. These interactions are also based upon the provided value-added descriptions of the surrounding environment.
  • the method may further comprise the step of receiving additional data of the first and the second object by means of an external device.
  • An advantage of the solution is that it provides additional possibilities to gather information of the real-world activities of the first and the second object. It could specifically generate in an intelligent manner an ever more value-added description of the surrounding environment based upon retrieved and processed information/data. It could also even better illustrate the final description of the surrounding environment on a user interface. It could also even better provide the possibility to send specific instructions to the first or the second object or to the first or the second nearby object.
  • the first or the second object being e.g. a human/humans/a vehicle/vehicles using the system
  • the first or the second nearby object being e.g. a building or any other object found in the surrounding environment.
  • These instructions are partly being based upon information/data collected by the external device and possibly also the first and the second behavioural device about for instance the real-world activities of the first and the second object. Furthermore, it provides the possibility to trigger corresponding activities of the first or the second object or of the first or the second nearby object. In this manner even more interesting interactions between the system and e.g. real humans/vehicles using it and/or nearby objects found in the real world could occur. These interactions are also based upon the provided value-added descriptions of the surrounding environment.
  • the retrieving a data set step may comprise retrieving a first data set of the first and the second nearby object from the first database based upon the physical position of the positioning device, and retrieving a second data set of the first and the second nearby object from the second database based upon the physical position of the positioning device.
  • An advantage of the solution is that several independent databases, also externally found, could be used to provide data/information of the first and the second nearby object being e.g. buildings or any other objects found in the surrounding environment
  • the deleting redundant sub data step is conducted by using an optimisation algorithm.
  • An advantage of the solution is that it generates in an even more efficient manner an intelligent and value-added description of the surrounding environment based upon retrieved and processed information/data.
  • the invention concerns a computer system for creation of augmented reality experiences and connection of these to the real world.
  • the computer system is arranged to communicate with a first database and is arranged to: A. receive descriptive data of a first object from a first client, B. receive a physical position of a positioning device being external to and in close proximity to or internal the first object, C. retrieve from the first database a data set of a first nearby object having a physical position being related to the physical position of the positioning device, D. create a description of the surrounding environment by combining: the retrieved data set of the first nearby object with the descriptive data of the first object, the physical position of the positioning device, and the physical position of the first nearby object, E.
  • An advantage of the solution is that the computer system handles and improves augmented reality experiences and connects these to the real world. Furthermore, it specifically generates in an intelligent manner a value-added description of the surrounding environment based upon retrieved and processed information/data.
  • the invention concerns a computer program, comprising computer readable code means, which when run on a computer system causes the computer system to perform the corresponding method.
  • the invention concerns a computer program product, comprising a computer readable medium and a computer program, wherein the computer program is stored on the computer readable medium.
  • the physical positions could be positions outdoors or indoors.
  • An activity of a first and a second object could for instance be two humans being triggered to look at a specific part of a building for an interesting architectural item.
  • An activity of a first and a second nearby object could for instance be these buildings providing specific information, on e.g. digital billboards, to two specific humans being in close proximity to these buildings.
  • the description of the surrounding environment is a conceptual and dynamic model, which provides information of the interrelations between all nearby objects.
  • the user interface could for instance, among others, be a monitor delivering an image/video or a loudspeaker delivering a voice message.
  • the first and the second object could for instance, among others, be a person or a vehicle.
  • the first and the second nearby object could for instance, among others, be a house, a bridge, a road, or a human being.
  • the behavioural device could for instance, among others, be a motion sensor or a VR headset.
  • the external device could for instance, among others, be a video camera.
  • the optimisation algorithm could for instance be a geometric optimisation model, among others.
  • a collision mesh method is an example of a geometric optimisation model which could be used as an optimisation algorithm in this invention.
  • the descriptive data of an object can, among others, be metadata with regards to the object as well as data describing a feature of the object. Metadata is data that provides information about other data.
  • a feature of the object can e.g. be a feature that can be perceived by another physical being or object.
  • the first and the second databases mentioned may be external or internal of the computer system.
  • That the physical position of a nearby object having an association with the physical position of a positioning device, may imply that the positions are physically close to each other. However, other associations may also occur.
  • the user of the computer system being a human may be equivalent to the first object.
  • the user being a human could also be managing the first object, wherein the first object could be e.g. a vehicle.
  • the deleting of redundant sub data of the retrieved data set of the first nearby object could for instance imply that all data of the parts of e.g. a building, being the first nearby object, which the first object, for instance a human being, does not visually see in the real world is deleted.
  • This data to be deleted could for instance be all data representing the back of the building and the internal parts of the building, leaving only the data representing the building which is visually seen by the human being.
  • Receiving descriptive data of a first object from a first client may imply that a description of a human being is sent from the smartphone belonging to the human, and received by the system.
  • Receiving a physical position of the positioning device being external to and in close proximity to or internal the first object may imply that the XYX coordinates of the smartphone are collected by a GPS device being embedded in the smartphone and further sent to the system by the smartphone.
  • the human being the holder of the smartphone.
  • the XYZ coordinates of the smartphone further being received by the system.
  • Retrieving from the first database a data set of a first nearby object having a physical position being related to the physical position of the positioning device may imply that a data set is retrieved comprising technical characteristics of a building, defined as the nearby object.
  • the building having XYZ coordinates rather close to the XYZ coordinates of the smartphone of the human being.
  • D. Creating a description of the surrounding environment by combining: the retrieved data set of the first nearby object with the descriptive data of the first object, the physical position of the positioning device, and the physical position of the first nearby object, may imply that a thorough description of the surrounding environment is compiled by the technical characteristics of the building, the description of the human being, the XYZ coordinates of the smartphone i.e.
  • E. Deleting redundant sub data of the retrieved data set of the first nearby object using an optimisation algorithm creating an optimised sub data, may imply that a collision mesh method is used to decrease the number om data points within the data set needed to illustrate the building.
  • F. Retrieving saliency data being related to the first nearby object from the first database may imply that the system receives data of how large portion the building represents of the total line-of-sight-view for the human being and how much more the building should be highlighted in addition for the user, based upon e.g. colour of the building, height of the building etc.
  • Saliency data may be characteristic data of the first nearby object.
  • Saliency data may be descriptive data of the first nearby object.
  • Saliency data may be objective data of the first nearby object.
  • Saliency data may comprise the colour of the first nearby object.
  • Saliency data may comprise the width of the first nearby object.
  • Saliency data may comprise the height of the first nearby object, among other data descriptive of the first nearby object.
  • G. Calculating a stand-out ratio for the first nearby object based on the saliency data may imply that the stand-out ratio for the building, i.e. how much the building relatively should stand out in an overall illustration of the surrounding environment, is calculated based upon of how large portion the building represents of the total line-of-sight-view for the human being and how much more the building should be highlighted in addition for the user, based upon e.g.
  • Creating a final description of the surrounding environment based on the descriptive data of the first object, the physical position of the positioning device, the optimised sub data, the stand-out ratio for the first nearby object, and the physical position of the first nearby object may imply that an illustration of the surrounding environment is created based upon the description of the human being, the XYZ coordinates of the smartphone held by the human being, the optimised number of data points for illustrating the building, and the XYZ coordinates of the building.
  • the general conditions of the method are the following:
  • the user/object does not need to be real; Mixing real objects, found in the real world, and non-real objects, in the software model is feasible;
  • the clients can also serve as servers, i.e. manage other clients and create their own environments.
  • the clients/servers may communicate with each other;
  • the system can run in off-line mode without own direct access to mobile data by transferring data instead via wifi, bluetooth etc. This could be a preferable solution during a crisis, in the mining industry, and in non-connected systems such as in the defence sector etc.;
  • the GPU on the graphics card or elsewhere, may work with a lot of the real-time calculations.
  • the method in the software solution may be described by the following six steps: 1.
  • the server/computer system receives any type of position of an object, an individual or a physical object, e.g. a car, a robot, from the positioning device for instance via the client.
  • This position could be provided by for instance a GPS, beacon: indoor/outdoor, or wifi technology, etc.
  • other types of external devices such as cameras for instance, could be used to receive/extract data around and with regards to the object, and subsequently transmit the information to the computer system. This can also be conducted in a virtual reality setting.
  • the server/computer system downloads basic data around the object with regards to the given position.
  • This data is for example sourced from:—databases such as Esri (GIS), Streetmap (GIS), Bisnode (business data) etc. or from—local data sources, e.g. point clouds—a large number of coordinates in 3D), or from—Building Information Modeling (BIM) data (3D data).
  • This step implies population of data. Additional data, that is applicable to the nearby objects, is thus sourced from databases and added and to the nearby objects or to the object itself. This resembles of how layers are created in CAD environments. A data model has now been created. Remark: Step 2 may be conducted before step 1 , and/or the data in step 2 can already be found on the server/computer system. 3.
  • FIG. 8 illustrates how large percentage specific objects (A-D) would represent in a line-of-sight view for an object or user at a specific geographical position.
  • A-D large percentage specific objects
  • C should be made salient for the object or end user, as it is a high-rise building. This even though C only represents 12% of the total line-of-sight. 5. Track an individual's/an object's behaviour in the environment, e.g.: —how fast an object or individual moves (fast paced running, slow walking).
  • the system may in this step start to interact with the object or user. Now, the object or user is suddenly a part of the environment/computer model. I.e. the user/object in the virtual environment is consistent with the user/object in the real environment. The system can for instance start influencing the person or object to look at or approach different specific nearby objects.
  • FIG. 1 is a flow chart of a procedure according to an embodiment.
  • FIG. 2 is a flow chart of a procedure according to an embodiment.
  • FIG. 3 is a flow chart of a procedure according to an embodiment.
  • FIG. 4 is a flow chart of a procedure according to an embodiment.
  • FIG. 5 is a flow chart of a procedure according to an embodiment.
  • FIG. 6 is a block diagram illustrating a computer system, according to a possible embodiment.
  • FIG. 7 is a block diagram illustrating a computer system in connection with other parts, according to a possible embodiment.
  • FIG. 8 is a block diagram illustrating a line-of-sight model, according to a possible embodiment.
  • FIG. 1 shows a flow chart illustrating a procedure in a computer system.
  • the various actions may come in different orders than presented in this description, or in a different order than shown in this or other flowcharts related to this description, or some steps may be performed in parallel.
  • a step S 100 descriptive data 3 of a first object 4 is received from a first client 5 at the computer system 1 .
  • a physical position 6 a of a positioning device 7 being external to and in close proximity to or internal the first object 4 , is received at the computer system 1 .
  • a data set 8 of a first nearby object 9 having a physical position 6 b being related to the physical position 6 a of the positioning device 7 is retrieved from the first database 2 a .
  • a description of the surrounding environment 11 is created at the computer system 1 by combining: the retrieved data set 8 of the first nearby object 9 with the descriptive data 3 of the first object 4 , the physical position 6 a of the positioning device 7 , and the physical position 6 b of the first nearby object 9 .
  • redundant sub data 8 a of the retrieved data set 8 of the first nearby object 9 is deleted creating an optimised sub data 8 b .
  • saliency data 13 being related to the first nearby object 9 is retrieved from the first database 2 a .
  • a stand-out ratio 14 for the first nearby object 9 is calculated based on the saliency data 13 .
  • a final description of the surrounding environment 15 is created based on the descriptive data 3 of the first object 4 , the physical position 6 a of the positioning device 7 , the optimised sub data 8 b , the stand-out ratio 14 for the first nearby object 9 , and the physical position 6 b of the first nearby object 9 .
  • FIG. 2 shows a flow chart illustrating a procedure in a computer system.
  • the various actions may come in different orders than presented in this description, or in a different order than shown in this or other flowcharts related to this description, or some steps may be performed in parallel.
  • a first instruction 16 is retrieved from the optimised sub data 8 b .
  • the first instruction 16 is sent to the first client 5 for triggering an activity 17 of the first object 4 .
  • the first instruction 16 is sent to a second client 18 for triggering an activity 19 of a second object 20 , wherein the second object 20 being related to the first object 4 .
  • the first instruction 16 is sent to a third client 21 for triggering an activity 22 of the first nearby object 9 .
  • the first instruction 16 is sent to a fourth client 23 for triggering an activity 24 of a second nearby object 25 , wherein the second nearby object 25 being related to the first nearby object 9 .
  • FIG. 3 shows a flow chart illustrating a procedure in a computer system.
  • the various actions may come in different orders than presented in this description, or in a different order than shown in this or other flowcharts related to this description, or some steps may be performed in parallel.
  • a first behavioural data set 27 a of the first object 4 is received from the first client 5 by means of a first behavioural device 28 a being external to and in close proximity to or internal the first object 4
  • a second behavioural data set 27 b of the second object 20 is received from the second client 18 by means of a second behavioural device 28 b being external to and in close proximity to or internal the second object 20 .
  • FIG. 4 shows a flow chart illustrating a procedure in a computer system.
  • the various actions may come in different orders than presented in this description, or in a different order than shown in this or other flowcharts related to this description, or some steps may be performed in parallel.
  • a second instruction 29 from the optimised sub data 8 b is retrieved as a response of the received first and second behavioural data set 27 a , 27 b of the first and the second object 4 , 20 .
  • the second instruction 29 is sent to the first client 5 for triggering an activity 17 of the first object 4 .
  • the second instruction 29 is sent to the second client 18 for triggering an activity 19 of the second object 20 , wherein the second object 20 being related to the first object 4 .
  • the second instruction 29 is sent to the third client 21 for triggering an activity 22 of the first nearby object 9 .
  • the second instruction 29 is sent to the fourth client 23 for triggering an activity 22 of a second nearby object 25 , wherein the second nearby object 25 being related to the first nearby object 9 .
  • FIG. 5 shows a flow chart illustrating a procedure in a computer system.
  • the various actions may come in different orders than presented in this description, or in a different order than shown in this or other flowcharts related to this description, or some steps may be performed in parallel.
  • a step 5 . 1 descriptive data 3 of a first object 4 is received from a first client 5 at the computer system 1 .
  • a physical position 6 a of a positioning device 7 being external to and in close proximity to or internal the first object 4 , is received at the computer system 1 .
  • data set 8 of a first nearby object 9 having a physical position 6 b being related to the physical position 6 a of the positioning device 7 is retrieved from the first database 2 a .
  • a description of the surrounding environment 11 is created at the computer system 1 by combining: the retrieved data set 8 of the first nearby object 9 with the descriptive data 3 of the first object 4 , the physical position 6 a of the positioning device 7 , and the physical position 6 b of the first nearby object 9 .
  • redundant sub data 8 a of the retrieved data set 8 of the first nearby object 9 is deleted creating an optimised sub data 8 b .
  • saliency data 13 being related to the first nearby object 9 is retrieved from the first database 2 a .
  • a stand-out ratio 14 for the first nearby object 9 is calculated based on the saliency data 13 .
  • a step 5 is
  • a final description of the surrounding environment 15 is created based on the descriptive data 3 of the first object 4 , the physical position 6 a of the positioning device 7 , the optimised sub data 8 b , the stand-out ratio 14 for the first nearby object 9 , and the physical position 6 b of the first nearby object 9 .
  • FIG. 6 shows a computer 1 , including components such as a processor 1 . a , a user interface 1 . b , a memory 1 . c and communication gateways 1 . d.
  • FIG. 7 shows a user interface 5 . a of a first client 5 used by a first object 4 , a positioning device 7 , a first database 2 a and a first nearby object 9 .
  • the first client 5 , the positioning device 7 , the first database 2 a and the first nearby object 9 being connected to a computer 1 .
  • FIG. 8 shows how a line-of-sight of a first object, in this case a human being, is modelled in general.
  • the first, second, third and fourth nearby objects being buildings.
  • the saliency data retrieved is used to calculate a stand-out ratio for each building.
  • building C will have a stand-out ratio higher than its relative size, as it's a high-rise building compared to the other buildings being of normal height.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • Computer Graphics (AREA)
  • Remote Sensing (AREA)
  • Computational Linguistics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Processing Or Creating Images (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Computer And Data Communications (AREA)

Abstract

Method performed by a computer system (1). The computer system (1) is arranged to communicate with a first database (2a). The method comprising the steps of: A. Receiving descriptive data (3) of a first object (4) from a first client (5). B. Receiving a physical position (6a) of a positioning device (7). C. Retrieving from the first database (2a) a data set (8) of a first nearby object (9) having a physical position (6b). D. Creating a description of the surrounding environment (11). E. Deleting redundant sub data (8a) of the retrieved data set (8) creating an optimised sub data (8b). F. Retrieving saliency data (13) being related to the first nearby object (9) from the first database (2a). G. Calculating a stand-out ratio (14) for the first nearby object (9) based on the saliency data (13). H. Creating a final description of the surrounding environment (15).

Description

RELATED APPLICATIONS
This application claims priority under 35 U.S.C. § 119(a) to Swedish Patent Application No. 2030252-7 filed Aug. 13, 2020, which application is incorporated herein by reference.
TECHNICAL FIELD
The present invention relates to a method performed by a computer system for creation of augmented reality experiences and connection of these to the real world.
BACKGROUND
Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information. AR can be defined as a system that fulfils three basic features: a combination of real and virtual worlds, real-time interaction, and accurate 3D registration of virtual and real objects. The overlaid sensory information can be constructive (i.e. additive to the natural environment), or destructive (i.e. masking of the natural environment). Augmented reality is used to enhance natural environments or situations and offer perceptually enriched experiences. With the help of advanced AR technologies (e.g. adding computer vision, incorporating AR cameras into smartphone applications and object recognition) the information about the surrounding real world of the user becomes interactive and digitally manipulated. Information about the environment and its objects is overlaid on the real world.
Computer systems for handling augmented reality (AR) have in the past been introduced on the market. These systems comprise different types of hardware devices interconnected to each other. There are many organisations trying to enhance these systems to provide improved value-added services.
The feedback loops retrieving and transmitting data between different parts of the computer system handling the AR experience are however not very well developed. If the computer system could handle augmented reality experiences and connect these to the real world, by using more advanced feedback loops retrieving and transmitting data between different parts of the computer system, it would be highly valuable.
It would be highly beneficial if data could be handled in a certain manner, that would make it possible to generate a value-added description of the surrounding environment. This would provide the end user with an enhanced and intelligent view of the environment.
Furthermore, it would also be beneficial if instructions could be managed and sent to different parts of the computer system so that an efficient interaction between different users and the surrounding environment could become viable.
Finally, it would be advantageous if the behaviour of different users or objects present in the surrounding environment could be registered and processed for providing a value-added service.
It is therefore desirable to accomplish a system, for creation of augmented reality experiences and connection of these to the real world, that generates a value-added description of the surrounding environment, that provides an efficient interaction between different users and the surrounding environment, and that manages the behaviour of different users or objects present in the surrounding environment.
SUMMARY OF INVENTION
An objective of the present invention is thus to accomplish a method performed by a computer system for creation of augmented reality experiences and connection of these to the real world, that generates a value-added description of the surrounding environment, that provides an efficient interaction between different users and the surrounding environment, and that manages the behaviour of different users or objects present in the surrounding environment.
According to one aspect, the invention concerns a method performed by a computer system for creation of augmented reality experiences and connection of these to the real world. The computer system is arranged to communicate with a first database. The method comprising the steps of: A. receiving descriptive data of a first object from a first client, B. receiving a physical position of a positioning device being external to and in close proximity to or internal the first object, C. retrieving from the first database a data set of a first nearby object having a physical position being related to the physical position of the positioning device, D. creating a description of the surrounding environment by combining: the retrieved data set of the first nearby object with the descriptive data of the first object, the physical position of the positioning device, and the physical position of the first nearby object, E. deleting redundant sub data of the retrieved data set of the first nearby object creating an optimised sub data, F. retrieving saliency data being related to the first nearby object from the first database, G. calculating a stand-out ratio for the first nearby object based on the saliency data, and H. creating a final description of the surrounding environment based on the descriptive data of the first object, the physical position of the positioning device, the optimised sub data, the stand-out ratio for the first nearby object, and the physical position of the first nearby object, for providing the description of the surrounding environment to e.g. a user of the computer system. The user may make better decisions on various matters based upon the provided description of the surrounding environment.
An advantage of the solution, is that it handles and improves augmented reality experiences and connects these to the real world. Furthermore, it specifically generates in an intelligent manner a value-added description of the surrounding environment based upon retrieved and processed information/data.
The method above may be configured according to different optional embodiments. For example, wherein the method may further comprise the steps of: A. retrieving a first instruction from the optimised sub data, and B. sending the first instruction to the first client for triggering an activity of the first object, or C. sending the first instruction to a second client for triggering an activity of a second object, wherein the second object being related to the first object, or D. sending the first instruction to a third client for triggering an activity of the first nearby object, or E. sending the first instruction to a fourth client for triggering an activity of a second nearby object, wherein the second nearby object being related to the first nearby object.
An advantage of the solution, is that it provides the possibility to send specific instructions to the first or the second object or to the first or the second nearby object. The first or the second object being e.g. a human/humans/a vehicle/vehicles using the system The first or the second nearby object being e.g. a building or any other object found in the surrounding environment. Furthermore, it provides the possibility to trigger corresponding activities of the first or the second object or of the first or the second nearby object. In this manner interesting interactions between the system and e.g. real humans/vehicles using it and/or nearby objects found in the real world could occur. These interactions are based upon the provided value-added descriptions of the surrounding environment.
According to an embodiment of the invention wherein the method may further comprise the step of illustrating the final description of the surrounding environment on a user interface of the first client.
An advantage of the solution, is that it generates a value-added description of the surrounding environment and specifically depicts it on a user interface.
According to an embodiment of the invention wherein the method may further comprise receiving a first behavioural data set of the first object from the first client by means of a first behavioural device being external to and in close proximity to or internal the first object. The method may further comprise receiving a second behavioural data set of the second object from the second client by means of a second behavioural device being external to and in close proximity to or internal the second object.
An advantage of the solution, is that a behavioural device could collect information/data about for instance the real world activities of a first and a second object being e.g. a human/humans or a vehicle/vehicles using the system.
According to an embodiment of the invention wherein the method may further comprise the steps of: A. retrieving a second instruction from the optimised sub data as a response of the received first and second behavioural data set of the first and the second object, and B. sending the second instruction to the first client for triggering an activity of the first object, or C. sending the second instruction to the second client for triggering an activity of the second object, wherein the second object being related to the first object, or D. sending the second instruction to the third client for triggering an activity of the first nearby object, or E. sending the second instruction to the fourth client for triggering an activity of a second nearby object, wherein the second nearby object being related to the first nearby object.
An advantage of the solution, is that it provides the possibility to send specific instructions to the first or the second object or to the first or the second nearby object. The first or the second object being e.g. a human/humans/a vehicle/vehicles using the system The first or the second nearby object being e.g. a building or any other object found in the surrounding environment. These instructions are partly being based upon information/data collected by the first and the second behavioural device about for instance the real-world activities of the first and the second object. Furthermore, it provides the possibility to trigger corresponding activities of the first or the second object or of the first or the second nearby object. In this manner even more interesting interactions between the system and e.g. real humans/vehicles using it and/or nearby objects found in the real world could occur. These interactions are also based upon the provided value-added descriptions of the surrounding environment.
According to an embodiment of the invention wherein the method may further comprise the step of receiving additional data of the first and the second object by means of an external device.
An advantage of the solution, is that it provides additional possibilities to gather information of the real-world activities of the first and the second object. It could specifically generate in an intelligent manner an ever more value-added description of the surrounding environment based upon retrieved and processed information/data. It could also even better illustrate the final description of the surrounding environment on a user interface. It could also even better provide the possibility to send specific instructions to the first or the second object or to the first or the second nearby object. The first or the second object being e.g. a human/humans/a vehicle/vehicles using the system The first or the second nearby object being e.g. a building or any other object found in the surrounding environment. These instructions are partly being based upon information/data collected by the external device and possibly also the first and the second behavioural device about for instance the real-world activities of the first and the second object. Furthermore, it provides the possibility to trigger corresponding activities of the first or the second object or of the first or the second nearby object. In this manner even more interesting interactions between the system and e.g. real humans/vehicles using it and/or nearby objects found in the real world could occur. These interactions are also based upon the provided value-added descriptions of the surrounding environment.
According to an embodiment of the invention wherein the computer system is also arranged to communicate with a second database. The retrieving a data set step may comprise retrieving a first data set of the first and the second nearby object from the first database based upon the physical position of the positioning device, and retrieving a second data set of the first and the second nearby object from the second database based upon the physical position of the positioning device.
An advantage of the solution, is that several independent databases, also externally found, could be used to provide data/information of the first and the second nearby object being e.g. buildings or any other objects found in the surrounding environment
According to an embodiment of the invention wherein the deleting redundant sub data step is conducted by using an optimisation algorithm.
An advantage of the solution, is that it generates in an even more efficient manner an intelligent and value-added description of the surrounding environment based upon retrieved and processed information/data.
According to a second aspect, the invention concerns a computer system for creation of augmented reality experiences and connection of these to the real world. The computer system is arranged to communicate with a first database and is arranged to: A. receive descriptive data of a first object from a first client, B. receive a physical position of a positioning device being external to and in close proximity to or internal the first object, C. retrieve from the first database a data set of a first nearby object having a physical position being related to the physical position of the positioning device, D. create a description of the surrounding environment by combining: the retrieved data set of the first nearby object with the descriptive data of the first object, the physical position of the positioning device, and the physical position of the first nearby object, E. delete redundant sub data of the retrieved data set of the first nearby object using an optimisation algorithm creating an optimised sub data, F. retrieve saliency data being related to the first nearby object from the first database, G. calculate a stand-out ratio for the first nearby object based on the saliency data, and H. create a final description of the surrounding environment based on the descriptive data of the first object, the physical position of the positioning device, the optimised sub data, the stand-out ratio for the first nearby object, and the physical position of the first nearby object, for providing the description of the surrounding environment to e.g. a user of the computer system. The user may make better decisions on various matters based upon the provided description of the surrounding environment.
An advantage of the solution, is that the computer system handles and improves augmented reality experiences and connects these to the real world. Furthermore, it specifically generates in an intelligent manner a value-added description of the surrounding environment based upon retrieved and processed information/data.
According to a third aspect, the invention concerns a computer program, comprising computer readable code means, which when run on a computer system causes the computer system to perform the corresponding method.
According to a fourth aspect, the invention concerns a computer program product, comprising a computer readable medium and a computer program, wherein the computer program is stored on the computer readable medium.
The physical positions could be positions outdoors or indoors.
An activity of a first and a second object could for instance be two humans being triggered to look at a specific part of a building for an interesting architectural item. An activity of a first and a second nearby object could for instance be these buildings providing specific information, on e.g. digital billboards, to two specific humans being in close proximity to these buildings.
The description of the surrounding environment is a conceptual and dynamic model, which provides information of the interrelations between all nearby objects.
The user interface could for instance, among others, be a monitor delivering an image/video or a loudspeaker delivering a voice message.
The first and the second object could for instance, among others, be a person or a vehicle.
The first and the second nearby object could for instance, among others, be a house, a bridge, a road, or a human being.
The behavioural device could for instance, among others, be a motion sensor or a VR headset.
The external device could for instance, among others, be a video camera.
The optimisation algorithm could for instance be a geometric optimisation model, among others. Furthermore, a collision mesh method is an example of a geometric optimisation model which could be used as an optimisation algorithm in this invention.
The descriptive data of an object can, among others, be metadata with regards to the object as well as data describing a feature of the object. Metadata is data that provides information about other data. A feature of the object can e.g. be a feature that can be perceived by another physical being or object.
The first and the second databases mentioned may be external or internal of the computer system.
That the physical position of a nearby object having an association with the physical position of a positioning device, may imply that the positions are physically close to each other. However, other associations may also occur.
The user of the computer system being a human may be equivalent to the first object. The user being a human could also be managing the first object, wherein the first object could be e.g. a vehicle.
The deleting of redundant sub data of the retrieved data set of the first nearby object, could for instance imply that all data of the parts of e.g. a building, being the first nearby object, which the first object, for instance a human being, does not visually see in the real world is deleted. This data to be deleted could for instance be all data representing the back of the building and the internal parts of the building, leaving only the data representing the building which is visually seen by the human being.
An example of the method is here provided. A. Receiving descriptive data of a first object from a first client, may imply that a description of a human being is sent from the smartphone belonging to the human, and received by the system. B. Receiving a physical position of the positioning device being external to and in close proximity to or internal the first object, may imply that the XYX coordinates of the smartphone are collected by a GPS device being embedded in the smartphone and further sent to the system by the smartphone. The human being the holder of the smartphone. The XYZ coordinates of the smartphone further being received by the system. C. Retrieving from the first database a data set of a first nearby object having a physical position being related to the physical position of the positioning device, may imply that a data set is retrieved comprising technical characteristics of a building, defined as the nearby object. The building having XYZ coordinates rather close to the XYZ coordinates of the smartphone of the human being. D. Creating a description of the surrounding environment by combining: the retrieved data set of the first nearby object with the descriptive data of the first object, the physical position of the positioning device, and the physical position of the first nearby object, may imply that a thorough description of the surrounding environment is compiled by the technical characteristics of the building, the description of the human being, the XYZ coordinates of the smartphone i.e. the human being which is the holder of the smartphone and the XYZ coordinates of the building. E. Deleting redundant sub data of the retrieved data set of the first nearby object using an optimisation algorithm creating an optimised sub data, may imply that a collision mesh method is used to decrease the number om data points within the data set needed to illustrate the building. F. Retrieving saliency data being related to the first nearby object from the first database, may imply that the system receives data of how large portion the building represents of the total line-of-sight-view for the human being and how much more the building should be highlighted in addition for the user, based upon e.g. colour of the building, height of the building etc. Saliency data may be characteristic data of the first nearby object. Saliency data may be descriptive data of the first nearby object. Saliency data may be objective data of the first nearby object. Saliency data may comprise the colour of the first nearby object. Saliency data may comprise the width of the first nearby object. Saliency data may comprise the height of the first nearby object, among other data descriptive of the first nearby object. G. Calculating a stand-out ratio for the first nearby object based on the saliency data, may imply that the stand-out ratio for the building, i.e. how much the building relatively should stand out in an overall illustration of the surrounding environment, is calculated based upon of how large portion the building represents of the total line-of-sight-view for the human being and how much more the building should be highlighted in addition for the user, based upon e.g. colour of the building, height of the building etc. H. Creating a final description of the surrounding environment based on the descriptive data of the first object, the physical position of the positioning device, the optimised sub data, the stand-out ratio for the first nearby object, and the physical position of the first nearby object, may imply that an illustration of the surrounding environment is created based upon the description of the human being, the XYZ coordinates of the smartphone held by the human being, the optimised number of data points for illustrating the building, and the XYZ coordinates of the building.
The general conditions of the method are the following: The user/object does not need to be real; Mixing real objects, found in the real world, and non-real objects, in the software model is feasible; There may be a plurality of users/objects in the same world, controlled by the system, that may interact; The clients can also serve as servers, i.e. manage other clients and create their own environments. The clients/servers may communicate with each other; The system can run in off-line mode without own direct access to mobile data by transferring data instead via wifi, bluetooth etc. This could be a preferable solution during a crisis, in the mining industry, and in non-connected systems such as in the defence sector etc.; The GPU, on the graphics card or elsewhere, may work with a lot of the real-time calculations.
The method in the software solution may be described by the following six steps: 1. The server/computer system receives any type of position of an object, an individual or a physical object, e.g. a car, a robot, from the positioning device for instance via the client. This position could be provided by for instance a GPS, beacon: indoor/outdoor, or wifi technology, etc. Also, other types of external devices, such as cameras for instance, could be used to receive/extract data around and with regards to the object, and subsequently transmit the information to the computer system. This can also be conducted in a virtual reality setting. 2. The server/computer system downloads basic data around the object with regards to the given position. This data is for example sourced from:—databases such as Esri (GIS), Streetmap (GIS), Bisnode (business data) etc. or from—local data sources, e.g. point clouds—a large number of coordinates in 3D), or from—Building Information Modeling (BIM) data (3D data). This step implies population of data. Additional data, that is applicable to the nearby objects, is thus sourced from databases and added and to the nearby objects or to the object itself. This resembles of how layers are created in CAD environments. A data model has now been created. Remark: Step 2 may be conducted before step 1, and/or the data in step 2 can already be found on the server/computer system. 3. Swiftly create an image of the surrounding environment by populating the description of the surrounding environment with nearby objects such as roads, houses, stores, etc. The nearby objects are dynamic. However, this description of the surrounding environment may not be made visible to the end user. This is comparable to a 3D engine, but with data that is not visible. 4. Optimise basic data, i.e. information is deleted. This is done e.g. by a “collision mesh” method, which results in a less complicated image of the object. Data is simplified, and retains only what is in “line-of-sight”. A real-time analysis of what an object or user would see is conducted. A calculation is conducted of what an object or a user sees and which objects in the environment that should stand out, i.e. which objects are salient and which are not. If something stands out in a very obvious way, it can be called salient. For instance, a high-rise building should be listed as salient, even though it only represents a small percentage of pixels in an image. A proprietary technology is used for this. FIG. 8 illustrates how large percentage specific objects (A-D) would represent in a line-of-sight view for an object or user at a specific geographical position. Remark: C should be made salient for the object or end user, as it is a high-rise building. This even though C only represents 12% of the total line-of-sight. 5. Track an individual's/an object's behaviour in the environment, e.g.: —how fast an object or individual moves (fast paced running, slow walking). This can be achieved by using sensors in a mobile phone. —what the object or person is looking at. This can be achieved by using a 3D compass in a headset, Google Glass, HTC Vive etc. 6. The system may in this step start to interact with the object or user. Now, the object or user is suddenly a part of the environment/computer model. I.e. the user/object in the virtual environment is consistent with the user/object in the real environment. The system can for instance start influencing the person or object to look at or approach different specific nearby objects.
BRIEF DESCRIPTION OF DRAWINGS
The invention is now described, by way of example, with reference to the accompanying drawings, in which:
FIG. 1 is a flow chart of a procedure according to an embodiment.
FIG. 2 is a flow chart of a procedure according to an embodiment.
FIG. 3 is a flow chart of a procedure according to an embodiment.
FIG. 4 is a flow chart of a procedure according to an embodiment.
FIG. 5 is a flow chart of a procedure according to an embodiment.
FIG. 6 is a block diagram illustrating a computer system, according to a possible embodiment.
FIG. 7 is a block diagram illustrating a computer system in connection with other parts, according to a possible embodiment.
FIG. 8 is a block diagram illustrating a line-of-sight model, according to a possible embodiment.
 DESCRIPTION OF EMBODIMENTS
In the following, a detailed description of a method performed by a computer system for creation of augmented reality experiences and connection of these to the real world is provided.
FIG. 1 shows a flow chart illustrating a procedure in a computer system. The various actions may come in different orders than presented in this description, or in a different order than shown in this or other flowcharts related to this description, or some steps may be performed in parallel.
In a step S100 descriptive data 3 of a first object 4 is received from a first client 5 at the computer system 1. In a step S110 a physical position 6 a of a positioning device 7 being external to and in close proximity to or internal the first object 4, is received at the computer system 1. In a step S120 a data set 8 of a first nearby object 9 having a physical position 6 b being related to the physical position 6 a of the positioning device 7 is retrieved from the first database 2 a. In a step S130 a description of the surrounding environment 11 is created at the computer system 1 by combining: the retrieved data set 8 of the first nearby object 9 with the descriptive data 3 of the first object 4, the physical position 6 a of the positioning device 7, and the physical position 6 b of the first nearby object 9. In a step S140 redundant sub data 8 a of the retrieved data set 8 of the first nearby object 9 is deleted creating an optimised sub data 8 b. In a step S150 saliency data 13 being related to the first nearby object 9 is retrieved from the first database 2 a. In a step S160 a stand-out ratio 14 for the first nearby object 9 is calculated based on the saliency data 13. In a step S170 a final description of the surrounding environment 15 is created based on the descriptive data 3 of the first object 4, the physical position 6 a of the positioning device 7, the optimised sub data 8 b, the stand-out ratio 14 for the first nearby object 9, and the physical position 6 b of the first nearby object 9.
FIG. 2 shows a flow chart illustrating a procedure in a computer system. The various actions may come in different orders than presented in this description, or in a different order than shown in this or other flowcharts related to this description, or some steps may be performed in parallel.
In a step S200 a first instruction 16 is retrieved from the optimised sub data 8 b. In a step S210/S220 the first instruction 16 is sent to the first client 5 for triggering an activity 17 of the first object 4. In a step S210/S230 the first instruction 16 is sent to a second client 18 for triggering an activity 19 of a second object 20, wherein the second object 20 being related to the first object 4. In a step S210/S240 the first instruction 16 is sent to a third client 21 for triggering an activity 22 of the first nearby object 9. In a step S210/S250 the first instruction 16 is sent to a fourth client 23 for triggering an activity 24 of a second nearby object 25, wherein the second nearby object 25 being related to the first nearby object 9.
FIG. 3 shows a flow chart illustrating a procedure in a computer system. The various actions may come in different orders than presented in this description, or in a different order than shown in this or other flowcharts related to this description, or some steps may be performed in parallel.
In a step S300 a first behavioural data set 27 a of the first object 4 is received from the first client 5 by means of a first behavioural device 28 a being external to and in close proximity to or internal the first object 4, and a second behavioural data set 27 b of the second object 20 is received from the second client 18 by means of a second behavioural device 28 b being external to and in close proximity to or internal the second object 20.
FIG. 4 shows a flow chart illustrating a procedure in a computer system. The various actions may come in different orders than presented in this description, or in a different order than shown in this or other flowcharts related to this description, or some steps may be performed in parallel.
In a step S400 a second instruction 29 from the optimised sub data 8 b is retrieved as a response of the received first and second behavioural data set 27 a, 27 b of the first and the second object 4, 20. In a step S410/S420 the second instruction 29 is sent to the first client 5 for triggering an activity 17 of the first object 4. In a step S410/S430 the second instruction 29 is sent to the second client 18 for triggering an activity 19 of the second object 20, wherein the second object 20 being related to the first object 4. In a step S410/S440 the second instruction 29 is sent to the third client 21 for triggering an activity 22 of the first nearby object 9. In a step S410/S450 the second instruction 29 is sent to the fourth client 23 for triggering an activity 22 of a second nearby object 25, wherein the second nearby object 25 being related to the first nearby object 9.
FIG. 5 shows a flow chart illustrating a procedure in a computer system. The various actions may come in different orders than presented in this description, or in a different order than shown in this or other flowcharts related to this description, or some steps may be performed in parallel.
In a step 5.1 descriptive data 3 of a first object 4 is received from a first client 5 at the computer system 1. In a step 5.2 a physical position 6 a of a positioning device 7 being external to and in close proximity to or internal the first object 4, is received at the computer system 1. In a step 5.3 data set 8 of a first nearby object 9 having a physical position 6 b being related to the physical position 6 a of the positioning device 7 is retrieved from the first database 2 a. In a step 5.4 a description of the surrounding environment 11 is created at the computer system 1 by combining: the retrieved data set 8 of the first nearby object 9 with the descriptive data 3 of the first object 4, the physical position 6 a of the positioning device 7, and the physical position 6 b of the first nearby object 9. In a step 5.5 redundant sub data 8 a of the retrieved data set 8 of the first nearby object 9 is deleted creating an optimised sub data 8 b. In a step 5.6 saliency data 13 being related to the first nearby object 9 is retrieved from the first database 2 a. In a step 5.7 a stand-out ratio 14 for the first nearby object 9 is calculated based on the saliency data 13. In a step 5.8 a final description of the surrounding environment 15 is created based on the descriptive data 3 of the first object 4, the physical position 6 a of the positioning device 7, the optimised sub data 8 b, the stand-out ratio 14 for the first nearby object 9, and the physical position 6 b of the first nearby object 9.
FIG. 6 shows a computer 1, including components such as a processor 1.a, a user interface 1.b, a memory 1.c and communication gateways 1.d.
FIG. 7 shows a user interface 5.a of a first client 5 used by a first object 4, a positioning device 7, a first database 2 a and a first nearby object 9. The first client 5, the positioning device 7, the first database 2 a and the first nearby object 9 being connected to a computer 1.
FIG. 8 shows how a line-of-sight of a first object, in this case a human being, is modelled in general. The first, second, third and fourth nearby objects being buildings. The saliency data retrieved is used to calculate a stand-out ratio for each building. In this particular example, building C will have a stand-out ratio higher than its relative size, as it's a high-rise building compared to the other buildings being of normal height.

Claims (11)

The invention claimed is:
1. Method performed by a computer system (1) for creation of augmented reality experiences and connection of these to the real world, wherein the computer system (1) is arranged to communicate with a first database (2 a) comprising data on a surrounding environment (11), and communicate with a first client (5), and
wherein the data on the surrounding environment (11) comprises one or more of geographic data, building data, and map data, the method comprising the steps of:
A. receiving descriptive data (3) of a first object (4) from the first client (5), the descriptive data comprising metadata of the first object (4) and/or data describing a feature of the first object (4),
B. receiving a physical position (6 a) of a location determining device (7) being external to and in close proximity to or internal of the first object (4),
C. retrieving from the first database (2 a) a data set (8) of a first nearby object (9) having a physical position (6 b) in the surrounding environment (11) wherein the data set is sourced from an application comprising one or more of a geographic information system (GIS), building information mapping (BIM), or business data (Bisnode),
D. creating a description of the surrounding environment (11) by combining: the retrieved data set (8) of the first nearby object (9) with the descriptive data (3) of the first object (4), the physical position (6 a) retrieved by the location determining device (7), and the physical position (6 b) of the first nearby object (9),
E. deleting redundant sub data (8 a) of the retrieved data set (8) of the first nearby object (9) creating an optimised sub data (8 b) wherein the redundant sub data (8 a) comprises all data of portions of the first nearby object (9) that the first object (4) does not visually observe in the real world,
F. retrieving a dataset of saliency data (13) of the first nearby object (9) from the first database (2 a),
wherein the saliency data (13) comprises data related to physical features or rendering aspects of the first nearby object (9),
G. calculating a stand-out ratio (14) for the first nearby object (9) based on the saliency data (13),
wherein the stand-out ratio (14) is a value of how much the first nearby object (9) stands out in an overall illustration of the surrounding environment (11), and calculated based on the portion of the first nearby object (9) represents of the total line-of-sight-view, and
H. creating a final description of the surrounding environment (15) based on the descriptive data (3) of the first object (4), the physical position (6 a) of the location determining device (7), the optimised sub data (8 b), the stand-out ratio (14) for the first nearby object (9), and the physical position (6 b) of the first nearby object (9), for providing the description of the surrounding environment (15).
2. Method performed by a computer system (1) according to claim 1, wherein the method further comprises the steps of:
A. retrieving a first instruction (16) from the optimised sub data (8 b), and
B. sending the first instruction (16) to the first client (5) for triggering an activity (17) of the first object (4), or
C. sending the first instruction (16) to a second client (18) for triggering an activity (19) of a second object (20), wherein the second object (20) being related to the first object (4), or
D. sending the first instruction (16) to a third client (21) for triggering an activity (22) of the first nearby object (9), or
E. sending the first instruction (16) to a fourth client (23) for triggering an activity (24) of a second nearby object (25), wherein the second nearby object (25) being related to the first nearby object (9).
3. Method performed by a computer system (1) according to claim 2, wherein the method further comprises receiving a first behavioural data set (27 a) describing a behaviour of the first object (4) from the first client (5) by means of a first behavioural device (28 a) that is collecting information comprising real world activities of the first object (4) using the computer system (1) and being external to and in close proximity to or internal of the first object (4), and wherein the method further comprises receiving a second behavioural data set (27 b) describing a behaviour of the second object (20) from the second client (18) by means of a second behavioural device (28 b) that is collecting information comprising real world activities of the second object (20) using the computer system (1) and being external to and in close proximity to or internal of the second object (20).
4. Method performed by a computer system (1) according to claim 3, wherein the method further comprises the steps of:
A. retrieving a second instruction (29) from the optimised sub data (8 b) as a response of the received first and second behavioural data set (27 a, 27 b) of the first and the second object (4, 20), and
B. sending the second instruction (29) to the first client (5) for triggering an activity (17) of the first object (4), or
C. sending the second instruction (29) to the second client (18) for triggering an activity (19) of the second object (20), wherein the second object (20) being related to the first object (4), or
D. sending the second instruction (29) to the third client (21) for triggering an activity (22) of the first nearby object (9), or
E. sending the second instruction (29) to the fourth client (23) for triggering an activity (22) of a second nearby object (25), wherein the second nearby object (25) being related to the first nearby object (9).
5. The method performed by a computer system (1) according to claim 3, wherein the behavioral device (28 a, b) is a sensor configured to track a behavior of an object comprising movement, pace, and direction.
6. Method performed by a computer system (1) according to claim 2, wherein the computer system (1) is configured to communicate with a second database (2 b), wherein the retrieving a data set (8) step comprises retrieving a first data set (8 c) of the first nearby object (9) and the second nearby object (25) from the first database (2 a) based upon the physical position (6 a) of the location determining device (7), and retrieving a second data set (8 d) of the first nearby object (9) and the second nearby object (25) from the second database (2 b) based upon the physical position (6 a) of the location determining device (7).
7. Method performed by a computer system (1) according to claim 1, wherein the method further comprises the step of sending and illustrating the final description of the surrounding environment (15) on a user interface (5 a) of the first client (5).
8. Method performed by a computer system (1) according to claim 1, wherein the deleting redundant sub data step is conducted by using a geometric optimization model.
9. The method performed by a computer system (1) according to claim 8, wherein the geometric optimization model comprises a collision mesh model.
10. A computer system (1) for creation of augmented reality experiences and connection of these to the real world, wherein the computer system (1) is configured to communicate with a first database (2 a) and configured to:
A. receive descriptive data (3) of a first object (4) from a first client (5), the descriptive data being metadata comprising the first object (4) and/or data describing a feature of the first object (4),
B. receive a physical position (6 a) of a location determining device (7) being external to and in close proximity to or internal of the first object (4),
C. retrieve from the first database (2 a) a data set (8) of a first nearby object (9) having a physical position (6 b) in the surrounding environment (11) wherein the data set (8) is sourced from an application comprises one or more of a geographic information system (GIS), building information mapping (BIM), or business data (Bisnode),
D. create a description of the surrounding environment (11) by combining: the retrieved data set (8) of the first nearby object (9) with the descriptive data (3) of the first object (4), the physical position (6 a) retrieved by the location determining device (7), and the physical position (6 b) of the first nearby object (9),
E. delete redundant sub data (8 a) of the retrieved data set (8) of the first nearby object (9) using a geometric optimization model creating an optimised sub data (8 b), wherein the redundant sub data (8 a) comprises all data of portions of the first nearby object (9) that the first object (4) does not visually observe in the real world,
F. retrieve datasets of saliency data (13) of the first nearby object (9) from the first database (2 a), wherein the saliency data (13) comprises data related to physical features or rendering aspects of the first nearby object (9),
G. calculate a stand-out ratio (14) for the first nearby object (9) based on the saliency data (13),
wherein the stand-out ratio is a value of how much the first nearby object (9) stands out in an overall illustration of the surrounding environment (11), and calculated based on the portion of the first nearby object (9) represents of the total line-of-sight-view, and
H. create a final description of the surrounding environment (15) based on the descriptive data (3) of the first object (4), the physical position (6 a) of the location determining device (7), the optimised sub data (8 b), the stand-out ratio (14) for the first nearby object (9), and the physical position (6 b) of the first nearby object (9), for providing the final description of the surrounding environment (15).
11. A non-transitory computer-readable medium storing a computer program including instructions that, which when run on a computer system for creation of augmented reality experiences and connection of these to the real world, causes the computer system to perform the method according to claim 1, wherein the computer system is configured to communicate with a first database (2 a) and is arranged to:
A. receive descriptive data (3) of a first object (4) from a first client (5),
B. receive a physical position (6 a) of a location determining device (7) being external to and in close proximity to or internal of the first object (4),
C. retrieve from the first database (2 a) a data set (8) of a first nearby object (9) having a physical position (6 b) in the environment, wherein the data set sourced from an application comprises one or more of geographic information system (GIS), building information mapping (BIM), or business data (Bisnode),
D. create a description of the surrounding environment (11) by combining: the retrieved data set (8) of the first nearby object (9) with the descriptive data (3) of the first object (4), the physical position (6 a) retrieved by the location determining device (7), and the physical position (6 b) of the first nearby object (9),
E. delete redundant sub data (8 a) of the retrieved data set (8) of the first nearby object (9) using a geometric optimization model creating an optimised sub data (8 b),
wherein the redundant sub data (8 a) comprises all data of portions of the first nearby object (9) that the first object (4) does not visually observe in the real world,
F. retrieve datasets of saliency data (13) of the first nearby object (9) from the first database (2 a),
wherein the saliency data (13) comprises data related to physical features or rendering aspects of the first nearby object (9),
G. calculate a stand-out ratio (14) for the first nearby object (9) based on the saliency data (13), wherein the stand-out ratio (14) is a value of how much the first nearby object (9) stands out in an overall illustration of the surrounding environment (11), and calculated based on the portion of the first nearby object (9) represents of the total line-of-sight-view, and
H. create a final description of the surrounding environment (15) based on the descriptive data (3) of the first object (4), the physical position (6 a) of the location determining device (7), the optimised sub data (8 b), the stand-out ratio (14) for the first nearby object (9), and the physical position (6 b) of the first nearby object (9), for providing the description of the surrounding environment (15).
US17/399,389 2020-08-13 2021-08-11 Method performed by a computer system for creation of augmented reality experiences and connection of these to the real world Active US11762453B2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
SE2030252A SE2030252A1 (en) 2020-08-13 2020-08-13 Method performed by a computer system for creation of augmented reality experiences and connection of these to the real world
SE2030252-7 2020-08-13

Publications (2)

Publication Number Publication Date
US20220050519A1 US20220050519A1 (en) 2022-02-17
US11762453B2 true US11762453B2 (en) 2023-09-19

Family

ID=77316830

Family Applications (1)

Application Number Title Priority Date Filing Date
US17/399,389 Active US11762453B2 (en) 2020-08-13 2021-08-11 Method performed by a computer system for creation of augmented reality experiences and connection of these to the real world

Country Status (3)

Country Link
US (1) US11762453B2 (en)
EP (1) EP3955094B1 (en)
SE (1) SE2030252A1 (en)

Citations (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110050732A1 (en) 2009-09-03 2011-03-03 Nokia Corporation Method and apparatus for customizing map presentations based on user interests
US20130147837A1 (en) 2011-12-13 2013-06-13 Matei Stroila Augmented reality personalization
US20140136726A1 (en) * 2007-10-24 2014-05-15 Social Communications Company Realtime kernel
US20160026253A1 (en) * 2014-03-11 2016-01-28 Magic Leap, Inc. Methods and systems for creating virtual and augmented reality
US20160133054A1 (en) 2014-11-12 2016-05-12 Canon Kabushiki Kaisha Information processing apparatus, information processing method, information processing system, and storage medium
US9345957B2 (en) * 2011-09-30 2016-05-24 Microsoft Technology Licensing, Llc Enhancing a sport using an augmented reality display
US20160366326A1 (en) * 2015-06-10 2016-12-15 Microsoft Technology Licensing, Llc Determination of exposure time for an image frame
WO2017031246A1 (en) 2015-08-18 2017-02-23 Magic Leap, Inc. Virtual and augmented reality systems and methods
US20170068323A1 (en) * 2015-09-08 2017-03-09 Timoni West System and method for providing user interface tools
US20170364752A1 (en) 2016-06-17 2017-12-21 Dolby Laboratories Licensing Corporation Sound and video object tracking
US20180302590A1 (en) 2017-04-16 2018-10-18 Facebook, Inc. Systems and methods for presenting content
US20190051014A1 (en) * 2017-08-14 2019-02-14 Trimble Inc. Self positioning camera system to 3d cad/bim model
US20190102939A1 (en) * 2017-09-29 2019-04-04 Sony Interactive Entertainment America Llc Projectile object rendering for a virtual reality spectator
WO2019079826A1 (en) 2017-10-22 2019-04-25 Magical Technologies, Llc Systems, methods and apparatuses of digital assistants in an augmented reality environment and local determination of virtual object placement and apparatuses of single or multi-directional lens as portals between a physical world and a digital world component of the augmented reality environment
US10403050B1 (en) * 2017-04-10 2019-09-03 WorldViz, Inc. Multi-user virtual and augmented reality tracking systems
CN110285818A (en) 2019-06-28 2019-09-27 武汉大学 A kind of Relative Navigation of eye movement interaction augmented reality
US20200051336A1 (en) * 2016-11-21 2020-02-13 Sony Corporation Information processing device, information processing method, and program
US20200090407A1 (en) 2018-08-13 2020-03-19 Magic Leap, Inc. Cross reality system
US20200111255A1 (en) * 2018-10-05 2020-04-09 Magic Leap, Inc. Rendering location specific virtual content in any location
US20200335001A1 (en) * 2019-04-22 2020-10-22 The Commons XR LLC Virtual, augmented and extended reality system
US20200401593A1 (en) * 2018-07-24 2020-12-24 MachEye, Inc. Dynamic Phase Generation And Resource Load Reduction For A Query
US20210104100A1 (en) * 2019-10-02 2021-04-08 Magic Leap, Inc. Mission driven virtual character for user interaction
US20210173887A1 (en) * 2009-02-02 2021-06-10 Cria, Inc. Sytem and method for filtering and creating points-of-interest

Patent Citations (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140136726A1 (en) * 2007-10-24 2014-05-15 Social Communications Company Realtime kernel
US20210173887A1 (en) * 2009-02-02 2021-06-10 Cria, Inc. Sytem and method for filtering and creating points-of-interest
US20110050732A1 (en) 2009-09-03 2011-03-03 Nokia Corporation Method and apparatus for customizing map presentations based on user interests
US9345957B2 (en) * 2011-09-30 2016-05-24 Microsoft Technology Licensing, Llc Enhancing a sport using an augmented reality display
US20130147837A1 (en) 2011-12-13 2013-06-13 Matei Stroila Augmented reality personalization
US20160026253A1 (en) * 2014-03-11 2016-01-28 Magic Leap, Inc. Methods and systems for creating virtual and augmented reality
US20160133054A1 (en) 2014-11-12 2016-05-12 Canon Kabushiki Kaisha Information processing apparatus, information processing method, information processing system, and storage medium
US20160366326A1 (en) * 2015-06-10 2016-12-15 Microsoft Technology Licensing, Llc Determination of exposure time for an image frame
US20180061139A1 (en) * 2015-08-18 2018-03-01 Magic Leap, Inc. Virtual and augmented reality systems and methods
WO2017031246A1 (en) 2015-08-18 2017-02-23 Magic Leap, Inc. Virtual and augmented reality systems and methods
US20170068323A1 (en) * 2015-09-08 2017-03-09 Timoni West System and method for providing user interface tools
US20170364752A1 (en) 2016-06-17 2017-12-21 Dolby Laboratories Licensing Corporation Sound and video object tracking
US20200051336A1 (en) * 2016-11-21 2020-02-13 Sony Corporation Information processing device, information processing method, and program
US10403050B1 (en) * 2017-04-10 2019-09-03 WorldViz, Inc. Multi-user virtual and augmented reality tracking systems
US20180302590A1 (en) 2017-04-16 2018-10-18 Facebook, Inc. Systems and methods for presenting content
US20190051014A1 (en) * 2017-08-14 2019-02-14 Trimble Inc. Self positioning camera system to 3d cad/bim model
US20190102939A1 (en) * 2017-09-29 2019-04-04 Sony Interactive Entertainment America Llc Projectile object rendering for a virtual reality spectator
WO2019079826A1 (en) 2017-10-22 2019-04-25 Magical Technologies, Llc Systems, methods and apparatuses of digital assistants in an augmented reality environment and local determination of virtual object placement and apparatuses of single or multi-directional lens as portals between a physical world and a digital world component of the augmented reality environment
US20200401593A1 (en) * 2018-07-24 2020-12-24 MachEye, Inc. Dynamic Phase Generation And Resource Load Reduction For A Query
US20200090407A1 (en) 2018-08-13 2020-03-19 Magic Leap, Inc. Cross reality system
US20200111255A1 (en) * 2018-10-05 2020-04-09 Magic Leap, Inc. Rendering location specific virtual content in any location
US20200335001A1 (en) * 2019-04-22 2020-10-22 The Commons XR LLC Virtual, augmented and extended reality system
CN110285818A (en) 2019-06-28 2019-09-27 武汉大学 A kind of Relative Navigation of eye movement interaction augmented reality
US20210104100A1 (en) * 2019-10-02 2021-04-08 Magic Leap, Inc. Mission driven virtual character for user interaction

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Extended Search Report and Written Opinion of the European Patent Office dated Jan. 25, 2022 in EP Application No. 21190925.4 (EP3955094A2); 12pgs.
Jänicke, H., & Chen, M. (Jun. 2010). A salience-based quality metric for visualization. In Computer graphics forum (vol. 29, No. 3, pp. 1183-1192). Oxford, UK: Blackwell Publishing Ltd. *
Liu et al., "Locally Refined Collision Detection of Large Scale Complex Polygonal Meshes in Distributed Virtual Environments", 2006 IEEE International Conference on Multimedia and Expo, 2006, pp. 209-212, doi: 10.1109/ICME.2006.262419.

Also Published As

Publication number Publication date
US20220050519A1 (en) 2022-02-17
SE2030252A1 (en) 2022-02-14
EP3955094A2 (en) 2022-02-16
EP3955094B1 (en) 2024-09-04
EP3955094A3 (en) 2022-02-23

Similar Documents

Publication Publication Date Title
US11729245B2 (en) Platform for constructing and consuming realm and object feature clouds
CN112334953B (en) Multiple integration model for device localization
JP7179819B2 (en) Cloud-enabled augmented reality
JP2014106585A (en) Information processing device, terminal device, information processing method and program
WO2011109186A1 (en) Systems and methods for generating real-time three-dimensional graphics of an area of interest
JP2021056220A (en) Visual marker equipped with position recognition function
Narciso et al. Mixar mobile prototype: Visualizing virtually reconstructed ancient structures in situ
CN114385934A (en) System for jointly inquiring multiple AR maps
US11762453B2 (en) Method performed by a computer system for creation of augmented reality experiences and connection of these to the real world
KR100945288B1 (en) Applying Extended ARG to Augmented Reality Based Single Media Multi-Services Method and the system
US20220375175A1 (en) System For Improving The Precision and Accuracy of Augmented Reality
WO2022224964A1 (en) Information processing device and information processing method
KR20230166760A (en) Method for generating metaverse space of hyper-personalized design and a metaverse system for performing the same

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE