US20220143503A1 - Multiuser augmented reality method - Google Patents
Multiuser augmented reality method Download PDFInfo
- Publication number
- US20220143503A1 US20220143503A1 US17/584,319 US202217584319A US2022143503A1 US 20220143503 A1 US20220143503 A1 US 20220143503A1 US 202217584319 A US202217584319 A US 202217584319A US 2022143503 A1 US2022143503 A1 US 2022143503A1
- Authority
- US
- United States
- Prior art keywords
- processor
- anchor
- information
- communication
- view
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/50—Controlling the output signals based on the game progress
- A63F13/52—Controlling the output signals based on the game progress involving aspects of the displayed game scene
- A63F13/525—Changing parameters of virtual cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/20—Input arrangements for video game devices
- A63F13/21—Input arrangements for video game devices characterised by their sensors, purposes or types
- A63F13/213—Input arrangements for video game devices characterised by their sensors, purposes or types comprising photodetecting means, e.g. cameras, photodiodes or infrared cells
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/30—Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/70—Game security or game management aspects
- A63F13/79—Game security or game management aspects involving player-related data, e.g. identities, accounts, preferences or play histories
- A63F13/795—Game security or game management aspects involving player-related data, e.g. identities, accounts, preferences or play histories for finding other players; for building a team; for providing a buddy list
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/80—Special adaptations for executing a specific game genre or game mode
- A63F13/847—Cooperative playing, e.g. requiring coordinated actions from several players to achieve a common goal
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/90—Constructional details or arrangements of video game devices not provided for in groups A63F13/20 or A63F13/25, e.g. housing, wiring, connections or cabinets
- A63F13/92—Video game devices specially adapted to be hand-held while playing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/02—Details
- H04L12/16—Arrangements for providing special services to substations
- H04L12/18—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast
- H04L12/1813—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast for computer conferences, e.g. chat rooms
- H04L12/1822—Conducting the conference, e.g. admission, detection, selection or grouping of participants, correlating users to one or more conference sessions, prioritising transmission
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H04L67/38—
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
- A63F2300/80—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game specially adapted for executing a specific type of game
- A63F2300/8082—Virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
- G06T2219/024—Multi-user, collaborative environment
Definitions
- the present invention relates generally to an augmented reality system. More particularly, it relates to a system and method for allowing multiple participants to share in an augmented reality session, based on the real-world location of one or more anchors.
- Augmented reality (AR) toys have been introduced.
- the basic hardware required, an AR device is a smartphone, laptop, tablet computer, dedicated device, or the like, that views the real world with a camera, and displays that view on a display to an user.
- the AR device runs software that recognizes certain objects (herein termed “anchors”) in the camera's field of view, including the anchor's apparent distance and orientation relative to the camera, and then augments the image from the camera with computer generated graphics that seem to appear in, on, or near the anchor.
- the resulting modified image is displayed to the user (often, a game player, in which case, the AR device may also be executing game code).
- Such toys include a line of jigsaw puzzles, by Ravensburger AG of Ravensburg, Germany which when viewed with the corresponding AR app on an smartphone or tables replaces a portion of the puzzle with animated figures.
- Another one is called “AR Games” by Nintendo Co., Ltd. of Kyoto, Japan, which provides preprinted paper cards as anchors and plays on their Nintendo 3 DS handheld gaming system.
- the Eclipse AR Card Game by MoOn-Studio of also uses preprinted cards as anchors, and allows two players to interact by manipulating the cards, however the two players share a single camera and display attached to a personal computer provides the same view to both players.
- None of these games offers each of several players their own AR device (e.g., smartphone, tablet, or game system) to allow the players to each view the same anchors and see a mutually coherent augmented reality.
- AR device e.g., smartphone, tablet, or game system
- None of the existing AR systems insert a representation of the users into the AR in place of their own AR device, whether by recognizing the AR device as an anchor itself, or by having an AR device report its orientation and distance from another anchor.
- existing AR systems do not offer an ability to cojoin a first site having anchors and AR devices with anchors and AR devices at a second site, so as to give users at the distinct sites an apparently coherent AR world.
- the object of the present invention is to allow several players, each with their own AR device, to share a mutually coherent augmented reality.
- FIG. 1 is a block diagram of a collaborative AR system being used in a situation where there are two sessions
- FIG. 2 illustrates the detailed displays from the three AR devices in the system of FIG. 1 ;
- FIG. 3 shows one example block diagram for an AR device, as might be implemented using a tablet
- FIG. 4 is one example for a portion of a database accumulated and maintained locally by an AR device for operation in conjunction with other AR devices;
- FIG. 5 is one example database maintained by a server for facilitating joint AR sessions and mitigating collisions of identical anchors
- FIG. 6 show one process for acquiring AR anchors for use in a shared AR session
- FIG. 7 shows block diagram of another embodiment for a collaborative AR system, where multiple participants use anchors and AR devices at separate sites, yet share a coherent AR experience;
- FIG. 8 illustrates a detailed display from one AR device in the system of FIG. 7 .
- FIG. 1 shows three AR users 101 , 102 , 103 , at site 100 , in this case players of an AR game, each having a corresponding AR device 110 , 120 , 130 .
- Each of the AR devices 110 , 120 , 130 has controls for use by respective user, e.g., button 111 and touchscreen display 112 .
- Each has a camera offering corresponding fields of view 114 , 124 , 134 and corresponding wireless connections 113 , 123 , 133 to wireless station 140 which offers connectivity, for example via Internet 141 , to server 142 having communication with database 143 .
- FIG. 1 shows two toy castles 150 , 160 , used as anchors recognizable by the AR devices.
- AR devices 110 , 120 , 130 could not determine whether their respective views of castles 150 and 160 would correspond to a single anchor, or whether they were distinct copies.
- AR devices 110 and 120 each viewing castle/anchor 150 could not be sure that there were not two castles that looked alike (castle 150 and a duplicate), or if there was only one castle 150 .
- each AR device registers the anchors it recognizes in conjunction with the general location of the AR device, for example as determined by GPS (Global Positioning System), cellular telephone or WiFi antenna proximity, etc.
- GPS Global Positioning System
- each of the AR devices 110 , 120 , 130 report proximity to wireless station 140 .
- An example series of transactions might go like this: First, AR device is directed by user 101 to start a session. AR device 110 recognizes an anchor (castle 150 ) and registers it in conjunction with the session just started with the location information identifying wireless station 140 (or perhaps GPS information obtained directly by AR device 110 or attributed to wireless station 140 ).
- player 102 attempts to play with AR device 120 .
- Player 102 can either explicitly join the session of player 101 , or AR device 120 can recognize the anchor that is castle 150 and report it to server 142 .
- AR device 120 reports anchor 150 in conjunction with the location as determined by AR device 120 (which can include location information identifying wireless station 140 )
- server 142 can respond that the anchor embodied in castle 150 is already assigned to a session belonging to player 101 and question whether player 102 wants to join the session of player 101 .
- the anchors reported by AR device 120 are considered to be the same as like anchors reported by AR device 110 (i.e., castle 150 ).
- AR device 130 determines its location, for example being proximate to wireless station 140 , or by GPS coordinate, etc., and recognizes castle 160 as an anchor in its field of view 134 . Initially, take the anchor provided by castle 160 as being a duplicate of castle 150 , that is, the AR devices cannot distinguish between them.
- server 142 will ask whether player 103 is attempting to join the session of player 101 .
- castle 150 has two towers 151 , 152 .
- castle 160 once modified by removing a tower, has only one remaining, tower 161 (the tower removed from castle 160 is not shown).
- markers or labels (not shown) applied to various parts of castle 160 could be switched, or reoriented, or added.
- Pennants small colored or patterned flags arranged on flagpoles where they can be easily seen and identified by the AR devices and recognition software, might be added, removed, or altered.
- server 142 might even propose the modification to be made. For example, server 142 might prescribe a particular pennant or sequence of pennants to a player starting a session. That way, any player directing his AR device at an anchor so configured will be automatically entered into the corresponding session having substantially the same location. Server 142 might prescribe the configuration for every anchor being added to a session or may prescribe the configuration only for those anchors which seem to conflict with anchors already registered in session with a sufficiently similar location.
- “sufficiently similar location” would be established by policy. If GPS is used, this could mean within the accuracy of the GPS reading, plus an error band, times two, which might be 100 m or some other predetermined value, which may vary by the location itself (e.g., larger in cities and smaller in rural areas where GPS can be more accurate). The same is true of location services that use the cellular telephone network antennas for triangulation of location. If proximity to WiFi is used for location, “sufficiently similar” might be when both AR devices can detect the same WiFi antenna, though policy might limit that to detection with more than a predetermined signal strength (e.g., “two bars”, so many dBm, or milliwatts).
- a predetermined signal strength e.g., “two bars”, so many dBm, or milliwatts.
- the augmented reality views displayed by AR devices 110 and 120 are coherent, that is, they reflect the same augmented reality, even though the corresponding players views of that common augmented reality are different.
- FIG. 2 where AR devices 110 , 120 , 130 are shown (in this embodiment, each is a tablet).
- the touchscreen display 112 of AR device 110 shows image 210 in which castle 150 is shown from the vantage of field of view 114 as castle image 251 .
- the AR software has embellished this view of the castle with a computer generated dragon 212 just having set fire 213 to the far side of the castle.
- AR device 120 displays image 220 , showing castle 150 as seen in field of view 124 .
- the augmented reality elements of the dragon and fire are seen here, but since player 102 has a view of the back side of the castle, castle image 252 is seen with flames 223 coming out the near side with dragon 222 shown in the foreground.
- AR device 120 when AR device 120 is in the field of view 114 of AR device 110 , AR device 120 may be recognized as an anchor having the role associated with player 202 . In such a case, in augmented reality image 210 , the anchor that is AR device 120 might be replaced by an appropriate visage 214 representing the viewpoint of the other player.
- the AR devices need not be recognized as anchors themselves, nor ever be detected by the other AR devices.
- each AR device may determine its range from and orientation with respect to one or more anchors (e.g., castle 150 ) in the session, and report this information to the other AR devices, either directly or through server 142 .
- AR device 120 can place an indicator 224 to show in what direction the other player(s), i.e., player 101 , are located. This can be useful in a game context, to help a player anticipate what another player is thinking or doing, or where is attention is directed. This feature is particularly valuable with respect to games conducted across multiple sites, where direct viewing of the opposing player is not available, as discussed below in conjunction with FIGS. 7 and 8 .
- the augmented reality image 230 displayed by AR device 130 is show, with image 263 of castle 160 .
- no dragon is attacking the castle.
- a computer generated siege engine 232 is attacking the castle image 263 , and player 103 is engaged in a session separate from that which players 101 and 102 are enjoying together.
- FIG. 3 provides an example block diagram for AR device 110 .
- Processor (CPU) 310 uses memory 311 in which is stored the AR application software, corresponding data, and operating system code.
- Processor 310 directs graphic engine 312 which drives display 313 , which is overlayed by touch panel 314 which together form touchscreen 112 .
- Touch panel 314 is monitored by touch interface electronics 315 , which report user interactions back to processor 310 .
- Other user interface devices, such as button 111 are monitored by other I/O interface 316 , also reporting to processor 310 .
- Camera 317 provides an image for field of view 114 , which is made available to image processor 318 (which could be a software module instead of hardware) wherein the recognition of anchors is performed.
- Location sensor 319 may comprise one or more of a GPS receiver, accelerometers (for detecting the attitude of AR device 110 ), gyroscopes (for quickly detecting the magnitude of sudden changes, but also for detecting very slow changes), flux gate compasses (to detect bearing relative to the Earth's magnetic field), etc., all of which have communication with processor 310 , either directly or, in some embodiments, through intermediary utility processors (not shown). Finally, processor 310 has access to network communication channel 320 , for example, a wireless connection as to station 140 , through which communication is available with server 142 and in some embodiments, other AR devices (e.g., 120 ).
- network communication channel 320 for example, a wireless connection as to station 140 , through which communication is available with server 142 and in some embodiments, other AR devices (e.g., 120 ).
- server 142 may be performed by one of the AR devices, for example the AR device that starts a session, or may be
- an AR device attempting to start a new session might broadcast a signal soliciting information about other sessions or anchors already in use.
- the response if any, might indicate which anchors are already in use, in which sessions, and may further suggest other alternate configurations of selected anchors that might be used instead.
- FIG. 4 shows one example of portion 400 of a local database used to keep track of which anchors, wireless stations, and the AR devices known to one AR device (e.g., 110 ). While illustrated using a relational database connection, those skilled in the art will recognize that many other alternative choices for data representation could be applied and that this representation was chosen for clarity in this example.
- Device table 410 identifies each distinct AR device (including itself) by a unique identifier, ARDevicelD, which could be a globally unique identifier (GUID) or universally unique identifier (UUID), well known in the art, or other identifier that will not be repeated among other AR devices in the system.
- GUID globally unique identifier
- UUID universally unique identifier
- the ARDevicelD for the device itself might be provided at the time of manufacture, or by server 142 when AR device 110 first registers itself with the system.
- the table 410 also includes information relating to one or more communication addresses for each device, e.g., an IP (Internet Protocol) address, MAC (Media Access Control) address, etc. This allows individual AR devices to communicate with other AR devices without going though server 142 .
- IP Internet Protocol
- MAC Media Access Control
- One or more security keys may be provided, for example a public/private key, so as to allow a server to communicate securely and authoritatively to an AR device (for example, to supply a software update or carry out a financial transaction), in which case the AR device holds the public key for the server and the server uses its private key to sign or encrypt messages it sends; or if a message is to be authenticated as coming from the AR device, the AR device would use its private key to sign or encrypt the message and the public key (which is the only portion distributed to the server or other devices), would be used to authenticate the message upon receipt.
- an AR device for example, to supply a software update or carry out a financial transaction
- the AR device holds the public key for the server and the server uses its private key to sign or encrypt messages it sends
- the AR device would use its private key to sign or encrypt the message and the public key (which is the only portion distributed to the server or other devices), would be used to authenticate the message upon receipt.
- the current attitude and orientation in space (e.g., as determined by accelerometers, gyros, and magnetic compasses in location sensor 319 ) would be stored in the attitude field, whether obtained from its own sensors or received in communications from other AR devices. Likewise, latitude and longitude, if available whether from GPS or other location service, would be noted. Exactly one player identity (discussed below in conjunction with table 425 ) is associated via relationship 412 whenever the AR device is participating in a shared session but need not be otherwise. Other fields, not shown, for example, when each AR device was first or most recently contacted might also be recorded here, for example, to determine the age of the attitude and position information.
- Table 420 stores information about users of the device, for example their username and authentication. There might be only one record in this table, if the device has only one user. Each user record in table 420 may correspond to zero or more player personae, recorded in player table 425 , each having a unique player ID. Each player record is associated with, at most, one user record by the UserlD field that forms relationship 421 (Note: Players on other AR devices need not have a corresponding user record on this AR device). Additionally, the player name field records the in-game name for the player, which could be their game character's name.
- Other fields might include or reference a player-specific icon or image; for example, each player might have a unique visage for use when augmented reality scenes are rendered (e.g., for visage 214 , if it is to be specific to the player, rather than generic or determined by some other game mechanic).
- Player status such as ‘INACTIVE’ for all players not currently playing, or other game-mediated states would be entered for the local player and could be sent to other AR devices during a session.
- messages describing players received from other AR devices would be gathered here.
- anchors table 430 is constant, or could be authoritatively updated, for example by server 142 .
- the collective records represent a canonical description of each of the available anchors.
- a representation of the patterns that identify each kind of anchor, including the possible configuration variations, is provided sufficient to inform the image processing module 318 what to look for in the camera image, as to what constitutes an anchor to be recognized.
- a field describing in detail the geometry of the anchor e.g., the geometric shape of castle 150 ), sufficient so that graphics engine module 312 can orient the geometry of the anchor to the recognized patterns, so that when the camera image is being augmented, the geometry model can be used to enhance the rendition of the final image.
- the geometry of the castle anchor is used to mask flames 213 so that they appear behind the castle image 251 obtained from the camera, whereas in AR image 220 , the flames 223 are in front of the castle, but appear to be emanating from a particular window. Additional information (not shown) concerning anchor-specific animations or embellishments that might be added under certain conditions (e.g., how rain should funnel off the roof, or how the walls should look after receiving a certain amount of damage), might also be recorded here. Updates to table 430 would occur if the manufacturer offers new anchors, or if updates to the existing catalog of anchors were to be made.
- alternative patterns may also be included. This would allow readily available objects (e.g., a can from a particular brand of soda) to be used in lieu of castle model 150 . In such a case, the geometry information in table 430 would be used to render the castle image 251 , because the un-augmented camera image would show a soda can.
- objects e.g., a can from a particular brand of soda
- a catalog of anchors could be provided, but dynamically associated with a physical representation (i.e., the pattern), so that a user could make their own anchors, e.g., by printing a suitable barcode on paper, registering that barcode as a pattern associated with a particular anchor record, thereby allowing the printout to be used as the anchor (e.g., in lieu of castle model 150 ).
- the system may react to the difference between the actual physical model (e.g., castle 150 ) and the printed proxy, for example by reserving certain features or game capabilities associated with the castle anchor for sessions in which the actual castle model 150 is being used.
- a user inventory of anchors a user owns can be maintained in table 437 , allowing the system to keep track of which physical anchors (e.g., castle 150 ) a user owns.
- Table 437 implements a many-to-many relationship, with each record identifying that a particular user 432 owns a copy of a particular anchor 433 . This information can be useful if a server needs to suggest that a different anchor be used because a requested anchor is already committed in a nearby session.
- the count field would generally be ‘1’, but if a user owned duplicate anchors, the count could indicate how many copies.
- Table 435 stores records about the status of anchors that have been joined to (that is, associated with) the current session.
- the specific anchors detected are identified by relationship 431 , and the current configuration of each is noted. If a newly detected anchor is duplicative of another already in use at this site, the server may recommend a different configuration than the one currently in place (for example, where player 303 removed one of the towers from castle 160 ), or disallow use of the anchor (for example, if all the configurations are already in use nearby, or there are no alternative configurations). If an anchor is accepted into a session, the corresponding joined anchor record is provided with a JoinedAnchorlD that provides a unique reference to that specific anchor so that other participating AR devices and server 142 can refer to the anchor unambiguously.
- the record is further augmented with the identification of which AR device (in table 410 ) controls the game state of this anchor.
- other information about the anchor may be included, for instance its role (e.g., some anchors are for active play, some may be decoration, some anchors may have been detected, but not joined into the game).
- Status can comprise many components, as in FIG. 2 , anchor 150 (shown as castle images 251 , 252 ) exhibits a status being attacked by a dragon (shown as dragon images 212 , 222 ) and being on fire in the second story of the back side of the main tower (resulting in flame images 213 , 223 ).
- table 435 includes a timebase initialized by the controlling AR device (or alternatively by server 142 ) so that other AR devices have the opportunity to synchronize their representations of status applied to each anchor.
- each activity represented in the status of the detected anchor may have its own timebase.
- each participating AR device may begin tracking, recording, and reporting its range, bearing, and orientation with respect to each of the joined anchors. This is shown in table 415 , which uniquely relates an AR device (via relationship 411 ) to a joined anchor (via relationship 432 ) and then logs range, bearing, and orientation, relative to that anchor in the corresponding record.
- table 415 uniquely relates an AR device (via relationship 411 ) to a joined anchor (via relationship 432 ) and then logs range, bearing, and orientation, relative to that anchor in the corresponding record.
- one or more joined anchors may be out of an AR device's current field of view, in which case, the information presented therein may become stale. For this reason, a field is also maintained to record the last update time, so that an entry that is several minutes old might be recognized as possibly being out of date.
- the AR device may extrapolate its relationship to that anchor by considering information from its location sensors 319 (e.g., accelerometers, gyros, compass, etc.), which may be sufficient for some tracking purposes, at least for a short while.
- a flag (not shown) in each record may indicate whether or not the record is extrapolated, and when its last actual observation of the anchor occurred.
- Detected station table 445 keeps track of wireless stations 140 that an AR device is currently detecting.
- the MAC (media access control) address can be used to identify such stations unambiguously, but a separate station ID is used to keep individual track of the detection records, since more than one AR device might simultaneously be detecting each station.
- the strength of that detection may also be noted, which may serve as an indicator of distance from that station.
- one AR device detecting a particular station with a strong signal may or may not be considered proximal to another AR device detecting that same station with a much weaker signal, for the purposes of this invention.
- a longer-term record for stations historically detected may be kept in station table 440 , with which records in detected stations table 445 are associated by relationship 441 . If a station reports its location, or is otherwise able to be associated with a latitude and longitude, this data can be recorded in these records, but is not strictly required, though it can be useful for determining the location of an AR device.
- FIG. 5 shows a portion of the master database 500 and its association with the various local databases 400 maintained by each AR device.
- Records 410 B, 415 B, 425 B, 435 B, and 445 B represent individual records received from or sent to individual AR devices, corresponding to their similarly numbered tables (without the ‘B’).
- Received records (or messages having similar content) provide updates to the master database 500 , while updates to the individual AR devices can take similar form when sent and keep all the session participants updated.
- server 142 Upon receipt by server 142 , such records can be forwarded to other AR devices participating in the same session, though in some implementations this need not be done if the information is already communicated to the other AR devices by the originating AR device directly.
- Table 501 tracks ongoing sessions and their current location, for example by determining their latitude and longitude of one or more of the participating AR devices.
- a field may be included to identify which player ‘owns’ the session (not shown in table 501 ), for instance for use implementations where a new player attempting to join a session is to be a session-owner moderated event, then the session must be able to identify its owner so that the owner can decide whether to admit the new player.
- a record 410 B When a record 410 B is received, it can be associated with a pre-existing record in table 510 if relationship 512 identifies a record, otherwise, then a record 410 B can be inserted into table 510 , which is used to track devices present associated with ongoing session.
- the SessionlD field forms relationship 511 associated each AR device with exactly one session.
- an AR device in represented by a record in table 510 is participating in a session, there will be (in this embodiment) exactly one corresponding player in table 520 associated with each AR device via relationship 523 .
- Station detected records 445 B through relationship 542 populate and update records in table 540 for tracking stations that seems to be near the session identified by relationship 541 .
- Server 142 may keep a long-term database of stations identified in table 550 , which may be populated via relationship 545 , and updated (in case latitude or longitude changes, as might be the case if a station gets relocated or the estimate of its position is refined).
- Table 515 contains records analogous to those in table 415 and is populated and updated by the records 415 B received from individual AR device. (In FIG. 5 , the record 415 B is shown redacted due to space concerns, though the whole of the record is transferred). In records 415 B, the two fields ARDevicelD and JoinedAnchorlD form a composite foreign key used to form relation 517 to identify records in table 515 . Relationship 516 between the device-in-session table 510 and table 515 associates AR devices and the record of their physical position relative to the active anchors listed in table 530 and associated via relationship 533 .
- the insertion or update of records into the corresponding tables 510 , 520 , 530 , and 540 includes the additional information of with which session to associate the incoming record.
- the session identity is explicitly provided by the AR device sending the record, or whether the server 142 determines the session identity on the basis of the AR device sending the message (or the IP address from which the message is sent) is an implementation detail readily solved by those skilled in the art.
- Table 560 is the authoritative catalog of anchors
- Example process 600 for starting or enrolling in a session and joining anchors to that session is shown in FIG. 6 , wherein at 601 an AR device, e.g., 110 , is directed by the user to initiate or join a session.
- location sensors 319 e.g., GPS or other locator service
- the location of the AR device is obtained or estimated, if possible, and recorded in local database 400 , e.g., in the corresponding record in table 410 .
- any stations detected, such as station 140 are logged in local database 400 , in records of table 445 . If the stations are new, they can be added to table 440 also. If the location could not be determined at 602 , or is otherwise considered to be of low accuracy, a currently or previously determined location corresponding to the stations currently detected at 603 may be used to refine the estimate of the current location.
- a transaction is attempted with server 142 , to request a new session, or to join an already existing session.
- information concerning the AR device including its location, is provided.
- the location information may comprise some or all of the stations currently (or recently) detected, as logged in table 445 .
- Server 142 accepts any request for a new session and responds to a request to join a session with a list of available sessions proximal to the location of the AR device. If only one nearby session is available, the response may be automatically joined to that session without a further query from server 142 back to the requestor as to which nearby session to join.
- the initiator of a session controls who may join the session
- permission to join the session may wait for a response from the session owner.
- the choice of which session to join may be made using the player name of the owner or in response to an invitation (not shown) from the owner, which might identify the session by the SessionlD field in table 501 , or similar mechanism.
- anchors such as castle 150
- image processor module 318 As corresponding to an anchor record in table 430 .
- a check is made by examining local database 400 to see if that anchor is already listed in joined anchors table 435 , in the configuration noted by image processor module 318 . If not, then the anchor is new to this session and server 142 is queried at 607 to determine whether the new anchor will be admitted.
- Server 142 determines whether the new anchor is allowable by checking to see whether any copies of that anchor in the observed configuration are already registered in table 530 to another session having a location too close by (as determined by policy) or which is associated with nearby stations listed in table 540 that are also associated with the current session requesting the anchor. If either of these is the case, the anchor in its current configuration is rejected at 609 .
- the server 142 may recommend a particular different configuration for the anchor detected at 605 , which might be computed by the server by examining nearby sessions for all configurations of the same anchor, and removing those from a list of possible configurations as determined by the pattern information in the anchor catalog 560 .
- the server 142 may report this, too, whereby the player is saved from fruitlessly trying numerous configurations that would not have a chance of succeeding.
- a further computation could determine for which anchors in the players personal inventory, as represented by table 437 , there are available configurations, though this may require server 142 obtaining a copy of table 437 , which could be achieved by the above methods used to synchronize the various tables. The result of such advice would be that the player is provided with likely-to-succeed suggestions for other anchors that might be used.
- server 142 may set the controlling AR device field in the corresponding joined anchor record to identify the requesting AR device, or may choose a different AR device if the requestor is becoming overloaded (e.g., for being the controller of too many anchors).
- the anchor and configuration recognized at 605 may be found to already be represented in table 435 , which may be because it was previously added following an allowance at 608 or because it was already a part of the session before this AR device joined in and was populated during 604 . If this is the case, then at 606 the anchor is determined to not be new, and here too, processing continues at 610 .
- the controlling AR device field in the corresponding joined anchor record in table 435 is examined. If the field indicates that this AR device is the controller, then at 611 the AR devices begins to initialize the game processes corresponding to that anchor as defined in at least the corresponding record in anchor catalog 430 . As the anchor management activity at 611 has been dispatched (e.g., on other processing threads), process 600 continues at 613 as before. In some embodiments, this same initialization may be performed when a newly assigned controller receives an update message 435 B in which it is assigned the controllership of a new anchor, but one added to the session by a different AR device. In still other embodiments, an AR device might be passed the controllership of an anchor by the previously controlling AR device, for example when the previous controller is leaving the session, or communication with it has otherwise been lost.
- interaction with the anchor may be performed. This is especially true in subsequent passes, once the game gets going.
- any interaction with the anchor is directed to its current controller, either directly via station 140 (or other communication mechanism, e.g., BluetoothTM), or via server 142 .
- the interactions that can take place directly at 611 or indirectly at 612 are how the anchor and its augmented reality components are made interactive. For example, a player may direct the defenders of castle 150 to counterattack the dragon, or task them with knocking back the fire. Interactions with the AR rendition of the castle may allow walls to be opened so that the activities inside can be examined, etc.
- the augmented reality display may present controls for the augmented anchors that increase their shields, fire their weapons, or otherwise provide an interactive interface to the anchor's augmented rendition. The capabilities of such interactions are the essence of the game design.
- Some interactions may be limited to certain players, regardless of which AR device is the controller. For example, the player designated as the occupant of the castle (a game-specific state not explicitly represented
- outside of the joined anchor's ‘status’ or ‘role’ fields) may be allowed defensive interactions, e.g., closing the gates, placing the guard on alert, etc., and provided with comprehensive status and reporting, since all, or at least most, of the virtual personnel in the castle are his to command.
- Another player may be provided with a different set of interactions, e.g., transacting in the castle marketplace, or sending a thief to scale the walls, or attacking it with a dragon.
- each anchor in the current field of view may be detected.
- the interactions with them whether at 611 or 612 represent the bulk of the activity of process 600 for most of the session's duration.
- an update may be made to the corresponding record in table 415 , and at least occasionally sent (e.g., every 10 seconds, up to a few times per second) to server 142 as update records 415 B which are then propagated to the other participating AR devices.
- the anchors are released, or assigned to other controlling AR devices, and this is communicated to server 142 . If the session is over, all associated records in tables 510 , 515 , 520 , 530 , 540 are deleted from master database 500 and tables 435 and 415 are cleared in local database 400 .
- FIG. 7 shows a different configuration of the system, where a common augmented reality is shared at two different sites, 710 and 750 .
- Each site has its own wireless station 740 and 751 , respectively, each having communication through the Internet 741 to the shared augmented reality server 742 having database 743 comprising, in one example embodiment, master database portion 500 , discussed above.
- AR device 720 having wireless communication link 723 to station 740 .
- AR device 720 comprising touchscreen 722 , has field of view 724 , which at least occasionally includes anchor 730 (another castle) and anchor 731 (an arch).
- AR device 760 having wireless communication link 763 to station 751 .
- AR device 760 has field of view 764 that at least occasionally includes anchor 770 (another castle which need not be distinct from anchor 770 , since they are at different sites), and anchor 771 (another arch).
- the two arches 731 & 771 are given the role in tables 435 and 530 of portals to their corresponding sites.
- the AR software running on processor 310 can link the two sites 710 and 750 in a shared augmented reality for players 711 and 751 . This is achieved by first collecting and then analyzing the records in table 415 . On occasions where AR device 760 simultaneously sees and recognizes both of anchors 770 and 771 , the physical relationship between the two anchors 770 and 771 is established, as well as the relationship between those anchors and AR device 760 . This information is relayed to AR device 720 , which can map the arch 771 at site 750 as being in coincidence with the other side of arch 731 at site 710 .
- the augmented reality image 810 shown in detail in FIG. 8 , can be generated:
- a graphic rendition 870 of the castle corresponding to remote anchor 770 can be provided by graphics engine module 312 , along with a rendition 860 of player 751 as a game character located in coincidence with the AR position and attitude of remote AR device 760 .
- the castle image 830 is obtained of local castle 730 , and the dragon 812 and fire 813 graphically generated as previously discussed.
- the rendition 860 of a character whose gaze is directed by the field of view of another player's AR device (here, 760 ) provides insight into what the other player is thinking, studying, watching, controlling, even though that player is remotely located and potentially not otherwise viewable.
- the plane of the portal may be defined with respect to an anchor, but not necessarily be coincident with it. That is, for example, the portal that joins the two sites in the augmented reality may be a vertical plane that is set back some distance, say three feet, from a particular anchor, say the castle 730 , in a particular direction, say the direction of the main gate.
- arch 731 is not needed.
- anchors that exist physically only at the other site e.g., castle 770
- the AR image 810 in FIG. 8 would be the same, except the image of arch 831 would not be required, since arch 831 was not needed to define the portal.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Business, Economics & Management (AREA)
- Computer Security & Cryptography (AREA)
- General Business, Economics & Management (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computing Systems (AREA)
- Geometry (AREA)
- Processing Or Creating Images (AREA)
- Telephonic Communication Services (AREA)
- Information Transfer Between Computers (AREA)
Abstract
A multiuser, collaborative augmented reality (AR) system employs individual AR devices for viewing real-world anchors, that is, physical models that are recognizable to the camera and image processing module of the AR device. To mitigate ambiguous configurations when used in the collaborative mode, each anchor is registered with a server to ensure that only uniquely recognizable anchors are simultaneously active at a particular location. The system permits collaborative AR to span multiple sites, by associating a portal with an anchor at each site. Using the location of their corresponding AR device as a proxy for their position, AR renditions of the other participating users are provided. This AR system is particularly well suited for games.
Description
- This application is a continuation of pending application Ser. No. 17/208,241 filed on Mar. 22, 2021, which is a continuation of application Ser. No. 16/459,620 filed on Jul. 02, 2019 which issued as U.S. Pat. No. 10,957,111, which is a continuation of application Ser. No. 15/721,956 filed on Oct. 1, 2017 and issued as U.S. Pat. No. 10,339,717, which is a continuation of application Ser. No. 14/309,821 filed on Jun. 19, 2014 and issued as U.S. Pat. No. 9,779,548, which claims Priority from Provisional Application 61/839,364 filed on Jun. 25, 2013, each of which are incorporated by reference herein in their entirety.
- The present invention relates generally to an augmented reality system. More particularly, it relates to a system and method for allowing multiple participants to share in an augmented reality session, based on the real-world location of one or more anchors.
- Augmented reality (AR) toys have been introduced. The basic hardware required, an AR device, is a smartphone, laptop, tablet computer, dedicated device, or the like, that views the real world with a camera, and displays that view on a display to an user. Additionally, the AR device runs software that recognizes certain objects (herein termed “anchors”) in the camera's field of view, including the anchor's apparent distance and orientation relative to the camera, and then augments the image from the camera with computer generated graphics that seem to appear in, on, or near the anchor. The resulting modified image is displayed to the user (often, a game player, in which case, the AR device may also be executing game code).
- An example of such toys include a line of jigsaw puzzles, by Ravensburger AG of Ravensburg, Germany which when viewed with the corresponding AR app on an smartphone or tables replaces a portion of the puzzle with animated figures. Another one is called “AR Games” by Nintendo Co., Ltd. of Kyoto, Japan, which provides preprinted paper cards as anchors and plays on their Nintendo 3DS handheld gaming system. The Eclipse AR Card Game by MoOn-Studio of also uses preprinted cards as anchors, and allows two players to interact by manipulating the cards, however the two players share a single camera and display attached to a personal computer provides the same view to both players.
- None of these games offers each of several players their own AR device (e.g., smartphone, tablet, or game system) to allow the players to each view the same anchors and see a mutually coherent augmented reality.
- Further, in cases where potentially duplicative anchors coexist in proximity to each other, prior AR systems do not resolve the resulting ambiguity.
- None of the existing AR systems insert a representation of the users into the AR in place of their own AR device, whether by recognizing the AR device as an anchor itself, or by having an AR device report its orientation and distance from another anchor.
- Finally, existing AR systems do not offer an ability to cojoin a first site having anchors and AR devices with anchors and AR devices at a second site, so as to give users at the distinct sites an apparently coherent AR world.
- The object of the present invention is to allow several players, each with their own AR device, to share a mutually coherent augmented reality.
- It is a further object of the present invention to mitigate the ambiguities that would otherwise occur if two like anchors were used near each other.
- It is a still further object of this invention to allow sharing of a mutually coherent augmented reality from otherwise distinct sites.
- Finally, it is an additional object of this invention to render into the augmented reality a representation of the users, whether AR devices are recognized as anchors by other AR devices, or by having an AR device report to other AR devices its own orientation and distance with respect to one or more other anchors
- The aspects of the present invention will be apparent upon consideration of the following detailed description taken in conjunction with the accompanying drawings, in which like referenced characters refer to like parts throughout, and in which:
-
FIG. 1 is a block diagram of a collaborative AR system being used in a situation where there are two sessions - going on, one with one player, and one with two players;
-
FIG. 2 illustrates the detailed displays from the three AR devices in the system ofFIG. 1 ; -
FIG. 3 shows one example block diagram for an AR device, as might be implemented using a tablet; -
FIG. 4 is one example for a portion of a database accumulated and maintained locally by an AR device for operation in conjunction with other AR devices; -
FIG. 5 is one example database maintained by a server for facilitating joint AR sessions and mitigating collisions of identical anchors; -
FIG. 6 show one process for acquiring AR anchors for use in a shared AR session; -
FIG. 7 shows block diagram of another embodiment for a collaborative AR system, where multiple participants use anchors and AR devices at separate sites, yet share a coherent AR experience; and, -
FIG. 8 illustrates a detailed display from one AR device in the system ofFIG. 7 . - While the invention will be described and disclosed in connection with certain preferred embodiments and procedures, it is not intended to limit the invention to those specific embodiments. Rather it is intended to cover all such alternative embodiments and modifications as fall within the spirit and scope of the invention.
-
FIG. 1 shows threeAR users site 100, in this case players of an AR game, each having acorresponding AR device - Each of the
AR devices button 111 andtouchscreen display 112. Each has a camera offering corresponding fields ofview wireless connections wireless station 140 which offers connectivity, for example via Internet 141, toserver 142 having communication withdatabase 143. -
FIG. 1 shows twotoy castles AR devices castles AR devices anchor 150 could not be sure that there were not two castles that looked alike (castle 150 and a duplicate), or if there was only onecastle 150. - In the present invention, even though the anchors are not strictly unique, the potential ambiguity is resolved by communication with
server 142 in which each AR device registers the anchors it recognizes in conjunction with the general location of the AR device, for example as determined by GPS (Global Positioning System), cellular telephone or WiFi antenna proximity, etc. In this case, each of theAR devices wireless station 140. An example series of transactions might go like this: First, AR device is directed byuser 101 to start a session.AR device 110 recognizes an anchor (castle 150) and registers it in conjunction with the session just started with the location information identifying wireless station 140 (or perhaps GPS information obtained directly byAR device 110 or attributed to wireless station 140). Subsequently,player 102 attempts to play withAR device 120.Player 102 can either explicitly join the session ofplayer 101, orAR device 120 can recognize the anchor that iscastle 150 and report it toserver 142. IfAR device 120reports anchor 150 in conjunction with the location as determined by AR device 120 (which can include location information identifying wireless station 140),server 142 can respond that the anchor embodied incastle 150 is already assigned to a session belonging toplayer 101 and question whetherplayer 102 wants to join the session ofplayer 101. In a case whereAR device 120 reports that it is joining the session ofAR device 110, the anchors reported byAR device 120 are considered to be the same as like anchors reported by AR device 110 (i.e., castle 150). - However,
player 103 may choose to start a session of his own, usingAR device 130.AR device 130 determines its location, for example being proximate towireless station 140, or by GPS coordinate, etc., and recognizescastle 160 as an anchor in its field ofview 134. Initially, take the anchor provided bycastle 160 as being a duplicate ofcastle 150, that is, the AR devices cannot distinguish between them. WhenAR device 130 attempts to register castle 160 (then a duplicate of castle 150), at the same location ascastle 150 is already registered,server 142 will ask whetherplayer 103 is attempting to join the session ofplayer 101. Here,player 103 is not attempting to join a session already started, rather is attempting to start a second, independent session (which perhaps his friends, soon to arrive, will play with him). In this circumstance,server 142 must deny the registration of still-identical castle 160 to a separate session. It could be confused withcastle 150 and actions byplayer 103 could inadvertently and adversely affect game play corresponding tocastle 150. Instead,player 103 could use a different anchor (not shown here) or may modifycastle 160 to make it recognizably distinct fromcastle 150 to the AR devices. In this example embodiment,castle 150 has twotowers castle 160, once modified by removing a tower, has only one remaining, tower 161 (the tower removed fromcastle 160 is not shown). Instead of modifying towers, markers or labels (not shown) applied to various parts ofcastle 160 could be switched, or reoriented, or added. Pennants, small colored or patterned flags arranged on flagpoles where they can be easily seen and identified by the AR devices and recognition software, might be added, removed, or altered. - As an anchor is being registered for a new session,
server 142 might even propose the modification to be made. For example,server 142 might prescribe a particular pennant or sequence of pennants to a player starting a session. That way, any player directing his AR device at an anchor so configured will be automatically entered into the corresponding session having substantially the same location.Server 142 might prescribe the configuration for every anchor being added to a session or may prescribe the configuration only for those anchors which seem to conflict with anchors already registered in session with a sufficiently similar location. - In this case, “sufficiently similar location” would be established by policy. If GPS is used, this could mean within the accuracy of the GPS reading, plus an error band, times two, which might be 100 m or some other predetermined value, which may vary by the location itself (e.g., larger in cities and smaller in rural areas where GPS can be more accurate). The same is true of location services that use the cellular telephone network antennas for triangulation of location. If proximity to WiFi is used for location, “sufficiently similar” might be when both AR devices can detect the same WiFi antenna, though policy might limit that to detection with more than a predetermined signal strength (e.g., “two bars”, so many dBm, or milliwatts).
- Once multiple players have joined a common session, as with
players AR devices FIG. 2 , whereAR devices touchscreen display 112 ofAR device 110 showsimage 210 in whichcastle 150 is shown from the vantage of field ofview 114 ascastle image 251. The AR software has embellished this view of the castle with a computer generateddragon 212 just having setfire 213 to the far side of the castle.AR device 120displays image 220, showingcastle 150 as seen in field ofview 124. The augmented reality elements of the dragon and fire are seen here, but sinceplayer 102 has a view of the back side of the castle,castle image 252 is seen with flames 223 coming out the near side withdragon 222 shown in the foreground. - In some embodiments, when
AR device 120 is in the field ofview 114 ofAR device 110,AR device 120 may be recognized as an anchor having the role associated with player 202. In such a case, inaugmented reality image 210, the anchor that isAR device 120 might be replaced by anappropriate visage 214 representing the viewpoint of the other player. - In an alternative embodiment, the AR devices need not be recognized as anchors themselves, nor ever be detected by the other AR devices. In such a case, each AR device may determine its range from and orientation with respect to one or more anchors (e.g., castle 150) in the session, and report this information to the other AR devices, either directly or through
server 142. In this way, even whenAR device 110 does not fall within the field ofview 124 ofAR device 120, inAR image 220,AR device 120 can place anindicator 224 to show in what direction the other player(s), i.e.,player 101, are located. This can be useful in a game context, to help a player anticipate what another player is thinking or doing, or where is attention is directed. This feature is particularly valuable with respect to games conducted across multiple sites, where direct viewing of the opposing player is not available, as discussed below in conjunction withFIGS. 7 and 8 . - Also in
FIG. 2 , theaugmented reality image 230 displayed byAR device 130 is show, withimage 263 ofcastle 160. Here, no dragon is attacking the castle. Instead, a computer generatedsiege engine 232 is attacking thecastle image 263, andplayer 103 is engaged in a session separate from that whichplayers -
FIG. 3 provides an example block diagram forAR device 110. Processor (CPU) 310 usesmemory 311 in which is stored the AR application software, corresponding data, and operating system code.Processor 310 directsgraphic engine 312 which drivesdisplay 313, which is overlayed bytouch panel 314 which together formtouchscreen 112.Touch panel 314 is monitored bytouch interface electronics 315, which report user interactions back toprocessor 310. Other user interface devices, such asbutton 111 are monitored by other I/O interface 316, also reporting toprocessor 310.Camera 317 provides an image for field ofview 114, which is made available to image processor 318 (which could be a software module instead of hardware) wherein the recognition of anchors is performed.Location sensor 319 may comprise one or more of a GPS receiver, accelerometers (for detecting the attitude of AR device 110), gyroscopes (for quickly detecting the magnitude of sudden changes, but also for detecting very slow changes), flux gate compasses (to detect bearing relative to the Earth's magnetic field), etc., all of which have communication withprocessor 310, either directly or, in some embodiments, through intermediary utility processors (not shown). Finally,processor 310 has access tonetwork communication channel 320, for example, a wireless connection as tostation 140, through which communication is available withserver 142 and in some embodiments, other AR devices (e.g., 120). - In some embodiments the functions of
server 142 may be performed by one of the AR devices, for example the AR device that starts a session, or may be - distributed among some or all of the AR devices at a location. For example, an AR device attempting to start a new session might broadcast a signal soliciting information about other sessions or anchors already in use. The response, if any, might indicate which anchors are already in use, in which sessions, and may further suggest other alternate configurations of selected anchors that might be used instead.
-
FIG. 4 shows one example ofportion 400 of a local database used to keep track of which anchors, wireless stations, and the AR devices known to one AR device (e.g., 110). While illustrated using a relational database connection, those skilled in the art will recognize that many other alternative choices for data representation could be applied and that this representation was chosen for clarity in this example. Device table 410 identifies each distinct AR device (including itself) by a unique identifier, ARDevicelD, which could be a globally unique identifier (GUID) or universally unique identifier (UUID), well known in the art, or other identifier that will not be repeated among other AR devices in the system. The ARDevicelD for the device itself might be provided at the time of manufacture, or byserver 142 whenAR device 110 first registers itself with the system. The table 410 also includes information relating to one or more communication addresses for each device, e.g., an IP (Internet Protocol) address, MAC (Media Access Control) address, etc. This allows individual AR devices to communicate with other AR devices without going thoughserver 142. One or more security keys may be provided, for example a public/private key, so as to allow a server to communicate securely and authoritatively to an AR device (for example, to supply a software update or carry out a financial transaction), in which case the AR device holds the public key for the server and the server uses its private key to sign or encrypt messages it sends; or if a message is to be authenticated as coming from the AR device, the AR device would use its private key to sign or encrypt the message and the public key (which is the only portion distributed to the server or other devices), would be used to authenticate the message upon receipt. The current attitude and orientation in space (e.g., as determined by accelerometers, gyros, and magnetic compasses in location sensor 319) would be stored in the attitude field, whether obtained from its own sensors or received in communications from other AR devices. Likewise, latitude and longitude, if available whether from GPS or other location service, would be noted. Exactly one player identity (discussed below in conjunction with table 425) is associated via relationship 412 whenever the AR device is participating in a shared session but need not be otherwise. Other fields, not shown, for example, when each AR device was first or most recently contacted might also be recorded here, for example, to determine the age of the attitude and position information. - Table 420 stores information about users of the device, for example their username and authentication. There might be only one record in this table, if the device has only one user. Each user record in table 420 may correspond to zero or more player personae, recorded in player table 425, each having a unique player ID. Each player record is associated with, at most, one user record by the UserlD field that forms relationship 421 (Note: Players on other AR devices need not have a corresponding user record on this AR device). Additionally, the player name field records the in-game name for the player, which could be their game character's name. Other fields (not shown) might include or reference a player-specific icon or image; for example, each player might have a unique visage for use when augmented reality scenes are rendered (e.g., for
visage 214, if it is to be specific to the player, rather than generic or determined by some other game mechanic). Player status, such as ‘INACTIVE’ for all players not currently playing, or other game-mediated states would be entered for the local player and could be sent to other AR devices during a session. Likewise, messages describing players received from other AR devices would be gathered here. - In one embodiment, anchors table 430 is constant, or could be authoritatively updated, for example by
server 142. In it, the collective records represent a canonical description of each of the available anchors. A representation of the patterns that identify each kind of anchor, including the possible configuration variations, is provided sufficient to inform theimage processing module 318 what to look for in the camera image, as to what constitutes an anchor to be recognized. Also included is a field describing in detail the geometry of the anchor (e.g., the geometric shape of castle 150), sufficient so thatgraphics engine module 312 can orient the geometry of the anchor to the recognized patterns, so that when the camera image is being augmented, the geometry model can be used to enhance the rendition of the final image. For example, inAR image 210, the geometry of the castle anchor is used to maskflames 213 so that they appear behind thecastle image 251 obtained from the camera, whereas inAR image 220, the flames 223 are in front of the castle, but appear to be emanating from a particular window. Additional information (not shown) concerning anchor-specific animations or embellishments that might be added under certain conditions (e.g., how rain should funnel off the roof, or how the walls should look after receiving a certain amount of damage), might also be recorded here. Updates to table 430 would occur if the manufacturer offers new anchors, or if updates to the existing catalog of anchors were to be made. - In some embodiments, alternative patterns may also be included. This would allow readily available objects (e.g., a can from a particular brand of soda) to be used in lieu of
castle model 150. In such a case, the geometry information in table 430 would be used to render thecastle image 251, because the un-augmented camera image would show a soda can. - In other embodiments, a catalog of anchors could be provided, but dynamically associated with a physical representation (i.e., the pattern), so that a user could make their own anchors, e.g., by printing a suitable barcode on paper, registering that barcode as a pattern associated with a particular anchor record, thereby allowing the printout to be used as the anchor (e.g., in lieu of castle model 150). This would allow a player new to the hobby to quickly set up a complex AR scenario having multiple anchors, without requiring that user to acquire multiple, elaborate physical models such as
castle 150. Later, if the user remains interested, the physical models can be acquired. The system may react to the difference between the actual physical model (e.g., castle 150) and the printed proxy, for example by reserving certain features or game capabilities associated with the castle anchor for sessions in which theactual castle model 150 is being used. - A user inventory of anchors a user owns can be maintained in table 437, allowing the system to keep track of which physical anchors (e.g., castle 150) a user owns. Table 437 implements a many-to-many relationship, with each record identifying that a
particular user 432 owns a copy of aparticular anchor 433. This information can be useful if a server needs to suggest that a different anchor be used because a requested anchor is already committed in a nearby session. The count field would generally be ‘1’, but if a user owned duplicate anchors, the count could indicate how many copies. - Table 435 stores records about the status of anchors that have been joined to (that is, associated with) the current session. The specific anchors detected are identified by
relationship 431, and the current configuration of each is noted. If a newly detected anchor is duplicative of another already in use at this site, the server may recommend a different configuration than the one currently in place (for example, where player 303 removed one of the towers from castle 160), or disallow use of the anchor (for example, if all the configurations are already in use nearby, or there are no alternative configurations). If an anchor is accepted into a session, the corresponding joined anchor record is provided with a JoinedAnchorlD that provides a unique reference to that specific anchor so that other participating AR devices andserver 142 can refer to the anchor unambiguously. The record is further augmented with the identification of which AR device (in table 410) controls the game state of this anchor. Likewise, other information about the anchor may be included, for instance its role (e.g., some anchors are for active play, some may be decoration, some anchors may have been detected, but not joined into the game). Status can comprise many components, as inFIG. 2 , anchor 150 (shown ascastle images 251, 252) exhibits a status being attacked by a dragon (shown asdragon images 212, 222) and being on fire in the second story of the back side of the main tower (resulting inflame images 213, 223). The corresponding controlling AR device would be executing the appropriate code so that the status advances appropriately, e.g., the fire spreads, the dragon may come under attack by the castle defenders, etc., which would further update the status. Such updates would be propagated to other AR devices participating in the shared AR session so that theAR images anchor 150 there is an explosion, it is desirable for the video of the explosion and any sound that might correspond to that event be substantially synchronized, so thatplayers devices - Once an anchor has been joined into a session, each participating AR device may begin tracking, recording, and reporting its range, bearing, and orientation with respect to each of the joined anchors. This is shown in table 415, which uniquely relates an AR device (via relationship 411) to a joined anchor (via relationship 432) and then logs range, bearing, and orientation, relative to that anchor in the corresponding record. Often, one or more joined anchors may be out of an AR device's current field of view, in which case, the information presented therein may become stale. For this reason, a field is also maintained to record the last update time, so that an entry that is several minutes old might be recognized as possibly being out of date. Even when an anchor is no longer in the field of view of an AR device, the AR device may extrapolate its relationship to that anchor by considering information from its location sensors 319 (e.g., accelerometers, gyros, compass, etc.), which may be sufficient for some tracking purposes, at least for a short while. A flag (not shown) in each record may indicate whether or not the record is extrapolated, and when its last actual observation of the anchor occurred.
- Detected station table 445 keeps track of
wireless stations 140 that an AR device is currently detecting. The MAC (media access control) address can be used to identify such stations unambiguously, but a separate station ID is used to keep individual track of the detection records, since more than one AR device might simultaneously be detecting each station. The strength of that detection may also be noted, which may serve as an indicator of distance from that station. By policy, one AR device detecting a particular station with a strong signal may or may not be considered proximal to another AR device detecting that same station with a much weaker signal, for the purposes of this invention. A longer-term record for stations historically detected may be kept in station table 440, with which records in detected stations table 445 are associated by relationship 441. If a station reports its location, or is otherwise able to be associated with a latitude and longitude, this data can be recorded in these records, but is not strictly required, though it can be useful for determining the location of an AR device. -
FIG. 5 shows a portion of themaster database 500 and its association with the variouslocal databases 400 maintained by each AR device.Records master database 500, while updates to the individual AR devices can take similar form when sent and keep all the session participants updated. Upon receipt byserver 142, such records can be forwarded to other AR devices participating in the same session, though in some implementations this need not be done if the information is already communicated to the other AR devices by the originating AR device directly. - Table 501 tracks ongoing sessions and their current location, for example by determining their latitude and longitude of one or more of the participating AR devices. In some embodiments, a field may be included to identify which player ‘owns’ the session (not shown in table 501), for instance for use implementations where a new player attempting to join a session is to be a session-owner moderated event, then the session must be able to identify its owner so that the owner can decide whether to admit the new player.
- When a
record 410B is received, it can be associated with a pre-existing record in table 510 if relationship 512 identifies a record, otherwise, then arecord 410B can be inserted into table 510, which is used to track devices present associated with ongoing session. In table 510, the SessionlD field formsrelationship 511 associated each AR device with exactly one session. - Player records 425B, through relationship 522, populate and update records in table 520 for tracking players currently in the ongoing sessions in table 501 and identified by
relationship 521. When an AR device in represented by a record in table 510 is participating in a session, there will be (in this embodiment) exactly one corresponding player in table 520 associated with each AR device viarelationship 523. - Station detected
records 445B, throughrelationship 542 populate and update records in table 540 for tracking stations that seems to be near the session identified byrelationship 541.Server 142 may keep a long-term database of stations identified in table 550, which may be populated viarelationship 545, and updated (in case latitude or longitude changes, as might be the case if a station gets relocated or the estimate of its position is refined). - Joined
anchors records 435B, through relationship 532, populate and update the anchors-in-session table 530, associated a joined anchor with a particular session identified byrelationship 531. - Table 515 contains records analogous to those in table 415 and is populated and updated by the
records 415B received from individual AR device. (InFIG. 5 , therecord 415B is shown redacted due to space concerns, though the whole of the record is transferred). Inrecords 415B, the two fields ARDevicelD and JoinedAnchorlD form a composite foreign key used to formrelation 517 to identify records in table 515.Relationship 516 between the device-in-session table 510 and table 515 associates AR devices and the record of their physical position relative to the active anchors listed in table 530 and associated viarelationship 533. - In each case, for
inbound records server 142 determines the session identity on the basis of the AR device sending the message (or the IP address from which the message is sent) is an implementation detail readily solved by those skilled in the art. - Table 560 is the authoritative catalog of anchors
- that can be used to keep tables 430 in
local database 400 up to date as new anchors become available from the manufacturer. -
Example process 600 for starting or enrolling in a session and joining anchors to that session is shown inFIG. 6 , wherein at 601 an AR device, e.g., 110, is directed by the user to initiate or join a session. At 602, using location sensors 319 (e.g., GPS or other locator service) the location of the AR device is obtained or estimated, if possible, and recorded inlocal database 400, e.g., in the corresponding record in table 410. At 603, any stations detected, such asstation 140, are logged inlocal database 400, in records of table 445. If the stations are new, they can be added to table 440 also. If the location could not be determined at 602, or is otherwise considered to be of low accuracy, a currently or previously determined location corresponding to the stations currently detected at 603 may be used to refine the estimate of the current location. - At 604, a transaction is attempted with
server 142, to request a new session, or to join an already existing session. As part of the transaction, information concerning the AR device, including its location, is provided. The location information may comprise some or all of the stations currently (or recently) detected, as logged in table 445.Server 142 accepts any request for a new session and responds to a request to join a session with a list of available sessions proximal to the location of the AR device. If only one nearby session is available, the response may be automatically joined to that session without a further query fromserver 142 back to the requestor as to which nearby session to join. If, as a matter of policy, the initiator of a session (the session's “owner”) controls who may join the session, then permission to join the session may wait for a response from the session owner. In some embodiments, discussed below in conjunction withFIGS. 7 & 8 , one might join a session that is not nearby, in which case the choice of which session to join may be made using the player name of the owner or in response to an invitation (not shown) from the owner, which might identify the session by the SessionlD field in table 501, or similar mechanism. - Upon joining a session in progress, information in those records in tables 510, 515, 520, 530, and 540 having an association with the session just joined are sent to the AR device just joining, thereby adding records corresponding to the session tables 410, 415, 425, 435, and 445, allowing an augmented reality to be presented coherently over the multiple participating AR devices.
- Once having initiated or joined a session at 604, processing continues at 605 where anchors, such as
castle 150, are detected when corresponding field ofview 114 ofcamera 317 captures an image of the anchor (castle 150), which is then recognized byimage processor module 318 as corresponding to an anchor record in table 430. At 606 a check is made by examininglocal database 400 to see if that anchor is already listed in joined anchors table 435, in the configuration noted byimage processor module 318. If not, then the anchor is new to this session andserver 142 is queried at 607 to determine whether the new anchor will be admitted. -
Server 142 determines whether the new anchor is allowable by checking to see whether any copies of that anchor in the observed configuration are already registered in table 530 to another session having a location too close by (as determined by policy) or which is associated with nearby stations listed in table 540 that are also associated with the current session requesting the anchor. If either of these is the case, the anchor in its current configuration is rejected at 609. Theserver 142 may recommend a particular different configuration for the anchor detected at 605, which might be computed by the server by examining nearby sessions for all configurations of the same anchor, and removing those from a list of possible configurations as determined by the pattern information in theanchor catalog 560. If there are no available configurations for that particular anchor, theserver 142 may report this, too, whereby the player is saved from fruitlessly trying numerous configurations that would not have a chance of succeeding. A further computation could determine for which anchors in the players personal inventory, as represented by table 437, there are available configurations, though this may requireserver 142 obtaining a copy of table 437, which could be achieved by the above methods used to synchronize the various tables. The result of such advice would be that the player is provided with likely-to-succeed suggestions for other anchors that might be used. Once the proposed anchor has been rejected at 609, processing tests to see whether the session has been terminated at 613, and if not, loops back to 605. If at 608 the determination fromserver 142 is that the anchor is allowed, then a corresponding record is added to master joined anchors table 530 and propagated viatransmission record 435B to be recorded in the joined anchor table 435 of each of the AR devices participating in the same session.Server 142 may set the controlling AR device field in the corresponding joined anchor record to identify the requesting AR device, or may choose a different AR device if the requestor is becoming overloaded (e.g., for being the controller of too many anchors). - After a new anchor is allowed at 608, processing continues at 610. At 606, the anchor and configuration recognized at 605 may be found to already be represented in table 435, which may be because it was previously added following an allowance at 608 or because it was already a part of the session before this AR device joined in and was populated during 604. If this is the case, then at 606 the anchor is determined to not be new, and here too, processing continues at 610.
- At 610, the controlling AR device field in the corresponding joined anchor record in table 435 is examined. If the field indicates that this AR device is the controller, then at 611 the AR devices begins to initialize the game processes corresponding to that anchor as defined in at least the corresponding record in
anchor catalog 430. As the anchor management activity at 611 has been dispatched (e.g., on other processing threads),process 600 continues at 613 as before. In some embodiments, this same initialization may be performed when a newly assigned controller receives anupdate message 435B in which it is assigned the controllership of a new anchor, but one added to the session by a different AR device. In still other embodiments, an AR device might be passed the controllership of an anchor by the previously controlling AR device, for example when the previous controller is leaving the session, or communication with it has otherwise been lost. - At 611, interaction with the anchor, other than initializing it, may be performed. This is especially true in subsequent passes, once the game gets going.
- If at 610 the management of the present anchor is not the responsibility of the current AR device (i.e., the controlling AR device field does not match the current AR device), then at 612, any interaction with the anchor is directed to its current controller, either directly via station 140 (or other communication mechanism, e.g., Bluetooth™), or via
server 142. - The interactions that can take place directly at 611 or indirectly at 612 are how the anchor and its augmented reality components are made interactive. For example, a player may direct the defenders of
castle 150 to counterattack the dragon, or task them with knocking back the fire. Interactions with the AR rendition of the castle may allow walls to be opened so that the activities inside can be examined, etc. In other scenarios with different anchors, for example representing spaceships, the augmented reality display may present controls for the augmented anchors that increase their shields, fire their weapons, or otherwise provide an interactive interface to the anchor's augmented rendition. The capabilities of such interactions are the essence of the game design. - Some interactions may be limited to certain players, regardless of which AR device is the controller. For example, the player designated as the occupant of the castle (a game-specific state not explicitly represented
- in the database as illustrated, outside of the joined anchor's ‘status’ or ‘role’ fields) may be allowed defensive interactions, e.g., closing the gates, placing the guard on alert, etc., and provided with comprehensive status and reporting, since all, or at least most, of the virtual personnel in the castle are his to command. Another player may be provided with a different set of interactions, e.g., transacting in the castle marketplace, or sending a thief to scale the walls, or attacking it with a dragon.
- As before, at 613, if the session has not ended, the process returns to 605. Each anchor in the current field of view may be detected. As the session progresses, only previously added anchors are being identified at 606, and the interactions with them, whether at 611 or 612 represent the bulk of the activity of
process 600 for most of the session's duration. Also at either 611 or 612, if the range, bearing, or orientation of the AR device with respect to the detected anchor has changed, then an update may be made to the corresponding record in table 415, and at least occasionally sent (e.g., every 10 seconds, up to a few times per second) toserver 142 asupdate records 415B which are then propagated to the other participating AR devices. - If at 613 the session has ended, or the player of the current AR device has decided to quit, then at 614 the anchors are released, or assigned to other controlling AR devices, and this is communicated to
server 142. If the session is over, all associated records in tables 510, 515, 520, 530, 540 are deleted frommaster database 500 and tables 435 and 415 are cleared inlocal database 400. -
FIG. 7 shows a different configuration of the system, where a common augmented reality is shared at two different sites, 710 and 750. Each site has itsown wireless station Internet 741 to the sharedaugmented reality server 742 havingdatabase 743 comprising, in one example embodiment,master database portion 500, discussed above. - At
site 710,player 711 usesAR device 720 havingwireless communication link 723 tostation 740.AR device 720, comprisingtouchscreen 722, has field ofview 724, which at least occasionally includes anchor 730 (another castle) and anchor 731 (an arch). - At
site 750,player 751 usesAR device 760 havingwireless communication link 763 tostation 751.AR device 760 has field ofview 764 that at least occasionally includes anchor 770 (another castle which need not be distinct fromanchor 770, since they are at different sites), and anchor 771 (another arch). - In this configuration, the two
arches 731 & 771 are given the role in tables 435 and 530 of portals to their corresponding sites. With that definition, the AR software running onprocessor 310 can link the twosites players AR device 760 simultaneously sees and recognizes both ofanchors anchors AR device 760. This information is relayed toAR device 720, which can map the arch 771 atsite 750 as being in coincidence with the other side ofarch 731 atsite 710. With this mapping, theaugmented reality image 810, shown in detail inFIG. 8 , can be generated: Agraphic rendition 870 of the castle corresponding toremote anchor 770 can be provided bygraphics engine module 312, along with arendition 860 ofplayer 751 as a game character located in coincidence with the AR position and attitude ofremote AR device 760. Meanwhile, thecastle image 830 is obtained oflocal castle 730, and thedragon 812 andfire 813 graphically generated as previously discussed. - The
rendition 860 of a character whose gaze is directed by the field of view of another player's AR device (here, 760) provides insight into what the other player is thinking, studying, watching, controlling, even though that player is remotely located and potentially not otherwise viewable. - In another embodiment, the plane of the portal may be defined with respect to an anchor, but not necessarily be coincident with it. That is, for example, the portal that joins the two sites in the augmented reality may be a vertical plane that is set back some distance, say three feet, from a particular anchor, say the
castle 730, in a particular direction, say the direction of the main gate. In this embodiment,arch 731 is not needed. Thus, when viewed through an AR device, anchors that exist physically only at the other site (e.g., castle 770) would appear through the AR device to lie beyond a portal three feet in front of the local toy castle's main gate. TheAR image 810 inFIG. 8 would be the same, except the image ofarch 831 would not be required, sincearch 831 was not needed to define the portal. - The particular implementations described, and the discussions regarding details, and the specifics of the figures included herein, are purely exemplary; these implementations and the examples of them, may be modified, rearranged and/or enhanced without departing from the principles of the present invention. In particular, the computer architectures, data structures, and flowcharts herein are exemplary, and significant alteration can be made without departing from the spirit of the invention. As previously mentioned, the choice of a relational database for the representation and explanation of example data management and structures can be readily substituted with other equally effective mechanisms.
- Particular features of user interface, image processing module, graphics engine module, and the capabilities of the databases, will depend on the architecture used to implement a system of the present invention, the operating systems of the servers, and AR devices, the CPU selected, and the software code written for them. It is not necessary to describe the details of such programming to permit a person or team of ordinary skill in the art to implement the application, user interface, and services suitable for implementing a system within the scope of the present invention. The details of the software design and programming necessary to implement the principles of the present invention are readily understood from the description herein.
- Various additional modifications to the embodiments of the invention, specifically illustrated and described herein, will be apparent to those skilled in the art, particularly in light of the teachings of this invention. Further, it will be apparent that the functionality of this invention can be incorporated into and function from within the context of other products, including an e-commerce system. It is intended that these cover all modifications and embodiments that fall within the spirit and scope of the invention. Thus, while preferred embodiments of the present invention have been disclosed, it will be appreciated that it is not limited thereto but may be otherwise embodied within the scope of the following claims.
Claims (20)
1. A method for providing a shared augmented reality (AR), comprising:
recognizing a first anchor at a first position at a first site, by a first processor in communication with a first camera of a first AR device, on the basis of a first information describing the first anchor;
determining, by the first processor, a second position of the first AR device on the basis of at least the recognizing, the second position relative to the first position; and,
presenting, by a first display of the first AR device driven by direction of the first processor, a first AR view that shows a virtual object to appear to be at a third position, the third position relative to the first position, the virtual object associated with an updating status available to the first processor, the status comprising a description of the third position; the first AR view based on at least the second position and the status;
wherein the virtual object in the first AR view is in synchronization with the virtual object shown in a second AR view displayed by a second AR device, and the second AR view is based on at least the status.
2. The method of claim 1 , wherein the determining is on the further basis of a sensor of the first AR device in communication with the first processor, the sensor selected from an accelerometer, a gyroscope, and a magnetic compass.
3. The method of claim 1 , wherein
the second AR device is at the first site, the first anchor is recognized on the basis of the first information by a second processor in communication with a second camera of the second AR device, a fourth position of the second AR device is determined by the second processor, the second AR view further shows the virtual object to appear to be at the third position, and the second AR view is further based on the fourth position.
4. The method of claim 3 wherein the first information is predetermined.
5. The method of claim 3 , wherein the first processor has communication with the second processor, the method further comprising:
sharing the first information, by the first processor with the second processor.
6. The method of claim 3 , wherein the first processor has communication with a server, the method further comprising:
receiving the first information, by the first processor, from the server.
7. The method of claim 3 , wherein the first processor has communication with the second processor, the method further comprising:
sharing the status, by the first processor with the second processor.
8. The method of claim 3 , wherein the first processor has communication with a server, the method further comprising:
receiving the status, by the first AR device, from the server.
9. The method of claim 3 , further comprising:
receiving the fourth position, by the first processor as a communication from the second processor;
wherein the first AR view further shows a visage to appear to be at the fourth position, the visage represents a viewpoint of a user of the second AR device, the first AR view is further based on the fourth position.
10. The method of claim 1 , wherein
a second anchor is described by a second information and is at a fifth position at the first site, the second AR device is at the first site, the second anchor is recognized on the basis of the second information by a second processor in communication with a second camera of the second AR device, a fourth position of the second AR device is determined by the second processor, the fourth position relative to the fifth position, a third information available to the second processor describes a physical relationship between the first position and the fifth position, the second AR view further shows the virtual object to appear to be at the third position, and the second AR view is further based on the third information and the fourth position.
11. The method of claim 10 wherein the first and second information is predetermined.
12. The method of claim 10 wherein the first processor has communication with the second processor, the method further comprising:
sharing the first, second, and third information by the first processor with the second processor.
13. The method of claim 10 wherein the first processor has communication with a server, the method further comprising:
receiving the first information, by the first processor, from the server.
14. The method of claim 10 , wherein the first processor has communication with the second processor, the method further comprising:
sharing the status, by the first processor with the second processor.
15. The method of claim 10 , wherein the first processor has communication with a server, the method further comprising:
receiving the status, by the first AR device from the server.
16. The method of claim 10 , further comprising:
receiving the fourth position, by the first processor as a communication from the second processor;
wherein the first AR view further shows a visage to appear to be at the fourth position, the visage represents a viewpoint of a user of the second AR device, the first AR view is further based on the fourth position.
17. The method of claim 1 , wherein
a second anchor is described by a second information and is at a fifth position at a second site, the second AR device is at the second site, the second anchor is recognized on the basis of the second information by a second processor in communication with a second camera of the second AR device, a fourth position of the second AR device relative to the fifth position is determined by the second processor, a third information available to the second processor describes a mapping of the fifth position to the shared augmented reality, the second AR view further shows the virtual object to appear to be at the third position, and the second AR view is further based on the third information and the fourth position.
18. The method of claim 17 , further comprising:
receiving the fourth position, by the first processor as a communication from the second processor; and,
wherein a fourth information available to the first processor describes a mapping of the first position to the shared augmented reality, the first AR view further shows a visage to appear to be at the fourth position, the visage represents a viewpoint of the second AR device, the first AR view is further based on the fourth position and the fourth information.
19. The method of claim 1 , further comprising the step of:
e) accepting an interaction with a user interface having communication with the processor, wherein the interaction at least partially determines the updating status.
20. A non-transient computer readable medium containing instructions that when executed by a processor perform the method of claims 1 .
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/584,319 US11580710B2 (en) | 2013-06-25 | 2022-01-25 | Multiuser augmented reality method |
US18/108,531 US20240144609A1 (en) | 2013-06-25 | 2023-02-10 | Location-based augmented reality method |
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201361839364P | 2013-06-25 | 2013-06-25 | |
US14/309,821 US9779548B2 (en) | 2013-06-25 | 2014-06-19 | Multiuser augmented reality system |
US15/721,956 US10339717B2 (en) | 2013-06-25 | 2017-10-01 | Multiuser augmented reality system and method |
US16/459,620 US10957111B2 (en) | 2013-06-25 | 2019-07-02 | Collaborative augmented reality presentation method |
US17/208,241 US11263822B2 (en) | 2013-06-25 | 2021-03-22 | Multiuser augmented reality method |
US17/584,319 US11580710B2 (en) | 2013-06-25 | 2022-01-25 | Multiuser augmented reality method |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/208,241 Continuation US11263822B2 (en) | 2013-06-25 | 2021-03-22 | Multiuser augmented reality method |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/108,531 Continuation US20240144609A1 (en) | 2013-06-25 | 2023-02-10 | Location-based augmented reality method |
Publications (2)
Publication Number | Publication Date |
---|---|
US20220143503A1 true US20220143503A1 (en) | 2022-05-12 |
US11580710B2 US11580710B2 (en) | 2023-02-14 |
Family
ID=52110546
Family Applications (6)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/309,821 Active - Reinstated 2034-08-21 US9779548B2 (en) | 2013-06-25 | 2014-06-19 | Multiuser augmented reality system |
US15/721,956 Active 2034-07-24 US10339717B2 (en) | 2013-06-25 | 2017-10-01 | Multiuser augmented reality system and method |
US16/459,620 Active US10957111B2 (en) | 2013-06-25 | 2019-07-02 | Collaborative augmented reality presentation method |
US17/208,241 Active US11263822B2 (en) | 2013-06-25 | 2021-03-22 | Multiuser augmented reality method |
US17/584,319 Active US11580710B2 (en) | 2013-06-25 | 2022-01-25 | Multiuser augmented reality method |
US18/108,531 Pending US20240144609A1 (en) | 2013-06-25 | 2023-02-10 | Location-based augmented reality method |
Family Applications Before (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/309,821 Active - Reinstated 2034-08-21 US9779548B2 (en) | 2013-06-25 | 2014-06-19 | Multiuser augmented reality system |
US15/721,956 Active 2034-07-24 US10339717B2 (en) | 2013-06-25 | 2017-10-01 | Multiuser augmented reality system and method |
US16/459,620 Active US10957111B2 (en) | 2013-06-25 | 2019-07-02 | Collaborative augmented reality presentation method |
US17/208,241 Active US11263822B2 (en) | 2013-06-25 | 2021-03-22 | Multiuser augmented reality method |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/108,531 Pending US20240144609A1 (en) | 2013-06-25 | 2023-02-10 | Location-based augmented reality method |
Country Status (1)
Country | Link |
---|---|
US (6) | US9779548B2 (en) |
Families Citing this family (47)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8810598B2 (en) | 2011-04-08 | 2014-08-19 | Nant Holdings Ip, Llc | Interference based augmented reality hosting platforms |
US10262462B2 (en) | 2014-04-18 | 2019-04-16 | Magic Leap, Inc. | Systems and methods for augmented and virtual reality |
US9582516B2 (en) | 2013-10-17 | 2017-02-28 | Nant Holdings Ip, Llc | Wide area augmented reality location-based services |
WO2015072968A1 (en) * | 2013-11-12 | 2015-05-21 | Intel Corporation | Adapting content to augmented reality virtual objects |
US20150193979A1 (en) * | 2014-01-08 | 2015-07-09 | Andrej Grek | Multi-user virtual reality interaction environment |
WO2016077506A1 (en) | 2014-11-11 | 2016-05-19 | Bent Image Lab, Llc | Accurate positioning of augmented reality content |
US10799792B2 (en) * | 2015-07-23 | 2020-10-13 | At&T Intellectual Property I, L.P. | Coordinating multiple virtual environments |
US20170169617A1 (en) * | 2015-12-14 | 2017-06-15 | II Jonathan M. Rodriguez | Systems and Methods for Creating and Sharing a 3-Dimensional Augmented Reality Space |
US10311644B2 (en) * | 2016-12-14 | 2019-06-04 | II Jonathan M. Rodriguez | Systems and methods for creating and sharing a 3-dimensional augmented reality space |
WO2017165705A1 (en) * | 2016-03-23 | 2017-09-28 | Bent Image Lab, Llc | Augmented reality for the internet of things |
GB2550911B (en) * | 2016-05-27 | 2021-02-10 | Swap Bots Ltd | Augmented reality toy |
US10785443B2 (en) * | 2017-12-18 | 2020-09-22 | Streem, Inc. | Augmented reality video stream synchronization across a network |
US10380804B1 (en) | 2018-06-01 | 2019-08-13 | Imajion Corporation | Seamless injection of augmented three-dimensional imagery using a positionally encoded video stream |
US20190385372A1 (en) * | 2018-06-15 | 2019-12-19 | Microsoft Technology Licensing, Llc | Positioning a virtual reality passthrough region at a known distance |
CN110166799A (en) * | 2018-07-02 | 2019-08-23 | 腾讯科技(深圳)有限公司 | Living broadcast interactive method, apparatus and storage medium |
US11227435B2 (en) | 2018-08-13 | 2022-01-18 | Magic Leap, Inc. | Cross reality system |
US10957112B2 (en) | 2018-08-13 | 2021-03-23 | Magic Leap, Inc. | Cross reality system |
US10854004B2 (en) | 2018-08-24 | 2020-12-01 | Facebook, Inc. | Multi-device mapping and collaboration in augmented-reality environments |
JP7503542B2 (en) | 2018-10-05 | 2024-06-20 | マジック リープ, インコーポレイテッド | Rendering location-specific virtual content anywhere |
US11403823B2 (en) | 2019-02-14 | 2022-08-02 | Lego A/S | Toy system for asymmetric multiplayer game play |
US11090561B2 (en) | 2019-02-15 | 2021-08-17 | Microsoft Technology Licensing, Llc | Aligning location for a shared augmented reality experience |
US10997776B2 (en) * | 2019-02-23 | 2021-05-04 | Microsoft Technology Licensing, Llc | Connecting spatial anchors for augmented reality |
US10940387B2 (en) * | 2019-03-15 | 2021-03-09 | Disney Enterprises, Inc. | Synchronized augmented reality gameplay across multiple gaming environments |
US11097194B2 (en) | 2019-05-16 | 2021-08-24 | Microsoft Technology Licensing, Llc | Shared augmented reality game within a shared coordinate space |
CN112153083B (en) * | 2019-06-26 | 2022-07-19 | 浙江商汤科技开发有限公司 | Anchor point sharing method, device, system, electronic equipment and storage medium |
EP4046139A4 (en) | 2019-10-15 | 2023-11-22 | Magic Leap, Inc. | Cross reality system with localization service |
JP2022551734A (en) | 2019-10-15 | 2022-12-13 | マジック リープ, インコーポレイテッド | Cross-reality system that supports multiple device types |
JP2022551735A (en) | 2019-10-15 | 2022-12-13 | マジック リープ, インコーポレイテッド | Cross-reality system using wireless fingerprints |
WO2021096931A1 (en) | 2019-11-12 | 2021-05-20 | Magic Leap, Inc. | Cross reality system with localization service and shared location-based content |
CN114762008A (en) | 2019-12-09 | 2022-07-15 | 奇跃公司 | Simplified virtual content programmed cross reality system |
US11295511B1 (en) | 2020-02-11 | 2022-04-05 | Progress Software Corporation | Multi-user data presentation in AR/VR |
CN115398314A (en) | 2020-02-13 | 2022-11-25 | 奇跃公司 | Cross reality system for map processing using multi-resolution frame descriptors |
CN115398484A (en) | 2020-02-13 | 2022-11-25 | 奇跃公司 | Cross reality system with geolocation information priority for location |
WO2021163306A1 (en) | 2020-02-13 | 2021-08-19 | Magic Leap, Inc. | Cross reality system with accurate shared maps |
WO2021173779A1 (en) | 2020-02-26 | 2021-09-02 | Magic Leap, Inc. | Cross reality system with fast localization |
CN115803788A (en) | 2020-04-29 | 2023-03-14 | 奇跃公司 | Cross-reality system for large-scale environments |
US20230298279A1 (en) * | 2020-06-30 | 2023-09-21 | Surgical Theater, Inc. | Augmented reality shared anchoring system and method |
WO2022019955A1 (en) * | 2020-07-22 | 2022-01-27 | Google Llc | Methods and apparatus for adaptive augmented reality anchor generation |
CN112148122A (en) * | 2020-08-25 | 2020-12-29 | 中国电子科技集团公司第三十八研究所 | Third-party visual angle implementation method for wearable augmented/mixed reality equipment |
US11557097B2 (en) | 2020-11-13 | 2023-01-17 | Google Llc | Triggering a collaborative augmented reality environment using an ultrasound signal |
US11483533B2 (en) | 2021-01-05 | 2022-10-25 | At&T Intellectual Property I, L.P. | System and method for social immersive content rendering |
WO2022154847A1 (en) | 2021-01-12 | 2022-07-21 | Emed Labs, Llc | Health testing and diagnostics platform |
US11929168B2 (en) | 2021-05-24 | 2024-03-12 | Emed Labs, Llc | Systems, devices, and methods for diagnostic aid kit apparatus |
US11615888B2 (en) | 2021-03-23 | 2023-03-28 | Emed Labs, Llc | Remote diagnostic testing and treatment |
US11373756B1 (en) | 2021-05-24 | 2022-06-28 | Emed Labs, Llc | Systems, devices, and methods for diagnostic aid kit apparatus |
GB2623461A (en) | 2021-06-22 | 2024-04-17 | Emed Labs Llc | Systems, methods, and devices for non-human readable diagnostic tests |
US12014829B2 (en) | 2021-09-01 | 2024-06-18 | Emed Labs, Llc | Image processing and presentation techniques for enhanced proctoring sessions |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100191728A1 (en) * | 2009-01-23 | 2010-07-29 | James Francis Reilly | Method, System Computer Program, and Apparatus for Augmenting Media Based on Proximity Detection |
US20110164163A1 (en) * | 2010-01-05 | 2011-07-07 | Apple Inc. | Synchronized, interactive augmented reality displays for multifunction devices |
US20120198021A1 (en) * | 2011-01-27 | 2012-08-02 | Pantech Co., Ltd. | System and method for sharing marker in augmented reality |
US20120229508A1 (en) * | 2011-03-10 | 2012-09-13 | Microsoft Corporation | Theme-based augmentation of photorepresentative view |
US20120263154A1 (en) * | 2011-04-13 | 2012-10-18 | Autonomy Corporation Ltd | Methods and systems for generating and joining shared experience |
US8547401B2 (en) * | 2004-08-19 | 2013-10-01 | Sony Computer Entertainment Inc. | Portable augmented reality device and method |
US20130293468A1 (en) * | 2012-05-04 | 2013-11-07 | Kathryn Stone Perez | Collaboration environment using see through displays |
US20140015827A1 (en) * | 2012-07-13 | 2014-01-16 | Google Inc. | Sharing Photo Albums in Three Dimensional Environments |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8914897B2 (en) * | 2007-05-23 | 2014-12-16 | International Business Machines Corporation | Controlling access to digital images based on device proximity |
US8554784B2 (en) * | 2007-08-31 | 2013-10-08 | Nokia Corporation | Discovering peer-to-peer content using metadata streams |
US9153195B2 (en) * | 2011-08-17 | 2015-10-06 | Microsoft Technology Licensing, Llc | Providing contextual personal information by a mixed reality device |
-
2014
- 2014-06-19 US US14/309,821 patent/US9779548B2/en active Active - Reinstated
-
2017
- 2017-10-01 US US15/721,956 patent/US10339717B2/en active Active
-
2019
- 2019-07-02 US US16/459,620 patent/US10957111B2/en active Active
-
2021
- 2021-03-22 US US17/208,241 patent/US11263822B2/en active Active
-
2022
- 2022-01-25 US US17/584,319 patent/US11580710B2/en active Active
-
2023
- 2023-02-10 US US18/108,531 patent/US20240144609A1/en active Pending
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8547401B2 (en) * | 2004-08-19 | 2013-10-01 | Sony Computer Entertainment Inc. | Portable augmented reality device and method |
US20100191728A1 (en) * | 2009-01-23 | 2010-07-29 | James Francis Reilly | Method, System Computer Program, and Apparatus for Augmenting Media Based on Proximity Detection |
US20110164163A1 (en) * | 2010-01-05 | 2011-07-07 | Apple Inc. | Synchronized, interactive augmented reality displays for multifunction devices |
US20120198021A1 (en) * | 2011-01-27 | 2012-08-02 | Pantech Co., Ltd. | System and method for sharing marker in augmented reality |
US20120229508A1 (en) * | 2011-03-10 | 2012-09-13 | Microsoft Corporation | Theme-based augmentation of photorepresentative view |
US20120263154A1 (en) * | 2011-04-13 | 2012-10-18 | Autonomy Corporation Ltd | Methods and systems for generating and joining shared experience |
US20130293468A1 (en) * | 2012-05-04 | 2013-11-07 | Kathryn Stone Perez | Collaboration environment using see through displays |
US20140015827A1 (en) * | 2012-07-13 | 2014-01-16 | Google Inc. | Sharing Photo Albums in Three Dimensional Environments |
Also Published As
Publication number | Publication date |
---|---|
US20210209858A1 (en) | 2021-07-08 |
US20140375688A1 (en) | 2014-12-25 |
US10957111B2 (en) | 2021-03-23 |
US10339717B2 (en) | 2019-07-02 |
US11580710B2 (en) | 2023-02-14 |
US20180040167A1 (en) | 2018-02-08 |
US20190333279A1 (en) | 2019-10-31 |
US9779548B2 (en) | 2017-10-03 |
US11263822B2 (en) | 2022-03-01 |
US20240144609A1 (en) | 2024-05-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11580710B2 (en) | Multiuser augmented reality method | |
KR102212261B1 (en) | Verification of the player's real world position using activities in a parallel reality game | |
US8593535B2 (en) | Relative positioning of devices based on captured images of tags | |
US9573064B2 (en) | Virtual and location-based multiplayer gaming | |
US8675017B2 (en) | Real world gaming framework | |
US8611594B2 (en) | Dynamic display of virtual content on several devices using reference tags | |
EP2405626B1 (en) | Information processing system, information processing program, information processing apparatus, and information processing method | |
CN113597333B (en) | Verifying player real world locations using landmark image data corresponding to verification paths | |
JP5922897B2 (en) | Program, electronic device and server | |
US10271159B2 (en) | Information processing apparatus, information processing system, storage medium having stored therein information processing program, and information processing method | |
KR101388426B1 (en) | Online and offline linked game system | |
US20230162433A1 (en) | Information processing system, information processing method, and information processing program | |
WO2019009163A1 (en) | Information processing system, player-side device, control method, and program | |
JP5661729B2 (en) | Terminal identification method, server device, and program | |
JP7108202B2 (en) | Game program, game system, and server device | |
CN109417651B (en) | Generating challenges using location-based gaming companion applications | |
JP5728124B2 (en) | Terminal identification method, server device, and program | |
JP2020120707A (en) | Game system, mobile terminal, game apparatus, information processing unit, and game program | |
KR20190078187A (en) | Apparatus and method for sending overwhelming information, apparatus and method for displayng overwhelming information |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |