WO2024072608A1 - Moteur de création, de capture et de lecture de monde xr - Google Patents

Moteur de création, de capture et de lecture de monde xr Download PDF

Info

Publication number
WO2024072608A1
WO2024072608A1 PCT/US2023/032015 US2023032015W WO2024072608A1 WO 2024072608 A1 WO2024072608 A1 WO 2024072608A1 US 2023032015 W US2023032015 W US 2023032015W WO 2024072608 A1 WO2024072608 A1 WO 2024072608A1
Authority
WO
WIPO (PCT)
Prior art keywords
record
action
build
user
creator
Prior art date
Application number
PCT/US2023/032015
Other languages
English (en)
Inventor
Rachel Cross
Patricia DOOLEY
Jean CHIN
Katerina VASILIOU
Tiffany Madruga
Sarah BARRICK
Original Assignee
Meta Platforms Technologies, Llc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US18/068,975 external-priority patent/US20240112418A1/en
Application filed by Meta Platforms Technologies, Llc filed Critical Meta Platforms Technologies, Llc
Publication of WO2024072608A1 publication Critical patent/WO2024072608A1/fr

Links

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B19/00—Teaching not covered by other main groups of this subclass
    • G09B19/0053—Computers, e.g. programming
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/131—Protocols for games, networked simulations or virtual reality

Definitions

  • the present disclosure is generally directed to systems and methods for capturing, recording, and playing back steps for building objects and structures in an artificial reality (XR) world.
  • XR artificial reality
  • an artificial reality environment that includes real-world objects and/or two-dimensional (2D) and/or three-dimensional (3D) virtual objects.
  • the artificial reality environment can be a virtual environment depicted by a virtual reality (VR) device showing a set of virtual objects.
  • the artificial reality environment can be a mixed reality environment with real-world objects and virtual objects supplemented over the real-world objects.
  • a user can view the objects in the artificial reality environment and modify content in the artificial reality environment.
  • XR systems can provide intuitive ways of viewing, navigating through, and interacting with objects in an XR environment
  • Creating new structures can involve generating 2D surfaces, modifying 3D objects, moving those objects around (e.g., aligned with other objects, attached to other objects, etc.), configuring the properties of those objects, and/or a variety of other steps.
  • Users wishing to learn XR world building tools may resort to reading through documentation and/or watching tutorial videos outside of the XR environment, later returning to the XR world to attempt to reproduce those steps from memory.
  • Such traditional learning processes can be cumbersome and frustrating to users, making it difficult for them to learn how to use build tools in an XR world.
  • a method for generating a build record in a first artificial reality (XR) system comprising: capturing an action performed by a creator within a first XR environment; determining that the action is a recordable action based on the action affecting one or more properties of an object within the first XR environment; recording a data sub-record representative of the captured action; generating, based at least in part on the data sub-record and one or more other data sub-records, a build record representing one or more recordable actions performed by the creator when creating the object within the first XR environment; and transmitting a copy of the build record to a server; wherein the build record is played back on a second XR system by: retrieving the build record, representing one or more recorded actions performed by the creator and including one or more sub-records including the data sub-record; and for each sub-record in the build record, causing an avatar in a second XR environment different from
  • the data sub-record may include an audio recording and/or a written annotation, from the creator, which is provided when the avatar in the second XR environment performs the corresponding action for the data sub-record.
  • the build record may comprise a series of instructions, stored as a JSON or XML data structure, the can be executed to create an instance of the object.
  • the creator may set an element of captured action, that the data sub-record is representative of, as modifiable based on a provided parameter.
  • a second user other than the creator may provide the parameter to customize the performance of the action, for the second XR environment, in terms of one or more of selecting a particular style, linking to a specific URL or blockchain address, selecting a color, selecting a material, or any combination thereof.
  • the second user may provide the parameter through a menu specifying options for the action.
  • the creator may set an element of captured action, that the data sub-record is representative of, as modifiable based on a provided parameter.
  • a second user other than the creator may provide the parameter to customize the performance of the action, for the second XR environment, in terms of one or more of selecting a particular style, linking to a specific URL or blockchain address, selecting a color, selecting a material, or any combination thereof.
  • the creator may set an element of captured action, that the data sub-record is representative of, as modifiable based on a provided parameter.
  • a second user other than the creator may provide the parameter to customize the performance of the action for the second XR environment.
  • Performing the action, in the second artificial reality environment, represented by the sub-record may include providing in the second artificial reality environment playback tools to control the play back of the one or more recorded actions.
  • the playback tools may include a tool to control a playback speed, a tool to pause playback, and a tool to jump backwards or forwards in the one or more recorded actions.
  • a computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to carry out the method of the first aspect.
  • a computing system for generating a build record in a first artificial reality (XR) system, the computing system comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform the method of the first aspect.
  • XR artificial reality
  • a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of the first aspect.
  • Figure 1 is a conceptual diagram illustrating an example user interface (Ul) of a XR world build playback application.
  • Figure 2 is a conceptual diagram illustrating an example XR world build playback implementation in which a ghost of a creator’s avatar is rendered alongside the user.
  • Figure 3 is a flow diagram illustrating a process used in some implementations for capturing a creator’s XR world build actions.
  • Figure 4 is a flow diagram illustrating a process used in some implementations for playing back a creator’s XR world build actions to a user.
  • Figure 5 is a block diagram illustrating an overview of devices on which some implementations of the present technology can operate.
  • Figure 6A is a wire diagram illustrating a virtual reality headset which can be used in some implementations of the present technology.
  • Figure 6B is a wire diagram illustrating a mixed reality headset which can be used in some implementations of the present technology.
  • Figure 6C is a wire diagram illustrating controllers which, in some implementations, a user can hold in one or both hands to interact with an artificial reality environment.
  • Figure 7 is a block diagram illustrating an overview of an environment in which some implementations of the present technology can operate.
  • a unique aspect of artificial reality lies in the ability to create and explore expansive and immersive virtual worlds — both renditions of real-world locations, and new fanciful worlds that exist only within the metaverse.
  • Many artists, architects, and other world builders that discover XR want to build their own virtual worlds to create interactive experiences, games, art expositions, and a variety of other experiences.
  • building interactive objects and environments in an XR world can require skills and knowledge about a specific XR universe’s building tools, leading many users to consult documentation, video tutorials, and other resources outside of the XR environment to learn how to build new and exciting objects and environments in their XR world.
  • additional hardware e.g., video capture cards
  • software e.g., screen recorders
  • video capture software and/or hardware might be used to record a video of themselves building a particular object or structure in an XR world and subsequently upload that video to a video sharing website as educational content for other users to learn from.
  • the typical methods for generating new educational content for building XR worlds involved additional cost to the creator and the use of multiple, disconnected systems.
  • the process might involve the user removing a VR headset, going to their computer or mobile device, looking up some information, storing the steps in their short-term memory, putting the VR headset back on, and later attempting to recall the steps back in the XR environment.
  • Such a cumbersome process can make learning slow and challenging for users.
  • learning how to use computer software or do other real-world tasks learning how to use XR world build tools has previously involved switching in and out of the XR environment, which can be jarring to users and make the process challenging and unintuitive.
  • aspects of the present disclosure provide systems and methods to enable the capture and playback of actions performed in an XR environment by a creator, instructor, or other user.
  • a creator in an XR environment can trigger or initiate a capture mode with an XR build capture and playback engine (hereinafter “XR world build engine” or “engine”).
  • the engine can monitor the creator’s movements, menu invocations, menu selections, object manipulation, object configuration, and/or any other recordable action.
  • the engine can log that action (along with metadata, audio recording, annotations, and/or any other associated information) as a part of the capture session.
  • Some inputs by the creator may be specifically disregarded by the engine (i.e., actions not related to the creation of the object or structure in the XR world, similar to parametric modeling in computer aided design systems).
  • the creator can instruct the engine to exit capture mode and end the session.
  • Each action can be stored as data in an appropriate format (e.g., JSON, XML, blobs, binary, etc.) in the order that they were performed, such that they can serve as a series of instructions (i.e., similar to a macro or script) that can reproduce the steps the creator performed to create the object or structure.
  • the creator can edit one or more of the recorded actions after ending the capture session. For instance, the creator can add text and/or audio annotations linked to a corresponding action or sequence of actions (e.g, to provide supplementary explanations as to the significance of the actions performed).
  • the creator may have performed a particular action that they wish to parameterize or otherwise make modifiable by users upon playback of the build record, such as selecting a particular style, linking an object to a specific URL or blockchain address, or some other action that the creator foresees users wishing to modify.
  • the engine can allow one or more aspects of an action to be user-adjustable during playback, thereby allowing the user to personalize their learning experience and gain a deeper understanding of the XR world build tools.
  • the engine can store the recorded actions, annotations, and metadata as a build record on a server.
  • the build record may be made accessible to other users via a sharing service accessible within the XR world. For instance, a user might want to learn how to build a transparent window for their house in their XR world, and can open a sharing service within the XR application to discover tutorials and other build records that can be used to play back one or more techniques for building windows with the XR world building tools.
  • the engine can obtain the data from the server, along with any associated metadata and/or annotations.
  • the engine can, in some implementations, interpret the build record or otherwise convert portions of the build record into instructions (e.g., a script, SDK or API function calls, etc.).
  • the engine can place the user in an XR world and provide them with playback tools to play back the sequence of recorded actions.
  • each action may be performed automatically by the user’s avatar, such that the user experiences the actions from a first-person perspective.
  • the engine can allow the user to play back each step at a desired speed, pause the playback, jump backwards or forwards to a different step, and/or otherwise step through the build record at their own pace. For instance, the user can allow the engine to take control of their avatar to perform a step (e.g., resize a surface or object, snap one object to another, etc.), step backwards to “undo” that step, and then repeat the process manually to do it themselves. In this manner, the engine can provide an interactive and immersive learning experience, which does not involve the user exiting the XR world and use a different device to read documentation or watch a tutorial video.
  • a step e.g., resize a surface or object, snap one object to another, etc.
  • the user can switch from a first-person perspective to a different perspective (e.g., third person view, a free-floating camera view, etc.). For instance, if an object manipulation is not entirely visible from the first-person perspective, the user can “detach” from the avatar and move to a different perspective to better examine the effects of a given action.
  • a given action e.g., an action for adding indoor lighting elements to the user’s house may create an effect visible from outside the house, and the user may wish to see the visual effect the lights have from different angles or from outside the house.
  • the engine can allow the user to view the performance of the actions and/or the results of those actions from different perspectives to facilitate a better understanding of the build process and/or the XR world.
  • the user can configure the engine to play back the build record on a different “ghost” avatar — which is separate from the user’s avatar — to enable the user to observe the build process from an external viewpoint.
  • the engine can provide inputs that allow the user to control the “ghost” or teacher avatar in a manner similar to the above, first-person avatar playback example. However, by having a separate teacher avatar perform the actions, the user can observe and repeat those actions separately, in a manner similar to attending a class in the real world (e.g., watching an instructor perform a task and attempting to replicate that task independently).
  • the user may be placed in a sandboxed environment, such that the entire process of following along with an instructor does not affect the user’s XR world.
  • the user may be prompted during playback of the build script to configure the parameterized actions (e.g., either at the beginning of playback, or upon playback of the parameterized action).
  • the engine can provide a menu of options to the user and/or request a manual input from the user to personalize the build record to the user.
  • Such parameterized actions may be desirable if the user wants to use the resulting object or structure in their own XR world without having to repeat the steps a second time.
  • the user may be able to configure the action parameter for linking to their NFT (as opposed to whichever NFT was linked to by the creator of the build record). Then, when playback is finished, the engine can store the completed object to allow the user to easily add the prebuilt NFT object to their XR world at a later time without having to repeat the actions in the build record again.
  • NFT non-fungible token
  • a build record may be provided or sold in a marketplace as an automated script for generating non-standard objects or structures in an XR world.
  • Creators might develop custom staircase designs, unique pieces of furniture, interactive artwork installments, and/or other custom objects or structures that do not exist in a default or stock library of objects or structures.
  • creators may wish to monetize their custom designs by selling them in a marketplace to other users, who can purchase build records for educational purposes and/or to quickly replicate a design created by another user (i.e., similar to a Ul design kit for frontend Ul libraries).
  • Such build records may be parameterized to allow customers to configure the designs to suit their preferences before running the build record through the engine to generate the custom object or structure.
  • the XR world build capture and playback engine can facilitate the recording and playback of build actions from creators, all without having to leave the XR environment. In this manner, users can engage in more “hands-on” learning within the XR environment, as opposed to leaving the XR environment and referring to other sources outside of their XR world.
  • the actions taken to build a custom object or structure can be played back by the XR application itself, thereby enabling dynamic, interactive tutorials that users can engage with intuitively and in an immersive manner.
  • capturing build actions as data rather than as a video enables the parameterization of certain build actions, thereby allowing the same build record to be used to generate different objects and/or structures.
  • creators can develop a variety of new objects and structures, expanding the available resources for new users to learn how to build fun and interactive XR worlds of their own.
  • Figure 1 is a conceptual diagram 100 illustrating an example user interface (Ul) of a XR world build playback application. More particularly, Figure 1 illustrates an example playback mode of a recording of furniture building that includes captured actions previously performed by a creator.
  • the Ul includes a main window 110 in which the user’s avatar’s first-person perspective is rendered. In this example, menu and cursor elements 114 and 116 (corresponding the user’s left and right hands, respectively) are shown within the main window 110.
  • the Ul also includes a playback controls section 118, which can include a combination of buttons and inputs for controlling the playback of the build recording, the speed of the playback, and/or provide shortcuts to jump backwards or forwards across actions within the build record.
  • the Ul includes a build actions timeline 120, which includes an ordered sequence of icons corresponding to different actions performed in the course of the captured build process.
  • the current playback location may be denoted by a tick 122 or other graphical element, indicating to the user which step they are viewing in the build process.
  • One or more of the actions in the build record may be associated with annotations captured by or manually recorded by the creator of the build record — such as annotations 124 and 126 — which can provide additional information about the choice to perform that action, more detailed explanations about how those actions work, and/or other relevant information to the user.
  • the Ul can also include an option to change the perspective shown in the main window 110, such as perspective button 112.
  • pressing button 112 may cycle through two or more different perspectives (e.g., first person, third person, free camera, etc.).
  • the user can pause playback and freely control the avatar to interact with the XR environment, access menus, activate functions, and/or otherwise engage in whatever actions the user desires (e.g., repeating the steps of the build process to learn how to use the build tools).
  • the user can, in some cases, alter steps of the build process by pausing playback, performing their own action(s), and then resuming playback thereafter. In this manner, unlike reading documentation or watching a video guide, the user can easily transition between viewing an action and doing the action themselves, all without having to leave the XR environment.
  • FIG. 2 is a conceptual diagram 200 illustrating an example XR world build playback implementation in which a ghost of a creator’s avatar 220 is rendered alongside the user’s avatar 210. More particularly, the user is following along a build record that guides the user through the process of creating customized, realistic windows that can be added to their house in their XR world.
  • the user has chosen to instantiate the creator’s avatar 220 as separately from their own avatar 210, so that they can watch the creator’s avatar 220 step through the process of creating custom windows and then independently repeat those steps to make the window objects themselves.
  • the creator’s avatar 220 has its own menu palette 222, which is independent and separate from the menu palette 212 of the user’s avatar 210.
  • XR environment 202 may be a sandbox environment which is unrelated to the user’s XR world.
  • the user is observing the creator’s avatar 220 add a shiny reflection aesthetic to the glass of the creator’s window 224 — which the user may subsequently attempt to add to their own window 214 after learning how to do so.
  • the user can watch the creator’s avatar 220 in a similar manner as a student observing a teacher perform a task in the real world, which may be a more intuitive or natural learning experience for some users (compared to having the user’s own avatar be controlled by the engine).
  • FIG. 3 is a flow diagram illustrating a process 300 used in some implementations for capturing a creator’s XR world build actions.
  • process 300 can be performed as a response to a creator’s request to begin capturing steps of a build process.
  • Process 300 can be performed by, for example, XR build capture and playback engine 564 of Figure 5, either on a server or combination of servers over a network, or locally on a user system or device, i.e., an XR device (e.g., a head-mounted display) or 2D device, such as a computer or other display or processing device.
  • some steps of process 300 can be performed on a server, while other steps of process 300 can be performed locally on a user device.
  • process 300 can be performed multiple times, repeatedly, consecutively, concurrently, in parallel, etc., as requests to capture and store build records on a variety of devices associated with a respective variety of creators.
  • Some steps of process 300 can described as recording data (e.g., actions, metadata, annotations, etc.), some of which may be performed client side within an XR application, while other steps (e.g., storing the build record at block 318) might be performed server side within an XR universe server.
  • process 300 initiates a capture mode.
  • a creator may invoke a menu option, press a button, or otherwise request to start capturing build actions as a part of a tutorial process and/or to add to a marketplace of custom object designs.
  • the capture mode may be performed within a separate sandboxed XR environment specifically instantiated for capturing build processes, or may be invoked within the creator’s XR world, depending on the particular implementation.
  • process 300 captures the creator performing an action. For instance, if the creator opens a new menu, creates a new surface, shape, or object, modifies a surface, shape, or object, adds or modifies properties to surface, shape, or object (e.g., collidability, visibility, gravity, physical properties, deep links, etc ), adds a script or linked actions to an object (e.g., upon interaction, proximity triggered, etc.), and/or performs any other action that meaningfully changes an object’s properties — process 300 can capture that action with sufficient fidelity to be able to reproduce that action automatically during playback.
  • an action e.g., collidability, visibility, gravity, physical properties, deep links, etc
  • adds a script or linked actions e.g., upon interaction, proximity triggered, etc.
  • process 300 determines whether the action performed by the creator was a recordable action.
  • an action performed by a creator may not meaningfully alter an objects properties, such as moving the creator’s avatar, looking around, clicking in and out of a menu without selecting a particular menu item, and/or a variety of other actions that do not affect the object or XR environment.
  • process 300 may determine that the action is recordable or non-recordable. This determination may be based on a predetermined list of recordable actions, based on a parametric modeling paradigm (e.g., did the action change an object’s properties?), and/or using other suitable heuristics.
  • process 300 may proceed to block 308, where the action is ignored or otherwise not recorded. Alternatively, if the action was a recordable action, process 300 proceeds to block 310, where process 300 records the action and its associated metadata, if any (e.g., annotations, audio segments, etc.).
  • process 300 records the action and its associated metadata, if any (e.g., annotations, audio segments, etc.).
  • process 300 determines whether to exit capture mode. For example, if the creator might manually select to exit capture mode when the build process is complete. If process 300 determines to exit capture mode, process 300 proceeds to block 314, where process 300 may record and store post-capture annotations. For example, the creator may end the capture session and proceed to add additional text and/or audio explanations for certain actions. The creator may choose not to add any additional annotations after ending the capture session.
  • process 300 may add parameters to one or more of the recorded actions.
  • the creator might end the capture session and modify certain recorded actions that the creator wants to be user-configurable (e.g., material selection, colors, styling, etc.).
  • the engine can provide tools to the creator to convert a hard-coded action into a parameterized one (e.g., on actions where a particular option was selected from among a list of possible options).
  • process 300 stores the build record.
  • process 300 (executing on an VR device, for instance) transmits a copy of the build record to a server.
  • the server may store a copy of the record in a file storage system and/or a database, which may be accessed by other services and/or marketplaces to be shared or made discoverable to other users.
  • Figure 4 is a flow diagram illustrating a process used in some implementations for playing back a creator’s XR world build actions to a user.
  • process 400 can be performed as a response to a user’s request to begin playing back steps of a build process.
  • Process 400 can be performed by, for example, XR build capture and playback engine 564 of Figure 5, either on a server or combination of servers over a network, or locally on a user system or device, i.e., an XR device (e.g., a head-mounted display) or 2D device, such as a computer or other display or processing device.
  • an XR device e.g., a head-mounted display
  • 2D device such as a computer or other display or processing device.
  • process 400 can be performed on a server, while other steps of process 400 can be performed locally on a user device. Although illustrated as having only one iteration, it is contemplated that process 400 can be performed multiple times, repeatedly, consecutively, concurrently, in parallel, etc., such as when a user repeats the playback of certain steps in a build process, reviews a build process multiple times, plays back steps in a different order than they were recorded, etc.
  • process 400 initiates a playback mode.
  • a user may select a particular build record from a repository, marketplace, or other sharing service within an XR application which launches a playback mode of an XR world build capture and playback engine.
  • An XR application may instantiate an XR environment, load one or more avatars, instantiate playback and viewing controls, and then wait for the user to start playback of the build record.
  • process 400 retrieves a stored build record.
  • Block 404 may be performed before, during, or after block 402.
  • process 400 might asynchronously retrieve the build record while also initializing an XR environment at block 406, and instantiating the playback controls at block 402.
  • process 400 instantiates an XR environment.
  • a user’s XR environment may render similar Ul elements as those shown in Figure 1.
  • process 400 determines the next action in the retrieved build record. Immediately after initialization, block 408 may determine that the next action is the first action in the build process. If the user starts normal playback, process 400 can iteratively process the next action, cause an avatar to perform the action (possibly modified based on user-provided parameters), process the next action, and so on.
  • process 400 may receive a user configuration for the next action parameter(s).
  • the user might be prompted to configure parameters of a particular action before it is processed.
  • the user may configure the parameters in advance, and process 400 retrieves the previously-captured user parameter preferences at block 410. Regardless, block 410 may only be performed when an action is parameterized.
  • process 400 obtains annotation(s) for the next action, if any. For example, a voiceover or audio recording may be retrieved and played in parallel with the action, so as to emulate an explainer video or tutorial.
  • process 400 controls an avatar to perform the next action in the build record.
  • process 400 can take control of the user’s avatar to perform the next action in the build record.
  • process 400 can control a creator or “ghost” avatar to perform the next action in the build record.
  • Causing an avatar to perform an action may itself involve a sequence of actions.
  • the process of resizing an object may involve selecting a vertex of the object, pinching that vertex, dragging the vertex while pinching, and then releasing the pinch.
  • action may encompass multiple sub-operations which, when performed sequentially or in concert, results in a macro-level action.
  • the engine may capture individual operations, identify a pattern or particular sequence of operations as representing a macro-level action, and then label that macro-level action as a single action even though performing that action involves multiple sub-steps.
  • process 400 determines whether there are more actions in the build record. If there are more actions in the build record, process 400 returns to block 408. However, if there are no more actions in the build record, process 400 may complete. In some cases, process 400 may remain in playback mode even if no more actions remain in the build record (i.e., playback has finished because it reached the end of the captured build process). But process 400 may not exit playback mode until the user explicitly ends the playback session, such that the user can step back through the actions, review a particular action in the build record again, play back the entire build process again from the beginning, etc.). Thus, it will be appreciated that the end of process 400 does not necessarily mean that the engine exits playback mode.
  • FIG. 5 is a block diagram illustrating an overview of devices on which some implementations of the disclosed technology can operate.
  • the devices can comprise hardware components of a device 500 that captures and plays back a build process for an XR object or structure.
  • Device 500 can include one or more input devices 520 that provide input to the Processor(s) 510 (e.g., CPU(s), GPU(s), HPU(s), etc.), notifying it of actions.
  • the actions can be mediated by a hardware controller that interprets the signals received from the input device and communicates the information to the processors 510 using a communication protocol.
  • Input devices 520 include, for example, a mouse, a keyboard, a touchscreen, an infrared sensor, a touchpad, a wearable input device, a camera- or imagebased input device, a microphone, or other user input devices.
  • Processors 510 can be a single processing unit or multiple processing units in a device or distributed across multiple devices. Processors 510 can be coupled to other hardware devices, for example, with the use of a bus, such as a PCI bus or SCSI bus.
  • the processors 510 can communicate with a hardware controller for devices, such as for a display 530.
  • Display 530 can be used to display text and graphics. In some implementations, display 530 provides graphical and textual visual feedback to a user.
  • display 530 includes the input device as part of the display, such as when the input device is a touchscreen or is equipped with an eye direction monitoring system. In some implementations, the display is separate from the input device.
  • Display devices are: an LCD display screen, an LED display screen, a projected, holographic, or augmented reality display (such as a heads-up display device or a headmounted device), and so on.
  • Other I/O devices 540 can also be coupled to the processor, such as a network card, video card, audio card, USB, firewire or other external device, camera, printer, speakers, CD-ROM drive, DVD drive, disk drive, or Blu-Ray device.
  • the device 500 also includes a communication device capable of communicating wirelessly or wire-based with a network node.
  • the communication device can communicate with another device or a server through a network using, for example, TCP/IP protocols.
  • Device 500 can utilize the communication device to distribute operations across multiple network devices.
  • the processors 510 can have access to a memory 550 in a device or distributed across multiple devices.
  • a memory includes one or more of various hardware devices for volatile and non-volatile storage, and can include both read-only and writable memory.
  • a memory can comprise random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard drives, floppy disks, CDs, DVDs, magnetic storage devices, tape drives, and so forth.
  • RAM random access memory
  • ROM read-only memory
  • writable non-volatile memory such as flash memory, hard drives, floppy disks, CDs, DVDs, magnetic storage devices, tape drives, and so forth.
  • a memory is not a propagating signal divorced from underlying hardware; a memory is thus non-transitory.
  • Memory 550 can include program memory 560 that stores programs and software, such as an operating system 562, XR build capture and playback engine 564, and other application programs 566. Memory 550 can also include data memory 570, e.g., build actions, parameters, annotations, audio segments, video, configuration data, settings, user options or preferences, etc., which can be provided to the program memory 560 or any element of the device 500.
  • program memory 560 that stores programs and software, such as an operating system 562, XR build capture and playback engine 564, and other application programs 566.
  • Memory 550 can also include data memory 570, e.g., build actions, parameters, annotations, audio segments, video, configuration data, settings, user options or preferences, etc., which can be provided to the program memory 560 or any element of the device 500.
  • Some implementations can be operational with numerous other computing system environments or configurations.
  • Examples of computing systems, environments, and/or configurations that may be suitable for use with the technology include, but are not limited to, personal computers, server computers, handheld or laptop devices, cellular telephones, wearable electronics, gaming consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, or the like.
  • FIG. 6A is a wire diagram of a virtual reality head-mounted display (HMD) 600, in accordance with some embodiments.
  • the HMD 600 includes a front rigid body 605 and a band 610.
  • the front rigid body 605 includes one or more electronic display elements of an electronic display 645, an inertial motion unit (IMU) 615, one or more position sensors 620, locators 625, and one or more compute units 630.
  • the position sensors 620, the IMU 615, and compute units 630 may be internal to the HMD 600 and may not be visible to the user.
  • IMU inertial motion unit
  • the IMU 615, position sensors 620, and locators 625 can track movement and location of the HMD 600 in the real world and in an artificial reality environment in three degrees of freedom (3DoF) or six degrees of freedom (6DoF).
  • the locators 625 can emit infrared light beams which create light points on real objects around the HMD 600.
  • the IMU 615 can include e.g., one or more accelerometers, gyroscopes, magnetometers, other non-camera-based position, force, or orientation sensors, or combinations thereof.
  • One or more cameras (not shown) integrated with the HMD 600 can detect the light points.
  • Compute units 630 in the HMD 600 can use the detected light points to extrapolate position and movement of the HMD 600 as well as to identify the shape and position of the real objects surrounding the HMD 600.
  • the electronic display 645 can be integrated with the front rigid body 605 and can provide image light to a user as dictated by the compute units 630.
  • the electronic display 645 can be a single electronic display or multiple electronic displays (e.g., a display for each user eye).
  • Examples of the electronic display 645 include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels, a projector unit (e.g., microLED, LASER, etc.), some other display, or some combination thereof.
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • AMOLED active-matrix organic light-emitting diode display
  • QOLED quantum dot light-emitting diode
  • the HMD 600 can be coupled to a core processing component such as a personal computer (PC) (not shown) and/or one or more external sensors (not shown).
  • the external sensors can monitor the HMD 600 (e.g., via light emitted from the HMD 600) which the PC can use, in combination with output from the IMU 615 and position sensors 620, to determine the location and movement of the HMD 600.
  • Figure 6B is a wire diagram of a mixed reality HMD system 650 which includes a mixed reality HMD 652 and a core processing component 654.
  • the mixed reality HMD 652 and the core processing component 654 can communicate via a wireless connection (e.g., a 60 GHz link) as indicated by link 656.
  • the mixed reality system 650 includes a headset only, without an external compute device or includes other wired or wireless connections between the mixed reality HMD 652 and the core processing component 654.
  • the mixed reality HMD 652 includes a pass-through display 658 and a frame 660.
  • the frame 660 can house various electronic components (not shown) such as light projectors (e.g., LASERS, LEDs, etc.), cameras, eye-tracking sensors, MEMS components, networking components, etc.
  • the projectors can be coupled to the pass-through display 658, e.g., via optical elements, to display media to a user.
  • the optical elements can include one or more waveguide assemblies, reflectors, lenses, mirrors, collimators, gratings, etc., for directing light from the projectors to a user's eye.
  • Image data can be transmitted from the core processing component 654 via link 656 to HMD 652.
  • Controllers in the HMD 652 can convert the image data into light pulses from the projectors, which can be transmitted via the optical elements as output light to the user's eye.
  • the output light can mix with light that passes through the display 658, allowing the output light to present virtual objects that appear as if they exist in the real world.
  • the HMD system 650 can also include motion and position tracking units, cameras, light sources, etc., which allow the HMD system 650 to, e.g., track itself in 3DoF or 6DoF, track portions of the user (e.g., hands, feet, head, or other body parts), map virtual objects to appear as stationary as the HMD 652 moves, and have virtual objects react to gestures and other real-world objects.
  • motion and position tracking units cameras, light sources, etc.
  • FIG. 6C illustrates controllers 670 (including controller 676A and 676B), which, in some implementations, a user can hold in one or both hands to interact with an artificial reality environment presented by the HMD 600 and/or HMD 650.
  • the controllers 670 can be in communication with the HMDs, either directly or via an external device (e.g., core processing component 654).
  • the controllers can have their own IMU units, position sensors, and/or can emit further light points.
  • the HMD 600 or 650, external sensors, or sensors in the controllers can track these controller light points to determine the controller positions and/or orientations (e.g., to track the controllers in 3DoF or 6DoF).
  • the compute units 630 in the HMD 600 or the core processing component 654 can use this tracking, in combination with IMU and position output, to monitor hand positions and motions of the user.
  • the controllers can also include various buttons (e.g., buttons 672A-F) and/or joysticks (e.g., joysticks 676A-B), which a user can actuate to provide input and interact with objects.
  • the HMD 600 or 650 can also include additional subsystems, such as an eye tracking unit, an audio system, various network components, etc., to monitor indications of user interactions and intentions.
  • additional subsystems such as an eye tracking unit, an audio system, various network components, etc.
  • one or more cameras included in the HMD 600 or 650 can monitor the positions and poses of the user's hands to determine gestures and other hand and body motions.
  • one or more light sources can illuminate either or both of the user's eyes and the HMD 600 or 650 can use eye-facing cameras to capture a reflection of this light to determine eye position (e.g., based on set of reflections around the user's cornea), modeling the user's eye and determining a gaze direction.
  • FIG. 7 is a block diagram illustrating an overview of an environment 700 in which some implementations of the disclosed technology can operate.
  • Environment 700 can include one or more client computing devices 705A-D, examples of which can include device 500.
  • Client computing devices 705 can operate in a networked environment using logical connections through network 730 to one or more remote computers, such as a server computing device.
  • server 710 can be an edge server which receives client requests and coordinates fulfillment of those requests through other servers, such as servers 720A-C.
  • Server computing devices 710 and 720 can comprise computing systems, such as device 500. Though each server computing device 710 and 720 is displayed logically as a single server, server computing devices can each be a distributed computing environment encompassing multiple computing devices located at the same or at geographically disparate physical locations. In some implementations, each server 720 corresponds to a group of servers.
  • Client computing devices 705 and server computing devices 710 and 720 can each act as a server or client to other server/client devices.
  • Server 710 can connect to a database 715.
  • Servers 720A-C can each connect to a corresponding database 725A-C.
  • each server 720 can correspond to a group of servers, and each of these servers can share a database or can have their own database.
  • Databases 715 and 725 can warehouse (e.g., store) information. Though databases 715 and 725 are displayed logically as single units, databases 715 and 725 can each be a distributed computing environment encompassing multiple computing devices, can be located within their corresponding server, or can be located at the same or at geographically disparate physical locations.
  • Network 730 can be a local area network (LAN) or a wide area network (WAN), but can also be other wired or wireless networks.
  • Network 730 may be the Internet or some other public or private network.
  • Client computing devices 705 can be connected to network 730 through a network interface, such as by wired or wireless communication. While the connections between server 710 and servers 720 are shown as separate connections, these connections can be any kind of local, wide area, wired, or wireless network, including network 730 or a separate public or private network.
  • Embodiments of the disclosed technology may include or be implemented in conjunction with an artificial reality system.
  • Artificial reality or extra reality (XR) is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof.
  • Artificial reality content may include completely generated content or generated content combined with captured content (e.g., real-world photographs).
  • the artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer).
  • artificial reality may be associated with applications, products, accessories, services, or some combination thereof, that are, e.g., used to create content in an artificial reality and/or used in (e.g., perform activities in) an artificial reality.
  • the artificial reality system that provides the artificial reality content may be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, a "cave" environment or other projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.
  • HMD head-mounted display
  • VR virtual reality
  • AR Augmented reality
  • MR virtual reality
  • MR meand reality
  • a MR headset could be shaped as a pair of glasses with a pass-through display, which allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR headset, allowing the MR headset to present virtual objects intermixed with the real objects the user can see.
  • "Artificial reality,” “extra reality,” or “XR,” as used herein, refers to any of VR, AR, MR, or any combination or hybrid thereof. Additional details on XR systems with which the disclosed technology can be used are provided in U.S. Patent Application No. 17/170,839, titled “INTEGRATING ARTIFICIAL REALITY AND OTHER COMPUTING DEVICES,” filed 2/8/2021 , which is herein incorporated by reference.
  • the components and blocks illustrated above may be altered in a variety of ways. For example, the order of the logic may be rearranged, substeps may be performed in parallel, illustrated logic may be omitted, other logic may be included, etc.
  • the word "or" refers to any possible permutation of a set of items.
  • the phrase "A, B, or C” refers to at least one of A, B, C, or any combination thereof, such as any of: A; B; C; A and B; A and C; B and C; A, B, and C; or multiple of any item such as A and A; B, B, and C; A, A, B, C, and C; etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Business, Economics & Management (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Educational Technology (AREA)
  • Educational Administration (AREA)
  • Computer Hardware Design (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Processing Or Creating Images (AREA)

Abstract

Des aspects de la présente invention concernent des systèmes de capture, d'enregistrement et de lecture d'étapes visant à créer des objets et des structures dans un monde de réalité artificielle (XR). Un enseignant ou un créateur peut initier un contexte de capture dans un mode de création d'une application XR pendant lequel des étapes (et potentiellement d'autres métadonnées) d'un processus de création sont enregistrées et stockées. Un moteur de création de monde XR peut générer une relecture interactive du processus de création enregistré qui permet à un autre utilisateur de visualiser le processus de création dans un environnement XR au rythme préféré de l'utilisateur et/ou à partir de différentes perspectives. Dans certains modes de réalisation, le moteur de création de monde XR peut générer une relecture de l'avatar de l'enseignant ou du créateur en train d'effectuer le processus de création, de telle sorte que l'utilisateur peut visualiser le processus de création à la troisième personne et effectuer les mêmes étapes pour apprendre comment créer des objets et des structures dans un monde XR.
PCT/US2023/032015 2022-09-29 2023-09-05 Moteur de création, de capture et de lecture de monde xr WO2024072608A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US202263377571P 2022-09-29 2022-09-29
US63/377,571 2022-09-29
US18/068,975 2022-12-20
US18/068,975 US20240112418A1 (en) 2022-09-29 2022-12-20 XR World Build Capture and Playback Engine

Publications (1)

Publication Number Publication Date
WO2024072608A1 true WO2024072608A1 (fr) 2024-04-04

Family

ID=88236598

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2023/032015 WO2024072608A1 (fr) 2022-09-29 2023-09-05 Moteur de création, de capture et de lecture de monde xr

Country Status (1)

Country Link
WO (1) WO2024072608A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10551993B1 (en) * 2016-05-15 2020-02-04 Google Llc Virtual reality content development environment
US20220180612A1 (en) * 2019-07-16 2022-06-09 Robert E. McKeever Systems and methods for universal augmented reality architecture and development
US20220291808A1 (en) * 2021-02-08 2022-09-15 Meta Platforms Technologies, Llc Integrating Artificial Reality and Other Computing Devices

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10551993B1 (en) * 2016-05-15 2020-02-04 Google Llc Virtual reality content development environment
US20220180612A1 (en) * 2019-07-16 2022-06-09 Robert E. McKeever Systems and methods for universal augmented reality architecture and development
US20220291808A1 (en) * 2021-02-08 2022-09-15 Meta Platforms Technologies, Llc Integrating Artificial Reality and Other Computing Devices

Similar Documents

Publication Publication Date Title
Linowes et al. Augmented reality for developers: Build practical augmented reality applications with unity, ARCore, ARKit, and Vuforia
US10403050B1 (en) Multi-user virtual and augmented reality tracking systems
US20230092103A1 (en) Content linking for artificial reality environments
US10948997B1 (en) Artificial reality notification triggers
US20160321841A1 (en) Producing and consuming metadata within multi-dimensional data
Taylor Develop Microsoft hololens apps now
Linowes Unity virtual reality projects: Learn virtual reality by developing more than 10 engaging projects with unity 2018
US11409405B1 (en) Augment orchestration in an artificial reality environment
ONG. et al. Beginning windows mixed reality programming
EP4235629A1 (fr) Reproduction d'interaction physique enregistrée
Mack et al. Unreal Engine 4 virtual reality projects: build immersive, real-world VR applications using UE4, C++, and unreal blueprints
EP4325333A1 (fr) Partage de perspective dans un environnement de réalité artificielle entre des interfaces de réalité bidimensionnelle et artificielle
US20230419618A1 (en) Virtual Personal Interface for Control and Travel Between Virtual Worlds
US20240112418A1 (en) XR World Build Capture and Playback Engine
US11556172B1 (en) Viewpoint coordination on artificial reality models
Vroegop Microsoft HoloLens Developer's Guide
WO2024072608A1 (fr) Moteur de création, de capture et de lecture de monde xr
Jana et al. HoloLens blueprints
US20240192973A1 (en) Artificial Reality Simulation for Augment Manipulation Controls on a Two-Dimensional Interface
US12039141B2 (en) Translating interactions on a two-dimensional interface to an artificial reality experience
US11947862B1 (en) Streaming native application content to artificial reality devices
US20240311498A1 (en) Augment Graph for Selective Sharing of Augments Across Applications or Users
Kucherenko Webvr api description and a-frame application implementation
US11836205B2 (en) Artificial reality browser configured to trigger an immersive experience
US20230260239A1 (en) Turning a Two-Dimensional Image into a Skybox

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23782655

Country of ref document: EP

Kind code of ref document: A1