US20210327140A1 - Techniques for participation in a shared setting - Google Patents
Techniques for participation in a shared setting Download PDFInfo
- Publication number
- US20210327140A1 US20210327140A1 US17/359,169 US202117359169A US2021327140A1 US 20210327140 A1 US20210327140 A1 US 20210327140A1 US 202117359169 A US202117359169 A US 202117359169A US 2021327140 A1 US2021327140 A1 US 2021327140A1
- Authority
- US
- United States
- Prior art keywords
- user
- setting
- electronic device
- avatar
- physical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
- G06T13/205—3D [Three Dimensional] animation driven by audio data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
- G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
- G06T2219/024—Multi-user, collaborative environment
Definitions
- the present disclosure relates generally to enhanced reality (ER) settings, and more specifically to methods and techniques for managing members of a shared ER setting.
- ER enhanced reality
- Enhanced reality (ER) settings provide a convenient platform for a plurality of users to, together, experience and interact with a virtual, enhanced reality setting that is different from the physical setting.
- existing techniques for managing members of a shared ER setting can be cumbersome and inefficient.
- a method is needed that enables users to quickly and easily enter into a shared ER setting and to conveniently manage different types of conversations within the shared ER setting.
- a method performed at a first electronic device having one or more sensors and one or more displays and adapted to communicate with a second electronic device comprises: receiving a request to present an enhanced reality setting and determining whether a first user of the first electronic device and a second user of the second electronic device satisfy a physical interaction criterion.
- the method further comprises: in accordance with a determination that the physical interaction criterion is satisfied, presenting the enhanced reality setting, wherein presenting the enhanced reality setting includes displaying an avatar representing the second user in the enhanced reality setting; and in accordance with a determination that the physical interaction criterion is not satisfied, presenting the enhanced reality setting without the avatar representing the second user.
- a first electronic device comprising one or more sensors; one or more displays; a wireless communication radio configured to communicate with a second electronic device; one or more processors; and memory storing one or more programs configured to be executed by the one or more processors.
- the one or more programs include instructions for: receiving a request to present an enhanced reality setting and determining whether a first user of the first electronic device and a second user of the second electronic device satisfy a physical interaction criterion.
- the one or more programs further include instructions for: in accordance with a determination that the physical interaction criterion is satisfied, presenting the enhanced reality setting, wherein presenting the enhanced reality setting includes displaying an avatar representing the second user in the enhanced reality setting; and in accordance with a determination that the physical interaction criterion is not satisfied, presenting the enhanced reality setting without the avatar representing the second user.
- a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first electronic device having one or more sensors and one or more displays and adapted to communicate with a second electronic device.
- the one or more programs include instructions for: receiving a request to present an enhanced reality setting and determining whether a first user of the first electronic device and a second user of the second electronic device satisfy a physical interaction criterion.
- the one or more programs further include instructions for: in accordance with a determination that the physical interaction criterion is satisfied, presenting the enhanced reality setting, wherein presenting the enhanced reality setting includes displaying an avatar representing the second user in the enhanced reality setting; and in accordance with a determination that the physical interaction criterion is not satisfied, presenting the enhanced reality setting without the avatar representing the second user.
- a method comprises: displaying an enhanced reality setting, wherein the enhanced reality setting includes a plurality of avatars.
- the method further comprises, while displaying the enhanced reality setting: selecting a first avatar of the plurality of avatars as a recipient of received audio input; selecting a second avatar of the plurality of avatars as a non-recipient of the received audio input; receiving, via a microphone, audio input; causing playback of audio corresponding to the audio input and a corresponding visual avatar movement for the first avatar; and forgoing causing playback of the audio for the second avatar.
- a system comprising one or more processors of one or more electronic devices and memory storing one or more programs configured to be executed by the one or more processors.
- the one or more programs include instructions for: displaying an enhanced reality setting, wherein the enhanced reality setting includes a plurality of avatars.
- the one or more programs further include instructions for, while displaying the enhanced reality setting: selecting a first avatar of the plurality of avatars as a recipient of received audio input; selecting a second avatar of the plurality of avatars as a non-recipient of the received audio input; receiving, via a microphone, audio input; causing playback of audio corresponding to the audio input and a corresponding visual avatar movement for the first avatar; and forgoing causing playback of the audio for the second avatar.
- a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of one or more electronic devices.
- the one or more programs include instructions for: displaying an enhanced reality setting, wherein the enhanced reality setting includes a plurality of avatars.
- the one or more programs further include instructions for, while displaying the enhanced reality setting: selecting a first avatar of the plurality of avatars as a recipient of received audio input; selecting a second avatar of the plurality of avatars as a non-recipient of the received audio input; receiving, via a microphone, audio input; causing playback of audio corresponding to the audio input and a corresponding visual avatar movement for the first avatar; and forgoing causing playback of the audio for the second avatar.
- FIGS. 1A-1B depict exemplary systems for use in various computer enhanced reality technologies, including virtual reality and mixed reality.
- FIGS. 2A-2G depict an exemplary process for initializing members of a shared enhanced reality setting, in accordance with some embodiments.
- FIG. 3 is a flow diagram for initializing members of a shared enhanced reality setting, in accordance with some embodiments.
- FIGS. 4A-4I depict an exemplary process for forming a private sub-space and example features of the private sub-space, in accordance with some embodiments.
- FIG. 5 is a flow diagram for forming and maintaining a private sub-space within an enhanced reality setting, in accordance with some embodiments.
- a physical setting refers to a world with which various persons can sense and/or interact without use of electronic systems.
- Physical settings such as a physical park, include physical elements, such as, for example, physical wildlife, physical trees, and physical plants. Persons can directly sense and/or otherwise interact with the physical setting, for example, using one or more senses including sight, smell, touch, taste, and hearing.
- An enhanced reality (ER) setting in contrast to a physical setting, refers to an entirely (or partly) computer-produced setting that various persons, using an electronic system, can sense and/or otherwise interact with.
- ER a person's movements are in part monitored, and, responsive thereto, at least one attribute corresponding to at least one virtual object in the ER setting is changed in a manner that is consistent with one or more physical laws.
- the ER system may adjust various audio and graphics presented to the person in a manner consistent with how such sounds and appearances would change in a physical setting.
- Adjustments to attribute(s) of virtual object(s) in an ER setting also may be made, for example, in response to representations of movement (e.g., voice commands).
- a person may sense and/or interact with an ER object using one or more senses, such as sight, smell, taste, touch, and sound.
- a person may sense and/or interact with objects that create a multi-dimensional or spatial acoustic setting.
- Multi-dimensional or spatial acoustic settings provide a person with a perception of discrete acoustic sources in multi-dimensional space.
- Such objects may also enable acoustic transparency, which may selectively incorporate audio from a physical setting, either with or without computer-produced audio.
- a person may sense and/or interact with only acoustic objects.
- VR Virtual reality
- a VR setting refers to an enhanced setting that is configured to only include computer-produced sensory inputs for one or more senses.
- a VR setting includes a plurality of virtual objects that a person may sense and/or interact with.
- a person may sense and/or interact with virtual objects in the VR setting through a simulation of at least some of the person's actions within the computer-produced setting, and/or through a simulation of the person or her presence within the computer-produced setting.
- An MR setting refers to an enhanced setting that is configured to integrate computer-produced sensory inputs (e.g., virtual objects) with sensory inputs from the physical setting, or a representation of sensory inputs from the physical setting.
- an MR setting is between, but does not include, a completely physical setting at one end and a VR setting at the other end.
- MR settings computer-produced sensory inputs may be adjusted based on changes to sensory inputs from the physical setting.
- electronic systems for presenting MR settings may detect location and/or orientation with respect to the physical setting to enable interaction between real objects (i.e., physical elements from the physical setting or representations thereof) and virtual objects .
- a system may detect movements and adjust computer-produced sensory inputs accordingly, so that, for example, a virtual tree appears fixed with respect to a physical structure.
- Augmented reality is an example of MR.
- An AR setting refers to an enhanced setting where one or more virtual objects are superimposed over a physical setting (or representation thereof).
- an electronic system may include an opaque display and one or more imaging sensors for capturing video and/or images of a physical setting. Such video and/or images may be representations of the physical setting, for example. The video and/or images are combined with virtual objects, wherein the combination is then displayed on the opaque display.
- the physical setting may be viewed by a person, indirectly, via the images and/or video of the physical setting. The person may thus observe the virtual objects superimposed over the physical setting.
- a system When a system captures images of a physical setting, and displays an AR setting on an opaque display using the captured images, the displayed images are called a video pass-through.
- a transparent or semi-transparent display may be included in an electronic system for displaying an AR setting, such that an individual may view the physical setting directly through the transparent or semi-transparent displays.
- Virtual objects may be displayed on the semi-transparent or transparent display, such that an individual observes virtual objects superimposed over a physical setting.
- a projection system may be utilized in order to project virtual objects onto a physical setting. For example, virtual objects may be projected on a physical surface, or as a holograph, such that an individual observes the virtual objects superimposed over the physical setting.
- An AR setting also may refer to an enhanced setting in which a representation of a physical setting is modified by computer-produced sensory data.
- a representation of a physical setting may be graphically modified (e.g., enlarged), so that the modified portion is still representative of (although not a fully-reproduced version of) the originally captured image(s).
- one or more sensor images may be modified in order to impose a specific viewpoint different than a viewpoint captured by the image sensor(s).
- portions of a representation of a physical setting may be altered by graphically obscuring or excluding the portions.
- Augmented virtuality is another example of MR.
- An AV setting refers to an enhanced setting in which a virtual or computer-produced setting integrates one or more sensory inputs from a physical setting. Such sensory input(s) may include representations of one or more characteristics of a physical setting.
- a virtual object may, for example, incorporate a color associated with a physical element captured by imaging sensor(s).
- a virtual object may adopt characteristics consistent with, for example, current weather conditions corresponding to a physical setting, such as weather conditions identified via imaging, online weather information, and/or weather-related sensors.
- an AR park may include virtual structures, plants, and trees, although animals within the AR park setting may include features accurately reproduced from images of physical animals.
- a head mounted system may include one or more speakers and an opaque display.
- an external display e.g., a smartphone
- the head mounted system may include microphones for capturing audio of a physical setting, and/or image sensors for capturing images/video of the physical setting,.
- a transparent or semi-transparent display may also be included in the head mounted system.
- the semi-transparent or transparent display may, for example, include a substrate through which light (representative of images) is directed to a person's eyes.
- the display may also incorporate LEDs, OLEDs, liquid crystal on silicon, a laser scanning light source, a digital light projector, or any combination thereof.
- the substrate through which light is transmitted may be an optical reflector, holographic substrate, light waveguide, optical combiner, or any combination thereof.
- the transparent or semi-transparent display may, for example, transition selectively between a transparent/semi-transparent state and an opaque state.
- the electronic system may be a projection-based system.
- retinal projection may be used to project images onto a person's retina.
- a projection-based system also may project virtual objects into a physical setting, for example, such as projecting virtual objects as a holograph or onto a physical surface.
- ER systems include windows configured to display graphics, headphones, earphones, speaker arrangements, lenses configured to display graphics, heads up displays, automotive windshields configured to display graphics, input mechanisms (e.g., controllers with or without haptic functionality), desktop or laptop computers, tablets, or smartphones.
- input mechanisms e.g., controllers with or without haptic functionality
- FIG. 1A and FIG. 1B depict exemplary system 100 for use in various enhanced reality technologies.
- system 100 includes device 100 a.
- Device 100 a includes various components, such as processor(s) 102 , RF circuitry(ies) 104 , memory(ies) 106 , image sensor(s) 108 , orientation sensor(s) 110 , microphone(s) 112 , location sensor(s) 116 , speaker(s) 118 , display(s) 120 , and touch-sensitive surface(s) 122 . These components optionally communicate over communication bus(es) 150 of device 100 a.
- elements of system 100 are implemented in a base station device (e.g., a computing device, such as a remote server, mobile device, or laptop) and other elements of system 100 are implemented in a second device (e.g., a head-mounted device).
- a base station device e.g., a computing device, such as a remote server, mobile device, or laptop
- a second device e.g., a head-mounted device
- device 100 a is implemented in a base station device or a second device.
- system 100 includes two (or more) devices in communication, such as through a wired connection or a wireless connection.
- First device 100 b e.g., a base station device
- processor(s) 102 e.g., a base station device
- RF circuitry(ies) 104 e.g., RF circuitry(ies) 104
- memory(ies) 106 e.g.
- Second device 100 c (e.g., a head-mounted device) includes various components, such as processor(s) 102 , RF circuitry(ies) 104 , memory(ies) 106 , image sensor(s) 108 , orientation sensor(s) 110 , microphone(s) 112 , location sensor(s) 116 , speaker(s) 118 , display(s) 120 , and touch-sensitive surface(s) 122 . These components optionally communicate over communication bus(es) 150 of device 100 c.
- System 100 includes processor(s) 102 and memory(ies) 106 .
- Processor(s) 102 include one or more general processors, one or more graphics processors, and/or one or more digital signal processors.
- memory(ies) 106 are one or more non-transitory computer-readable storage mediums (e.g., flash memory, random access memory) that store computer-readable instructions configured to be executed by processor(s) 102 to perform the techniques described below.
- System 100 includes RF circuitry(ies) 104 .
- RF circuitry(ies) 104 optionally include circuitry for communicating with electronic devices, networks, such as the Internet, intranets, and/or a wireless network, such as cellular networks and wireless local area networks (LANs).
- RF circuitry(ies) 104 optionally includes circuitry for communicating using near-field communication and/or short-range communication, such as Bluetooth®.
- Display(s) 120 may have an opaque display.
- Display(s) 120 may have a transparent or semi-transparent display that may incorporate a substrate through which light representative of images is directed to an individual's eyes.
- Display(s) 120 may incorporate LEDs, OLEDs, a digital light projector, a laser scanning light source, liquid crystal on silicon, or any combination of these technologies.
- the substrate through which the light is transmitted may be a light waveguide, optical combiner, optical reflector, holographic substrate, or any combination of these substrates.
- the transparent or semi-transparent display may transition selectively between an opaque state and a transparent or semi-transparent state.
- system 100 may be designed to receive an external display (e.g., a smartphone).
- system 100 is a projection-based system that uses retinal projection to project images onto an individual's retina or projects virtual objects into a physical setting (e.g., onto a physical surface or as a holograph).
- system 100 includes touch-sensitive surface(s) 122 for receiving user inputs, such as tap inputs and swipe inputs.
- touch-sensitive surface(s) 122 for receiving user inputs, such as tap inputs and swipe inputs.
- display(s) 120 and touch-sensitive surface(s) 122 form touch-sensitive display(s).
- Image sensor(s) 108 optionally include one or more visible light image sensor, such as charged coupled device (CCD) sensors, and/or complementary metal—oxide—semiconductor (CMOS) sensors operable to obtain images of physical elements from the physical setting.
- Image sensor(s) also optionally include one or more infrared (IR) sensor(s), such as a passive IR sensor or an active IR sensor, for detecting infrared light from the physical setting.
- IR infrared
- an active IR sensor includes an IR emitter, such as an IR dot emitter, for emitting infrared light into the physical setting.
- Image sensor(s) 108 also optionally include one or more event camera(s) configured to capture movement of physical elements in the physical setting.
- Image sensor(s) 108 also optionally include one or more depth sensor(s) configured to detect the distance of physical elements from system 100 .
- system 100 uses CCD sensors, event cameras, and depth sensors in combination to detect the physical setting around system 100 .
- image sensor(s) 108 include a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical elements in the physical setting from two distinct perspectives.
- system 100 uses image sensor(s) 108 to receive user inputs, such as hand gestures.
- system 100 uses image sensor(s) 108 to detect the position and orientation of system 100 and/or display(s) 120 in the physical setting.
- system 100 uses image sensor(s) 108 to track the position and orientation of display(s) 120 relative to one or more fixed elements in the physical setting.
- system 100 includes microphones(s) 112 .
- System 100 uses microphone(s) 112 to detect sound from the user and/or the physical setting of the user.
- microphone(s) 112 includes an array of microphones (including a plurality of microphones) that optionally operate in tandem, such as to identify ambient noise or to locate the source of sound in space of the physical setting.
- System 100 includes orientation sensor(s) 110 for detecting orientation and/or movement of system 100 and/or display(s) 120 .
- system 100 uses orientation sensor(s) 110 to track changes in the position and/or orientation of system 100 and/or display(s) 120 , such as with respect to physical elements in the physical setting.
- Orientation sensor(s) 110 optionally include one or more gyroscopes and/or one or more accelerometers.
- FIGS. 2A-2G depict an exemplary technique for initializing members of a shared ER setting using a first electronic of a first user 202 and a second electronic device 208 of a second user 204 .
- first electronic device 206 and second electronic device 208 are similar to electronic device 100 a described above with reference to FIGS. 1A-1B .
- first electronic device 206 and/or second electronic device 208 are mobile electronic devices, such as a smartphone or a tablet computer.
- first electronic device 206 and/or second electronic device 208 are head-mounted devices (HMD).
- HMD head-mounted devices
- FIG. 2A illustrates a physical setting 200 (e.g., an outdoor park), with first user 202 and second user 204 in physical setting 200 , and where first user 202 is using first electronic device 206 and second user 204 is using second electronic device 208 .
- first user 202 and second user 204 are more than a predetermined distance (e.g., more than 5 meters, more than 3 meters, or more than 1 meter) away from one another within physical setting 200 .
- first electronic device 206 displays, on a display 210 of the first electronic device, an ER setting 214 (e.g., a room), as shown in FIG. 2B .
- second electronic device 206 displays, on a display 212 of the second electronic device, an ER setting 216 (e.g., a gym), as also shown in FIG. 2B .
- ER setting 214 and ER setting 216 are different and separate enhanced reality settings that are not shared by first user 202 and second user 204 .
- first user 202 and second user 204 are not sharing an ER setting.
- FIGS. 2C-2G illustrate different example methods for two or more users (e.g., first user 202 and second user 204 ) to enter into an ER setting that is shared by the two or more users.
- first user 202 and second user 204 have moved from their respective positions in FIG. 2A such that the two users are within a threshold distance (e.g., 5 meters, 3 meters, 1 meter) of one another in physical setting 200 .
- a threshold distance e.g., 5 meters, 3 meters, 1 meter
- first electronic device 206 displays, on display 210 of the first electronic device, an ER setting 218 (e.g., a desert) that includes a second avatar 222 corresponding to second user 204 , as shown in FIG. 2G .
- second electronic device 208 displays, on display 212 of the second electronic device, ER setting 218 (e.g., a desert) that includes a first avatar 220 corresponding to first user 202 , as also shown in FIG. 2G .
- ER setting 218 e.g., a desert
- first electronic device 206 and second electronic device 208 determines that they are within the threshold distance of one another in the physical setting, the devices provide their respective users with a shared ER setting in which the users can interact with one another using avatars.
- providing the respective users with the shared ER setting in which the users can interact with one another using avatars comprises second user 204 appearing as an avatar in the existing ER setting (e.g., ER setting 214 ) that is provided by first electronic device 206 to first user 202 . That is, in some embodiments, in response to detecting that the two users are within the predetermined threshold distance of one another, thereby satisfying the proximity criterion, second electronic device 208 displays, on display 212 of the second electronic device, ER setting 214 (e.g., a room) that includes first avatar 220 corresponding to first user 202 .
- ER setting 214 e.g., a room
- second user 204 appears as an avatar in the existing ER setting (e.g., ER setting 214 ) that is provided by first electronic device 206 to first user 202 .
- devices 206 and 208 together provide a new ER setting (e.g., ER setting 218 ) in which both users 202 and 204 participate via avatars.
- second electronic device 208 first displays, on display 212 , a user interface affordance alerting second user 204 to the presence of first user 202 of first electronic device 206 within the predetermined threshold distance in the physical setting without displaying a shared ER setting.
- second electronic device 208 in response to detecting second user 204 's confirmation (e.g., detecting the user's selection of the user interface affordance), second electronic device 208 enables second user 204 to participate in a shared ER setting with user 202 (e.g., share ER setting 214 with first user 202 ).
- first electronic device 206 receives a request from first user 202 to initialize an ER setting while the device is not displaying an ER setting (e.g., because the device is in an off state or an inactive state). In some embodiments, in response to receiving the request from first user 202 to initialize an ER setting while the device is not already displaying an ER setting, first electronic device 206 detects for whether another device (e.g., second electronic device 208 ) is within a predetermined distance from first electronic device 206 such that the proximity criterion is satisfied.
- another device e.g., second electronic device 208
- first electronic device 206 displays an ER setting (e.g., ER setting 218 ) that is shared by the user of the other device (e.g., second user 204 ), and thus includes an avatar corresponding to the user of the other device (e.g., avatar 222 of second user 204 ).
- ER setting e.g., ER setting 218
- first electronic device 206 displays, on display 210 , a user interface affordance for proceeding with the initiation of the ER setting (e.g., ER setting 218 ) that is shared by the user of the other device (e.g., second user 204 ) without first displaying the shared ER setting, and displays the shared ER setting (e.g., ER setting 218 ) in response to detecting first user 202 's selection or activation of the user interface affordance.
- a user interface affordance for proceeding with the initiation of the ER setting (e.g., ER setting 218 ) that is shared by the user of the other device (e.g., second user 204 ) without first displaying the shared ER setting, and displays the shared ER setting (e.g., ER setting 218 ) in response to detecting first user 202 's selection or activation of the user interface affordance.
- first electronic device 206 forgoes displaying the shared ER setting (e.g., ER setting 218 ).
- the user interface affordance for proceeding with the initiation of the shared ER setting includes an indication that another user (e.g., second user 204 ) is nearby and asks whether to initialize a shared ER setting that includes the other user detected nearby.
- FIG. 2D illustrates another exemplary method for two or more users (e.g., first user 202 and second user 204 ) to enter into an ER setting that is shared by the two or more users.
- first electronic device 206 detects (e.g., using an eye tracking sensor, an orientation sensor, and/or a location sensor) that first user 202 is gazing at second user 204 (e.g., looking at second user 204 for at least a predetermined time period, such as 10 seconds, 5 seconds, or 3 seconds).
- first electronic device 206 in response to detecting the gaze by first user 202 directed to second user 204 , if first electronic device 206 was displaying an ER setting that was not shared with second user 204 (e.g., ER setting 214 ), first electronic device 206 transitions to displaying, on display 210 of the device, an ER setting that is shared with second user 204 (e.g., ER setting 218 ), the target of first user 202 's gaze, where the ER setting that is shared with second user 204 includes an avatar (e.g., avatar 222 ) corresponding to second user 204 within the shared ER setting.
- an avatar e.g., avatar 222
- first electronic device 206 in response to detecting the gaze by first user 202 directed to second user 204 , if first electronic device 206 was displaying an ER setting that was not shared with second user 204 (e.g., ER setting 214 ), first electronic device 206 displays, on display 210 , a user interface affordance for transitioning to the ER setting that is shared with second user 204 (e.g., ER setting 218 ), the target of first user 202 ′s gaze, without first displaying the shared ER setting, and transitions to displaying the shared ER setting (e.g., ER setting 218 ) in response to detecting first user 202 ′s selection or activation of the user interface affordance.
- a user interface affordance for transitioning to the ER setting that is shared with second user 204 (e.g., ER setting 218 )
- the target of first user 202 ′s gaze without first displaying the shared ER setting
- transitions to displaying the shared ER setting e.g.
- first electronic device 206 forgoes transitioning to the shared ER setting (e.g., ER setting 218 ) and instead maintains display of the ER setting (e.g., ER setting 214 ) that was being displayed.
- the user interface affordance for proceeding with the initiation of the shared ER setting includes an indication that another user (e.g., second user 204 ) is nearby and asks whether to initialize a shared ER setting that includes the other user detected nearby.
- first electronic device 206 in response to detecting that the gaze by first user 202 directed to second user 204 , if first electronic device 206 was not displaying an ER setting, first electronic device 206 displays, on display 210 of the device, the ER setting shared with second user 204 (e.g., ER setting 218 ).
- first electronic device 206 in response to detecting the gaze by first user 202 directed to second user 204 , causes second electronic device 208 to also display the ER setting that is shared between first user 202 and second user 204 (e.g., ER setting 218 ).
- second electronic device 208 displays, on display 212 of the device, the ER setting that is shared with first user 202 (e.g., ER setting 218 ) that includes an avatar (e.g., avatar 220 ) corresponding to first user 202 within the shared ER setting.
- FIG. 2E illustrates another example method for two or more users (e.g., first user 202 and second user 204 ) to enter into an ER setting that is shared by the two or more users.
- first electronic device 206 detects (e.g., using one or more cameras of the device; using one or more downward-facing cameras of the device if the device is a HIVID) that first user 202 is engaging in a handshake with second user 204 .
- first electronic device 206 in response to detecting that first user 202 is engaging in the handshake with second user 204 , if first electronic device 206 was displaying an ER setting that was not shared with second user 204 (e.g., ER setting 214 ), first electronic device 206 transitions to displaying an ER setting that is shared with second user 204 (e.g., ER setting 218 ), where the ER setting that is shared with second user 204 includes an avatar (e.g., avatar 222 ) corresponding to second user 204 within the shared ER setting.
- an avatar e.g., avatar 222
- first electronic device 206 in response to detecting that first user 202 is engaging in the handshake with second user 204 , if first electronic device 206 was not displaying an ER setting, first electronic device 206 displays, on display 210 of the device, the ER setting that is shared with second user 204 (e.g., ER setting 218 ).
- second electronic device 206 detects (e.g., using one or more cameras of the device; using one or more downward-facing cameras of the device if the device is a HIVID) that second user 204 is engaging in the handshake with first user 202 .
- second electronic device 208 in response to detecting that second user 204 is engaging in the handshake with first user 202 , if second electronic device 208 was displaying an ER setting that was not shared with first user 202 (e.g., ER setting 218 ), second electronic device 208 transitions to displaying an ER setting that is shared with first user 202 (e.g., ER setting 218 ), where the ER setting that is shared with first user 202 includes an avatar (e.g., avatar 220 ) corresponding to first user 202 within the shared ER setting.
- an avatar e.g., avatar 220
- second electronic device 208 in response to detecting that second user 204 is engaging in the handshake with first user 202 , if second electronic device 208 was not displaying an ER setting, second electronic device 208 displays, on display 212 of the device, the ER setting that is shared with first user 202 (e.g., ER setting 218 ).
- FIG. 2F illustrates another example method for two or more users (e.g., first user 202 and second user 204 ) to enter into an ER setting that is shared by the two or more users.
- first electronic device 206 detects (e.g., using one or more mics of the device) that first user 202 is engaging in a conversation with second user 204 (e.g., an engaging conversation that has been ongoing for at least a predetermined amount of time, such as 10 seconds, 30 seconds, or 1 minute).
- first electronic device 206 in response to detecting that first user 202 is engaging in the conversation with second user 204 , if first electronic device 206 was displaying an ER setting that was not shared with second user 204 (e.g., ER setting 214 ), first electronic device 206 transitions to displaying an ER setting that is shared with second user 204 (e.g., ER setting 218 ), where the ER setting that is shared with second user 204 includes an avatar (e.g., avatar 222 ) corresponding to second user 204 within the shared ER setting.
- an avatar e.g., avatar 222
- first electronic device 206 in response to detecting that first user 202 is engaging in the conversation with second user 204 , if first electronic device 206 was not displaying an ER setting, first electronic device 206 displays, on display 210 of the device, the ER setting that is shared with second user 204 (e.g., ER setting 218 ).
- second electronic device 206 detects (e.g., using one or more mics of the device) that second user 204 is engaging in the conversation with first user 202 .
- second electronic device 208 in response to detecting that second user 204 is engaging in the handshake with first user 202 , if second electronic device 208 was displaying an ER setting that was not shared with first user 202 (e.g., ER setting 218 ), second electronic device 208 transitions to displaying an ER setting that is shared with first user 202 (e.g., ER setting 218 ), where the ER setting that is shared with first user 202 includes an avatar (e.g., avatar 220 ) corresponding to first user 202 within the shared ER setting.
- an avatar e.g., avatar 220
- second electronic device 208 in response to detecting that second user 204 is engaging in the conversation with first user 202 , if second electronic device 208 was not displaying an ER setting, second electronic device 208 displays, on display 212 of the device, the ER setting that is shared with first user 202 (e.g., ER setting 218 ).
- FIG. 2G illustrates ER setting 220 (e.g., a desert) that includes both avatar 220 corresponding to first user 202 and avatar 222 corresponding to second user 204 , shown from the perspective of first user 202 in display 210 of first electronic device 206 and from the perspective of second user 204 in display 212 of second electronic device 208 .
- ER setting 218 is an enhanced reality setting that was initialized (e.g., via one or more of the processes described above with reference to FIGS. 2C-2G ) to include both first user 202 and second user 204 .
- first user 202 can see, within ER setting 218 , avatar 222 corresponding to second user 204 and, similarly, second user 204 can see, within ER setting 218 , avatar 220 corresponding to first user 202 .
- ER setting 218 is not limited to being shared by two users.
- ER setting 218 can be shared by a plurality of users, including three or more users.
- three or more users can initialize a shared ER setting, such as ER setting 218 , by satisfying the proximity criterion described above with reference to FIG. 2C .
- methods to initialize a shared ER setting is not limited to the triggering events (e.g., satisfying a proximity criterion, detecting a gaze, detecting a handshake, detecting an engaging conversation) described.
- the initialization of an ER setting (e.g., ER setting 218 ) shared by two or more users (e.g., first user 202 and second user 204 ) can be triggered by first electronic device 206 detecting (e.g., using an eye tracking sensor of the device) a wink made by first user 202 directed towards second user 204 and/or by second electronic device 208 detecting (e.g., using an eye tracking sensor of the device) a wink made by second user 204 directed towards first user 202 .
- an ER setting (e.g., ER setting 218 ) shared by two or more users (e.g., first user 202 and second user 204 ) can be triggered by first electronic device 206 detecting (e.g., using an orientation sensor and/or movement sensor of the device) a nod made by first user 202 directed towards second user 204 and/or by second electronic device 208 detecting (e.g., using an orientation sensor and/or movement sensor of the device) a nod made by second user 204 directed towards first user 202 .
- first electronic device 206 detecting (e.g., using an orientation sensor and/or movement sensor of the device) a nod made by first user 202 directed towards second user 204 and/or by second electronic device 208 detecting (e.g., using an orientation sensor and/or movement sensor of the device) a nod made by second user 204 directed towards first user 202 .
- device 206 may allow its user to choose whether the device is discoverable by other devices such as device 208 . If device 206 is not discoverable, device 208 will not alert its user 208 to the presence of device 206 even if the two devices are in proximity.
- device 206 may allow user 202 to provide a list of trusted contacts with whom shared experiences in ER settings may be allowed. Conversely, device 206 may allow user 202 to specify limits, e.g., a blocked list, of users with whom ER settings are not to be allowed.
- device 206 may expressly confirm with user 202 as to whether to allow or disallow a particular ER session. In these ways, the present techniques preserve user privacy while improving a user's ability to easily invite others to share an ER setting as appropriate.
- FIG. 3 is a flow diagram for initializing members of a shared ER setting, in accordance with some embodiments.
- the flow of FIG. 3 is performed using an electronic device (e.g., electronic device 100 a, electronic device 206 , electronic device 208 ; a mobile electronic device such as a smartphone or a tablet computer; a HIVID) that is in communication with another electronic device (e.g., electronic device 100 a, electronic device 206 , electronic device 208 ; a mobile electronic device such as a smartphone or a tablet computer; a HIVID) and/or an external server.
- an electronic device e.g., electronic device 100 a, electronic device 206 , electronic device 208 ; a mobile electronic device such as a smartphone or a tablet computer; a HIVID
- an external server e.g., electronic device 100 a, electronic device 206 , electronic device 208 ; a mobile electronic device such as a smartphone or a tablet computer; a HIVID
- flow 300 of FIG. 3 is performed at a first electronic device (e.g., 206 ) having one or more sensors and one or more displays adapted to present an ER setting (e.g., 212 , 214 , 218 ; a VR setting; a MR setting, including an AR setting) and communicate with a second electronic device (e.g., 208 ) adapted to present the ER setting.
- a first electronic device e.g., 206
- a VR setting e.g., a VR setting
- a MR setting including an AR setting
- the first electronic device receives, at block 302 , a request to present the ER setting (e.g., 218 ).
- the first electronic device determines, at block 304 , whether a first user (e.g., 202 ) of the first electronic device (e.g., 206 ) and a second user (e.g., 204 ) of the second electronic device (e.g., 208 ) satisfy a physical interaction criterion (e.g., a criterion based on user actions in a physical setting (e.g., 200 ), such as proximity between the first user and the second user or a social action (e.g., a handshake, gaze) between the first user and the second user).
- a physical interaction criterion e.g., a criterion based on user actions in a physical setting (e.g., 200 )
- a social action e.g., a handshake, gaze
- the first electronic device e.g., 206
- the first electronic device e.g., 206
- the ER setting e.g., 214
- the avatar e.g., 222
- the second user e.g., 204
- presenting the ER setting comprises: displaying, at the first electronic device (e.g., 206 ), a confirmation affordance requesting confirmation for entering the ER setting with the second user (e.g., 204 ); and in response to detecting confirmation by the first user (e.g., 202 ), presenting the ER setting that includes the avatar (e.g., 222 ) representing the second user.
- the physical interaction criterion being satisfied includes the first user (e.g., 202 ) and the second user (e.g., 204 ) being within a predetermined distance of one another within a physical setting (e.g., 200 ).
- the physical interaction criterion being satisfied includes a social signal between the first user (e.g., 202 ) and the second user (e.g., 204 ) being detected within the physical setting (e.g., 200 ).
- the social signal is a handshake performed between the first user (e.g., 202 ) and the second user (e.g., 204 ) within the physical setting (e.g., 200 ).
- the social signal is a conversation performed between the first user (e.g., 202 ) and the second user (e.g., 204 ) within the physical setting (e.g., 200 ).
- the conversation is an engaging conversation such that the first user and the second user have been engaging in a continuous conversation for at least a predetermined amount of time.
- the social signal is a gaze (e.g., that is maintained for at least a predetermined amount of time) by the first user (e.g., 202 ) directed to the second user (e.g., 204 ) within the physical setting (e.g., 200 ).
- the social signal is a nod by the first user directed to the second user within the physical setting.
- the social signal is a wink by the first user directed to the second user within the physical setting.
- the first electronic device (e.g., 206 ) includes one or more sensors (e.g., one or more cameras, one or more movement sensors) for detecting a movement corresponding to a signal action (e.g., movement to perform a handshake, gazing at the second user, winking at the second user, nodding at the second user) made by the first user (e.g., 202 ) within the physical setting (e.g., 200 ).
- a signal action e.g., movement to perform a handshake, gazing at the second user, winking at the second user, nodding at the second user
- FIGS. 4A-4I depict an exemplary process for forming a private sub-space within an ER setting that includes a subset of the avatars within the ER setting (e.g., a sub-enhanced reality (sub-ER) setting within the ER setting) and example features of the private sub-space.
- a subset of the avatars within the ER setting e.g., a sub-enhanced reality (sub-ER) setting within the ER setting
- example features of the private sub-space e.g., a sub-enhanced reality (sub-ER) setting within the ER setting
- FIG. 4A illustrates an ER setting 400 (e.g., a virtual room initialized using one or more methods described above with reference to FIGS. 2A-2G ), where ER setting 400 includes a plurality of avatars, including a first avatar 402 corresponding to a first user of a first electronic device (e.g., similar to first electronic device 206 and second electronic device 208 ), a second avatar 404 corresponding to a second user of a second electronic device (e.g., similar to first electronic device 206 and second electronic device 208 ), and a third avatar 406 corresponding to a third user of a third electronic device (e.g., similar to first electronic device 206 and second electronic device 208 ).
- a first avatar 402 corresponding to a first user of a first electronic device
- a second avatar 404 corresponding to a second user of a second electronic device
- a third avatar 406 corresponding to a third user of a third electronic device (e.g., similar to first electronic device 206
- first avatar 402 is a virtual representation of, and is controlled by, the first user
- second avatar 404 is a virtual representation of, and is controlled by, the second user
- third avatar 406 is a virtual representation of, and is controlled by, the third user.
- first avatar 402 , second avatar 404 , and third avatar 406 are included within ER setting 400 such that interactions supported by ER setting 400 are made available to avatars 402 , 402 , and 406 .
- audio corresponding to an audio input e.g., a spoken input, a voice input
- the second user e.g., captured via a mic of the second electronic device of the second user
- second avatar 404 is played at the devices of both the first user of first avatar 402 and the third user of third avatar 406 .
- movements made by second avatar 404 within ER setting 400 is displayed to the first user of first avatar 402 via a display of the first device, and displayed to the third user of third avatar 406 via a display of the third device.
- the gaze triggers the formation of a private sub-space within the ER setting (e.g., a sub-ER setting within the ER setting that a subset of, but not all of, the avatars present within the ER setting is included in).
- a private sub-space 408 that includes first avatar 402 and second avatar 404 but does not include third avatar 406 is formed within ER setting 400 .
- a private sub-space 408 that includes first avatar 402 and second avatar 404 but does not include third avatar 406 is formed within ER setting 400 .
- a visual indication 410 of the formed private sub-space is further provided to indicate to the users of other avatars within ER setting 400 (e.g., the user of third avatar 406 ) that first avatar 402 and second avatar 404 are engaging in a private conversation in a sub-space within the ER setting (e.g., an ER-equivalent of texting under the table).
- FIG. 4D illustrates another example method for first avatar 402 and second avatar 404 to enter into a private sub-space, such as private sub-space 408 , within ER setting 400 .
- each avatar of the plurality of avatars in ER setting 400 e.g., first avatar 402 , second avatar 404 , and third avatar 406
- have a corresponding selectable affordance e.g., displayed above or adjacent to the respective avatar
- first avatar 402 has a selectable affordance 402 A
- second avatar 404 has a selectable affordance 404 A
- third avatar 406 has a selectable affordance 406 A.
- FIG. 4D illustrates another example method for first avatar 402 and second avatar 404 to enter into a private sub-space, such as private sub-space 408 , within ER setting 400 .
- each avatar of the plurality of avatars in ER setting 400 e.g., first avatar 402 , second avatar 404 , and third avatar 406
- second avatar 404 selects/activates selectable affordance 402 A of first avatar 402 .
- second avatar 404 ′s selection/activation of selectable affordance 402 A of first avatar 402 triggers the formation of private sub-space 408 that includes first avatar 402 and second avatar 404 but does not include third avatar 406 , as previously shown in FIG. 4C .
- Methods for the creation of a private sub-space is not limited to the two examples illustrated in FIG. 4B and 4D .
- Another example method for first avatar 402 and second avatar 404 to form a private sub-space includes first avatar 402 and second avatar 404 engaging in a particular type of social interaction (e.g., a nod, a wink, physical contact).
- a particular type of social interaction e.g., a nod, a wink, physical contact.
- the formation of a private sub-space between first avatar 402 and second avatar 404 can be triggered by a nod made by second avatar 404 directed to first avatar 402 .
- first avatar 402 and second avatar 404 the formation of a private sub-space between first avatar 402 and second avatar 404 and be triggered by a wink made by second avatar 404 directed to first avatar 402 .
- formation of a private sub-space between first avatar 402 and second avatar 404 can be triggered by a physical contact (e.g., second avatar 404 putting a hand on the shoulder of first avatar 402 ) between the two avatars.
- first avatar 402 and second avatar 404 to form a private sub-space includes first avatar 402 and second avatar 404 satisfying a proximity criterion.
- the formation of a private sub-space can be triggered by first avatar 402 and second avatar 404 moving to be within a predetermined distance of one another within ER setting 400 (e.g., and also maintaining the proximity for at least a predetermined time period).
- a private sub-space within an ER setting is not limited to being shared by two avatars.
- a private sub-space can be shared by a plurality of avatars, including three or more avatars.
- three avatars e.g., first avatar 402 , second avatar 404 , and third avatar 406
- FIG. 4E illustrates ER setting 400 that includes private sub-space 408 including first avatar 402 and second avatar 404 but not including third avatar 406 , as previously depicted in FIG. 4C .
- audio 412 corresponding to an audio input e.g., a spoken input, a voice input
- first user e.g., captured via a mic of the first device of the first user
- first avatar 402 is played at the second device of the second user of second avatar 404 , as first avatar 402 and second avatar 404 are included in private sub-space 408 .
- audio 412 is not played (e.g., it is prevented/blocked from being played) at the third device of the third user of third avatar 406 (and at the device of any other user of any other avatar within ER setting 400 ), as third avatar 406 is not included in private sub-space 408 .
- first avatar 402 corresponding to audio 412 can still be seen by third avatar 406 (and any other avatar in ER setting 400 ) from outside private sub-space 408 .
- a user of an avatar that is outside of the private sub-space e.g., the third user of third avatar 406
- a user of an avatar that is outside of the private sub-space cannot see any movements of avatars within the private sub-space (e.g., first avatar 402 and second avatar 404 ) via the device of the user.
- FIG. 4G illustrates ER setting 400 with first avatar 402 and second avatar 404 in a private sub-space 414 and third avatar 406 not included in private sub-space 414 .
- private sub-space 414 does not include visual indication 410 of private sub-space 408 that indicates to outside avatars (e.g., third avatar 406 ) that first avatar 402 and second avatar 404 is in a private sub-space.
- private sub-space 414 includes a visual indication 416 A above or adjacent to first avatar 402 (e.g., a glowing/shining light source) and a similar visual indication 416 B above or adjacent to second avatar 404 , thereby indicating to outside avatars (e.g., third avatar 406 ) that first avatar 402 and second avatar 404 are in a private sub-space.
- first avatar 402 e.g., a glowing/shining light source
- second avatar 404 e.g., a similar visual indication 416 B above or adjacent to second avatar 404 , thereby indicating to outside avatars (e.g., third avatar 406 ) that first avatar 402 and second avatar 404 are in a private sub-space.
- FIG. 4H illustrates an ER setting 420 that includes a first avatar 422 (corresponding to a first user of a first electronic device), a second avatar 424 (corresponding to a second user of a second electronic device), a third avatar 426 (corresponding to a third user of a third electronic device), and a fourth avatar 428 (corresponding to a fourth user of a fourth electronic device).
- first avatar 422 and second avatar 424 is in a first private sub-space 430 within ER setting 420 and third avatar 426 and fourth avatar 428 is in a second private sub-space 432 .
- audio 434 corresponding to an audio input (e.g., a spoken input, a voice input) made by the second user (e.g., captured via a mic of the second device of the second user) via second avatar 424 is played at the first device of the first user of first avatar 422 , as first avatar 422 and second avatar 424 are included in the same private sub-space 430 .
- an audio input e.g., a spoken input, a voice input
- audio 434 is not played (e.g., it is prevented/blocked from being played) at the third device of the third user of third avatar 426 and at the fourth device of the fourth user of fourth avatar 428 , as both third avatar 426 and fourth avatar 428 are not included in first private sub-space 430 (and are instead included in a different private sub-space 432 ).
- audio 436 corresponding to an audio input (e.g., a spoken input, a voice input) made by the third user (e.g., captured via a mic of the third device of the third user) via third avatar 426 is played at the fourth device of the fourth user of fourth avatar 428 , as third avatar 426 and fourth avatar 428 are included in the same private sub-space 432 .
- an audio input e.g., a spoken input, a voice input
- audio 436 is not played (e.g., it is prevented/blocked from being played) at the first device of the first user of first avatar 422 and at the second device of the second user of second avatar 424 , as both first avatar 422 and second avatar 424 are not included in second private sub-space 432 (and are instead included in a different private sub-space 430 ).
- FIG. 5 is a flow diagram for forming and maintaining a private sub-space within an ER setting, in accordance with some embodiments.
- flow 500 of FIG. 5 is performed by a system that includes one or more electronic devices (e.g., electronic device 100 a , electronic device 206 , electronic device 208 ; a mobile electronic device such as a smartphone or a tablet computer; a HMD) and/or an external server, where the one or more electronic devices are in communication with each other and/or with the external server.
- one or more electronic devices e.g., electronic device 100 a , electronic device 206 , electronic device 208 ; a mobile electronic device such as a smartphone or a tablet computer; a HMD
- the system e.g., 100 a, 206 , 208
- displays e.g., at an electronic device, such as electronic device 100 a, 206 , 208 , of the system
- an ER setting e.g., 400 , a VR setting, a MR setting, including an AR setting
- the ER setting includes a plurality of avatars (e.g., of other users that share the same ER setting with the current user).
- the system selects, at block 506 , a first avatar (e.g., 404 ) of the plurality of avatars (e.g., 402 , 404 , 406 ) as a recipient of received audio input (e.g., 412 , an audio input made by a user corresponding to a third avatar of the plurality of avatars).
- a first avatar e.g., 404
- the plurality of avatars e.g., 402 , 404 , 406
- a recipient of received audio input e.g., 412 , an audio input made by a user corresponding to a third avatar of the plurality of avatars.
- the system while displaying the ER setting (e.g. 400 ), the system also selects, at block 508 , a second avatar (e.g., 406 ) of the plurality of avatars (e.g., 402 , 404 , 406 ) as a non-recipient of the received audio input.
- a second avatar e.g., 406
- the plurality of avatars e.g., 402 , 404 , 406
- the system While displaying the ER setting (e.g., 400 ), the system also receives, at block 510 , via a microphone (e.g., of the electronic device of the current user), audio input (e.g., 412 ).
- a microphone e.g., of the electronic device of the current user
- audio input e.g., 412
- the system while displaying the ER setting (e.g., 400 ), the system also causes, at block 512 , playback and a corresponding visual avatar movement for the first avatar (e.g., 404 ).
- the system while displaying ER setting (e.g., 400 ), the system also forgoes causing, at block 514 , playback of the audio (e.g., 412 ) for the second avatar (e.g., 406 ).
- the audio e.g., 412
- the second avatar e.g., 406
- causing playback and the corresponding visual avatar movement for the first avatar comprises causing playback and the corresponding visual avatar movement for the first avatar in accordance with a determination that the first avatar is within (e.g., sharing with a current avatar) a sub-ER setting (e.g., 408 ) that includes the first avatar and does not include the second avatar (e.g., 406 ).
- a sub-ER setting e.g., 408
- the determination that the first avatar (e.g., 404 ) is within the sub-ER setting (e.g., 408 ) that includes the first avatar and does not include the second avatar is based on whether the first avatar satisfied a sub-ER setting criterion (e.g., with a third avatar of the plurality of avatars).
- satisfying the sub-ER setting criterion with a third avatar of the plurality of avatars enables the first avatar to enter and/or be associated with the sub-ER setting with the third avatar.
- the sub-ER setting criterion being satisfied includes the first avatar being within a predetermined distance from a third avatar (e.g., 402 ) of the plurality of avatars within the ER setting (e.g., 400 ).
- the sub-ER setting criterion being satisfied includes a gaze by the first avatar (e.g., 404 ) directed to a third avatar (e.g., 402 ) of the plurality of avatars being detected within the ER setting (e.g., 400 ).
- the sub-ER setting criterion being satisfied includes a nod by the first avatar (e.g., 404 ) directed to a third avatar (e.g., 402 ) of the plurality of avatars being detected within the ER setting (e.g., 400 ).
- the sub-ER setting criterion being satisfied includes a wink by the first avatar (e.g., 404 ) directed to a third avatar (e.g., 402 ) of the plurality of avatars being detected within the ER setting (e.g., 400 ).
- the determination that the first avatar (e.g., 404 ) is within the sub-ER setting (e.g., 408 ) that includes the first avatar (e.g., 404 ) and does not include the second avatar (e.g., 406 ) is based on whether an affordance (e.g., 404 A) associated with the first avatar has been selected (e.g., by a third avatar of the plurality of avatars).
- the determination that the first avatar is within the sub-ER setting that includes the first avatar and does not include the second avatar is based on whether an affordance associated with a third avatar of the plurality of avatars has been selected by the first avatar.
- the system while displaying the ER setting (e.g., 400 ), the system presents (e.g., to a fourth avatar of the plurality of avatars) a visual indication (e.g., 416 B, a light shown above or proximate to the first avatar) for the first avatar indicating that the first avatar is a recipient of the received user input.
- a visual indication e.g., 416 B, a light shown above or proximate to the first avatar
- the system while displaying the ER setting, the system forgoes presenting (e.g., to the fourth avatar of the plurality of avatars) the visual indication for the second avatar (e.g., because the second avatar is a non-recipient of the received user input).
- the visual indication (e.g., 416 B) for the first avatar is a light source shown proximate to (e.g., above the head of) the first avatar (e.g., 404 ) within the ER setting (e.g., 400 ).
- Various processes defined herein consider the option of obtaining and utilizing a user's personal information.
- personal information may be utilized in order to provide an improved experience for members of a shared enhanced reality setting on one or more electronic devices.
- personal information may be utilized in order to provide an improved experience for members of a shared enhanced reality setting on one or more electronic devices.
- personal information should be obtained with the user's informed consent.
- the user should have knowledge of and control over the use of their personal information.
- Users may, however, limit the degree to which such parties may access or otherwise obtain personal information. For instance, settings or other preferences may be adjusted such that users can decide whether their personal information can be accessed by various entities. Furthermore, while some features defined herein are described in the context of using personal information, various aspects of these features can be implemented without the need to use such information. As an example, if user preferences, account names, and/or location history are gathered, this information can be obscured or otherwise generalized such that the information does not identify the respective user.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Business, Economics & Management (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Computer Graphics (AREA)
- Strategic Management (AREA)
- Finance (AREA)
- Accounting & Taxation (AREA)
- Development Economics (AREA)
- Multimedia (AREA)
- Entrepreneurship & Innovation (AREA)
- General Health & Medical Sciences (AREA)
- Game Theory and Decision Science (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Economics (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- Health & Medical Sciences (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
In accordance with some embodiments, an exemplary process for initializing members of a shared enhanced reality setting is described. In accordance with some embodiments, an exemplary process for forming a private sub-space and example features of the private sub-space is described.
Description
- This application is a continuation of PCT Application No. PCT/US2020/27186, entitled “TECHNIQUES FOR PARTICIPATION IN A SHARED SETTING,” filed Apr. 8, 2020, which claims priority to U.S. Provisional Patent Application Ser. No. 62/832,161, entitled “TECHNIQUES FOR PARTICIPATION IN A SHARED COMPUTER-SIMULATED REALITY ENVIRONMENT,” filed Apr. 10, 2019, the contents of which are hereby incorporated by reference in their entirety.
- The present disclosure relates generally to enhanced reality (ER) settings, and more specifically to methods and techniques for managing members of a shared ER setting.
- Enhanced reality (ER) settings provide a convenient platform for a plurality of users to, together, experience and interact with a virtual, enhanced reality setting that is different from the physical setting. However, existing techniques for managing members of a shared ER setting can be cumbersome and inefficient. Thus, a method is needed that enables users to quickly and easily enter into a shared ER setting and to conveniently manage different types of conversations within the shared ER setting.
- In accordance with some embodiments, a method performed at a first electronic device having one or more sensors and one or more displays and adapted to communicate with a second electronic device is described. The method comprises: receiving a request to present an enhanced reality setting and determining whether a first user of the first electronic device and a second user of the second electronic device satisfy a physical interaction criterion. The method further comprises: in accordance with a determination that the physical interaction criterion is satisfied, presenting the enhanced reality setting, wherein presenting the enhanced reality setting includes displaying an avatar representing the second user in the enhanced reality setting; and in accordance with a determination that the physical interaction criterion is not satisfied, presenting the enhanced reality setting without the avatar representing the second user.
- In accordance with some embodiments, a first electronic device comprising one or more sensors; one or more displays; a wireless communication radio configured to communicate with a second electronic device; one or more processors; and memory storing one or more programs configured to be executed by the one or more processors is described. The one or more programs include instructions for: receiving a request to present an enhanced reality setting and determining whether a first user of the first electronic device and a second user of the second electronic device satisfy a physical interaction criterion. The one or more programs further include instructions for: in accordance with a determination that the physical interaction criterion is satisfied, presenting the enhanced reality setting, wherein presenting the enhanced reality setting includes displaying an avatar representing the second user in the enhanced reality setting; and in accordance with a determination that the physical interaction criterion is not satisfied, presenting the enhanced reality setting without the avatar representing the second user.
- In accordance with some embodiments, a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first electronic device having one or more sensors and one or more displays and adapted to communicate with a second electronic device is described. The one or more programs include instructions for: receiving a request to present an enhanced reality setting and determining whether a first user of the first electronic device and a second user of the second electronic device satisfy a physical interaction criterion. The one or more programs further include instructions for: in accordance with a determination that the physical interaction criterion is satisfied, presenting the enhanced reality setting, wherein presenting the enhanced reality setting includes displaying an avatar representing the second user in the enhanced reality setting; and in accordance with a determination that the physical interaction criterion is not satisfied, presenting the enhanced reality setting without the avatar representing the second user.
- In accordance with some embodiments, a method is described. The method comprises: displaying an enhanced reality setting, wherein the enhanced reality setting includes a plurality of avatars. The method further comprises, while displaying the enhanced reality setting: selecting a first avatar of the plurality of avatars as a recipient of received audio input; selecting a second avatar of the plurality of avatars as a non-recipient of the received audio input; receiving, via a microphone, audio input; causing playback of audio corresponding to the audio input and a corresponding visual avatar movement for the first avatar; and forgoing causing playback of the audio for the second avatar.
- In accordance with some embodiments, a system comprising one or more processors of one or more electronic devices and memory storing one or more programs configured to be executed by the one or more processors is described. The one or more programs include instructions for: displaying an enhanced reality setting, wherein the enhanced reality setting includes a plurality of avatars. The one or more programs further include instructions for, while displaying the enhanced reality setting: selecting a first avatar of the plurality of avatars as a recipient of received audio input; selecting a second avatar of the plurality of avatars as a non-recipient of the received audio input; receiving, via a microphone, audio input; causing playback of audio corresponding to the audio input and a corresponding visual avatar movement for the first avatar; and forgoing causing playback of the audio for the second avatar.
- In accordance with some embodiments, a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of one or more electronic devices is described. The one or more programs include instructions for: displaying an enhanced reality setting, wherein the enhanced reality setting includes a plurality of avatars. The one or more programs further include instructions for, while displaying the enhanced reality setting: selecting a first avatar of the plurality of avatars as a recipient of received audio input; selecting a second avatar of the plurality of avatars as a non-recipient of the received audio input; receiving, via a microphone, audio input; causing playback of audio corresponding to the audio input and a corresponding visual avatar movement for the first avatar; and forgoing causing playback of the audio for the second avatar.
- For a better understanding of the various described embodiments, reference should be made to the Description below, in conjunction with the following drawings in which like reference numerals refer to corresponding parts throughout the figures.
-
FIGS. 1A-1B depict exemplary systems for use in various computer enhanced reality technologies, including virtual reality and mixed reality. -
FIGS. 2A-2G depict an exemplary process for initializing members of a shared enhanced reality setting, in accordance with some embodiments. -
FIG. 3 is a flow diagram for initializing members of a shared enhanced reality setting, in accordance with some embodiments. -
FIGS. 4A-4I depict an exemplary process for forming a private sub-space and example features of the private sub-space, in accordance with some embodiments. -
FIG. 5 is a flow diagram for forming and maintaining a private sub-space within an enhanced reality setting, in accordance with some embodiments. - Various examples of electronic systems and techniques for using such systems in relation to various enhanced reality technologies are described.
- A physical setting refers to a world with which various persons can sense and/or interact without use of electronic systems. Physical settings, such as a physical park, include physical elements, such as, for example, physical wildlife, physical trees, and physical plants. Persons can directly sense and/or otherwise interact with the physical setting, for example, using one or more senses including sight, smell, touch, taste, and hearing.
- An enhanced reality (ER) setting, in contrast to a physical setting, refers to an entirely (or partly) computer-produced setting that various persons, using an electronic system, can sense and/or otherwise interact with. In ER, a person's movements are in part monitored, and, responsive thereto, at least one attribute corresponding to at least one virtual object in the ER setting is changed in a manner that is consistent with one or more physical laws. For example, in response to an ER system detecting a person looking upward, the ER system may adjust various audio and graphics presented to the person in a manner consistent with how such sounds and appearances would change in a physical setting. Adjustments to attribute(s) of virtual object(s) in an ER setting also may be made, for example, in response to representations of movement (e.g., voice commands).
- A person may sense and/or interact with an ER object using one or more senses, such as sight, smell, taste, touch, and sound. For example, a person may sense and/or interact with objects that create a multi-dimensional or spatial acoustic setting. Multi-dimensional or spatial acoustic settings provide a person with a perception of discrete acoustic sources in multi-dimensional space. Such objects may also enable acoustic transparency, which may selectively incorporate audio from a physical setting, either with or without computer-produced audio. In some ER settings, a person may sense and/or interact with only acoustic objects.
- Virtual reality (VR) is one example of ER. A VR setting refers to an enhanced setting that is configured to only include computer-produced sensory inputs for one or more senses. A VR setting includes a plurality of virtual objects that a person may sense and/or interact with. A person may sense and/or interact with virtual objects in the VR setting through a simulation of at least some of the person's actions within the computer-produced setting, and/or through a simulation of the person or her presence within the computer-produced setting.
- Mixed reality (MR) is another example of ER. An MR setting refers to an enhanced setting that is configured to integrate computer-produced sensory inputs (e.g., virtual objects) with sensory inputs from the physical setting, or a representation of sensory inputs from the physical setting. On a reality spectrum, an MR setting is between, but does not include, a completely physical setting at one end and a VR setting at the other end.
- In some MR settings, computer-produced sensory inputs may be adjusted based on changes to sensory inputs from the physical setting. Moreover, some electronic systems for presenting MR settings may detect location and/or orientation with respect to the physical setting to enable interaction between real objects (i.e., physical elements from the physical setting or representations thereof) and virtual objects . For example, a system may detect movements and adjust computer-produced sensory inputs accordingly, so that, for example, a virtual tree appears fixed with respect to a physical structure.
- Augmented reality (AR) is an example of MR. An AR setting refers to an enhanced setting where one or more virtual objects are superimposed over a physical setting (or representation thereof). As an example, an electronic system may include an opaque display and one or more imaging sensors for capturing video and/or images of a physical setting. Such video and/or images may be representations of the physical setting, for example. The video and/or images are combined with virtual objects, wherein the combination is then displayed on the opaque display. The physical setting may be viewed by a person, indirectly, via the images and/or video of the physical setting. The person may thus observe the virtual objects superimposed over the physical setting. When a system captures images of a physical setting, and displays an AR setting on an opaque display using the captured images, the displayed images are called a video pass-through. Alternatively, a transparent or semi-transparent display may be included in an electronic system for displaying an AR setting, such that an individual may view the physical setting directly through the transparent or semi-transparent displays. Virtual objects may be displayed on the semi-transparent or transparent display, such that an individual observes virtual objects superimposed over a physical setting. In yet another example, a projection system may be utilized in order to project virtual objects onto a physical setting. For example, virtual objects may be projected on a physical surface, or as a holograph, such that an individual observes the virtual objects superimposed over the physical setting.
- An AR setting also may refer to an enhanced setting in which a representation of a physical setting is modified by computer-produced sensory data. As an example, at least a portion of a representation of a physical setting may be graphically modified (e.g., enlarged), so that the modified portion is still representative of (although not a fully-reproduced version of) the originally captured image(s). Alternatively, in providing video pass-through, one or more sensor images may be modified in order to impose a specific viewpoint different than a viewpoint captured by the image sensor(s). As another example, portions of a representation of a physical setting may be altered by graphically obscuring or excluding the portions.
- Augmented virtuality (AV) is another example of MR. An AV setting refers to an enhanced setting in which a virtual or computer-produced setting integrates one or more sensory inputs from a physical setting. Such sensory input(s) may include representations of one or more characteristics of a physical setting. A virtual object may, for example, incorporate a color associated with a physical element captured by imaging sensor(s). Alternatively, a virtual object may adopt characteristics consistent with, for example, current weather conditions corresponding to a physical setting, such as weather conditions identified via imaging, online weather information, and/or weather-related sensors. As another example, an AR park may include virtual structures, plants, and trees, although animals within the AR park setting may include features accurately reproduced from images of physical animals.
- Various systems allow persons to sense and/or interact with ER settings. For example, a head mounted system may include one or more speakers and an opaque display. As another example, an external display (e.g., a smartphone) may be incorporated within a head mounted system. The head mounted system may include microphones for capturing audio of a physical setting, and/or image sensors for capturing images/video of the physical setting,. A transparent or semi-transparent display may also be included in the head mounted system. The semi-transparent or transparent display may, for example, include a substrate through which light (representative of images) is directed to a person's eyes. The display may also incorporate LEDs, OLEDs, liquid crystal on silicon, a laser scanning light source, a digital light projector, or any combination thereof. The substrate through which light is transmitted may be an optical reflector, holographic substrate, light waveguide, optical combiner, or any combination thereof. The transparent or semi-transparent display may, for example, transition selectively between a transparent/semi-transparent state and an opaque state. As another example, the electronic system may be a projection-based system. In a projection-based system, retinal projection may be used to project images onto a person's retina. Alternatively, a projection-based system also may project virtual objects into a physical setting, for example, such as projecting virtual objects as a holograph or onto a physical surface. Other examples of ER systems include windows configured to display graphics, headphones, earphones, speaker arrangements, lenses configured to display graphics, heads up displays, automotive windshields configured to display graphics, input mechanisms (e.g., controllers with or without haptic functionality), desktop or laptop computers, tablets, or smartphones.
-
FIG. 1A andFIG. 1B depictexemplary system 100 for use in various enhanced reality technologies. - In some examples, as illustrated in
FIG. 1A ,system 100 includesdevice 100a.Device 100a includes various components, such as processor(s) 102, RF circuitry(ies) 104, memory(ies) 106, image sensor(s) 108, orientation sensor(s) 110, microphone(s) 112, location sensor(s) 116, speaker(s) 118, display(s) 120, and touch-sensitive surface(s) 122. These components optionally communicate over communication bus(es) 150 ofdevice 100 a. - In some examples, elements of
system 100 are implemented in a base station device (e.g., a computing device, such as a remote server, mobile device, or laptop) and other elements ofsystem 100 are implemented in a second device (e.g., a head-mounted device). In some examples,device 100a is implemented in a base station device or a second device. - As illustrated in
FIG. 1B , in some examples,system 100 includes two (or more) devices in communication, such as through a wired connection or a wireless connection.First device 100 b (e.g., a base station device) includes processor(s) 102, RF circuitry(ies) 104, and memory(ies) 106. These components optionally communicate over communication bus(es) 150 ofdevice 100 b.Second device 100 c (e.g., a head-mounted device) includes various components, such as processor(s) 102, RF circuitry(ies) 104, memory(ies) 106, image sensor(s) 108, orientation sensor(s) 110, microphone(s) 112, location sensor(s) 116, speaker(s) 118, display(s) 120, and touch-sensitive surface(s) 122. These components optionally communicate over communication bus(es) 150 ofdevice 100 c. -
System 100 includes processor(s) 102 and memory(ies) 106. Processor(s) 102 include one or more general processors, one or more graphics processors, and/or one or more digital signal processors. In some examples, memory(ies) 106 are one or more non-transitory computer-readable storage mediums (e.g., flash memory, random access memory) that store computer-readable instructions configured to be executed by processor(s) 102 to perform the techniques described below. -
System 100 includes RF circuitry(ies) 104. RF circuitry(ies) 104 optionally include circuitry for communicating with electronic devices, networks, such as the Internet, intranets, and/or a wireless network, such as cellular networks and wireless local area networks (LANs). RF circuitry(ies) 104 optionally includes circuitry for communicating using near-field communication and/or short-range communication, such as Bluetooth®. -
System 100 includes display(s) 120. Display(s) 120 may have an opaque display. Display(s) 120 may have a transparent or semi-transparent display that may incorporate a substrate through which light representative of images is directed to an individual's eyes. Display(s) 120 may incorporate LEDs, OLEDs, a digital light projector, a laser scanning light source, liquid crystal on silicon, or any combination of these technologies. The substrate through which the light is transmitted may be a light waveguide, optical combiner, optical reflector, holographic substrate, or any combination of these substrates. In one example, the transparent or semi-transparent display may transition selectively between an opaque state and a transparent or semi-transparent state. Other examples of display(s) 120 include heads up displays, automotive windshields with the ability to display graphics, windows with the ability to display graphics, lenses with the ability to display graphics, tablets, smartphones, and desktop or laptop computers. Alternatively,system 100 may be designed to receive an external display (e.g., a smartphone). In some examples,system 100 is a projection-based system that uses retinal projection to project images onto an individual's retina or projects virtual objects into a physical setting (e.g., onto a physical surface or as a holograph). - In some examples,
system 100 includes touch-sensitive surface(s) 122 for receiving user inputs, such as tap inputs and swipe inputs. In some examples, display(s) 120 and touch-sensitive surface(s) 122 form touch-sensitive display(s). -
System 100 includes image sensor(s) 108. Image sensors(s) 108 optionally include one or more visible light image sensor, such as charged coupled device (CCD) sensors, and/or complementary metal—oxide—semiconductor (CMOS) sensors operable to obtain images of physical elements from the physical setting. Image sensor(s) also optionally include one or more infrared (IR) sensor(s), such as a passive IR sensor or an active IR sensor, for detecting infrared light from the physical setting. For example, an active IR sensor includes an IR emitter, such as an IR dot emitter, for emitting infrared light into the physical setting. Image sensor(s) 108 also optionally include one or more event camera(s) configured to capture movement of physical elements in the physical setting. Image sensor(s) 108 also optionally include one or more depth sensor(s) configured to detect the distance of physical elements fromsystem 100. In some examples,system 100 uses CCD sensors, event cameras, and depth sensors in combination to detect the physical setting aroundsystem 100. In some examples, image sensor(s) 108 include a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical elements in the physical setting from two distinct perspectives. In some examples,system 100 uses image sensor(s) 108 to receive user inputs, such as hand gestures. In some examples,system 100 uses image sensor(s) 108 to detect the position and orientation ofsystem 100 and/or display(s) 120 in the physical setting. For example,system 100 uses image sensor(s) 108 to track the position and orientation of display(s) 120 relative to one or more fixed elements in the physical setting. - In some examples,
system 100 includes microphones(s) 112.System 100 uses microphone(s) 112 to detect sound from the user and/or the physical setting of the user. In some examples, microphone(s) 112 includes an array of microphones (including a plurality of microphones) that optionally operate in tandem, such as to identify ambient noise or to locate the source of sound in space of the physical setting. -
System 100 includes orientation sensor(s) 110 for detecting orientation and/or movement ofsystem 100 and/or display(s) 120. For example,system 100 uses orientation sensor(s) 110 to track changes in the position and/or orientation ofsystem 100 and/or display(s) 120, such as with respect to physical elements in the physical setting. Orientation sensor(s) 110 optionally include one or more gyroscopes and/or one or more accelerometers. -
FIGS. 2A-2G depict an exemplary technique for initializing members of a shared ER setting using a first electronic of afirst user 202 and a secondelectronic device 208 of asecond user 204. In some embodiments, firstelectronic device 206 and secondelectronic device 208 are similar toelectronic device 100a described above with reference toFIGS. 1A-1B . In some embodiments, firstelectronic device 206 and/or secondelectronic device 208 are mobile electronic devices, such as a smartphone or a tablet computer. In some embodiments, firstelectronic device 206 and/or secondelectronic device 208 are head-mounted devices (HMD). -
FIG. 2A illustrates a physical setting 200 (e.g., an outdoor park), withfirst user 202 andsecond user 204 inphysical setting 200, and wherefirst user 202 is using firstelectronic device 206 andsecond user 204 is using secondelectronic device 208. InFIG. 2A ,first user 202 andsecond user 204 are more than a predetermined distance (e.g., more than 5 meters, more than 3 meters, or more than 1 meter) away from one another withinphysical setting 200. - In some embodiments (e.g., while
first user 202 andsecond user 204 are more than the predetermined distance away from one another), firstelectronic device 206 displays, on adisplay 210 of the first electronic device, an ER setting 214 (e.g., a room), as shown inFIG. 2B . Similarly (e.g., whilefirst user 202 andsecond user 204 are more than the predetermined distance away from one another), secondelectronic device 206 displays, on adisplay 212 of the second electronic device, an ER setting 216 (e.g., a gym), as also shown inFIG. 2B . InFIG. 2B , ER setting 214 and ER setting 216 are different and separate enhanced reality settings that are not shared byfirst user 202 andsecond user 204. - In
FIGS. 2A-2B ,first user 202 andsecond user 204 are not sharing an ER setting.FIGS. 2C-2G illustrate different example methods for two or more users (e.g.,first user 202 and second user 204) to enter into an ER setting that is shared by the two or more users. - In
FIG. 2C ,first user 202 andsecond user 204 have moved from their respective positions inFIG. 2A such that the two users are within a threshold distance (e.g., 5 meters, 3 meters, 1 meter) of one another inphysical setting 200. In some embodiments, in response to detecting (e.g., using one or more sensors of the device, such as an image sensor(s), orientation sensor(s), and/or a location sensor(s)) that the two users are within the predetermined threshold distance of one another, thereby satisfying a proximity criterion, firstelectronic device 206 displays, ondisplay 210 of the first electronic device, an ER setting 218 (e.g., a desert) that includes asecond avatar 222 corresponding tosecond user 204, as shown inFIG. 2G . - Likewise, in response to detecting (e.g., using one or more sensors of the device, such as an image sensor(s), orientation sensor(s), and/or a location sensor(s)) that the two users are within the predetermined threshold distance of one another, thereby satisfying the proximity criterion, second
electronic device 208 displays, ondisplay 212 of the second electronic device, ER setting 218 (e.g., a desert) that includes afirst avatar 220 corresponding tofirst user 202, as also shown inFIG. 2G . In this way, when firstelectronic device 206 and secondelectronic device 208 determines that they are within the threshold distance of one another in the physical setting, the devices provide their respective users with a shared ER setting in which the users can interact with one another using avatars. - In some embodiments, providing the respective users with the shared ER setting in which the users can interact with one another using avatars comprises
second user 204 appearing as an avatar in the existing ER setting (e.g., ER setting 214) that is provided by firstelectronic device 206 tofirst user 202. That is, in some embodiments, in response to detecting that the two users are within the predetermined threshold distance of one another, thereby satisfying the proximity criterion, secondelectronic device 208 displays, ondisplay 212 of the second electronic device, ER setting 214 (e.g., a room) that includesfirst avatar 220 corresponding tofirst user 202. Thus, in some embodiments,second user 204 appears as an avatar in the existing ER setting (e.g., ER setting 214) that is provided by firstelectronic device 206 tofirst user 202. In other embodiments,devices users - Optionally, in some embodiments (e.g., in response to detecting that the two users are within the predetermined threshold distance of one another), second
electronic device 208 first displays, ondisplay 212, a user interface affordance alertingsecond user 204 to the presence offirst user 202 of firstelectronic device 206 within the predetermined threshold distance in the physical setting without displaying a shared ER setting. In some embodiments, in response to detectingsecond user 204's confirmation (e.g., detecting the user's selection of the user interface affordance), secondelectronic device 208 enablessecond user 204 to participate in a shared ER setting with user 202 (e.g., share ER setting 214 with first user 202). - In some embodiments, first
electronic device 206 receives a request fromfirst user 202 to initialize an ER setting while the device is not displaying an ER setting (e.g., because the device is in an off state or an inactive state). In some embodiments, in response to receiving the request fromfirst user 202 to initialize an ER setting while the device is not already displaying an ER setting, firstelectronic device 206 detects for whether another device (e.g., second electronic device 208) is within a predetermined distance from firstelectronic device 206 such that the proximity criterion is satisfied. In some embodiments, in response to detecting that the proximity criterion is satisfied with another device (e.g., second electronic device 208), firstelectronic device 206 displays an ER setting (e.g., ER setting 218) that is shared by the user of the other device (e.g., second user 204), and thus includes an avatar corresponding to the user of the other device (e.g.,avatar 222 of second user 204). Alternatively, in other embodiments, in response to detecting that the proximity criterion is satisfied with another device (e.g., second electronic device 208), firstelectronic device 206 displays, ondisplay 210, a user interface affordance for proceeding with the initiation of the ER setting (e.g., ER setting 218) that is shared by the user of the other device (e.g., second user 204) without first displaying the shared ER setting, and displays the shared ER setting (e.g., ER setting 218) in response to detectingfirst user 202's selection or activation of the user interface affordance. In some embodiments, iffirst user 202's selection or activation of the user interface affordance is not detected (e.g., within a predetermined amount of time), firstelectronic device 206 forgoes displaying the shared ER setting (e.g., ER setting 218). In some examples, the user interface affordance for proceeding with the initiation of the shared ER setting includes an indication that another user (e.g., second user 204) is nearby and asks whether to initialize a shared ER setting that includes the other user detected nearby. -
FIG. 2D illustrates another exemplary method for two or more users (e.g.,first user 202 and second user 204) to enter into an ER setting that is shared by the two or more users. InFIG. 2D , firstelectronic device 206 detects (e.g., using an eye tracking sensor, an orientation sensor, and/or a location sensor) thatfirst user 202 is gazing at second user 204 (e.g., looking atsecond user 204 for at least a predetermined time period, such as 10 seconds, 5 seconds, or 3 seconds). - In some embodiments, in response to detecting the gaze by
first user 202 directed tosecond user 204, if firstelectronic device 206 was displaying an ER setting that was not shared with second user 204 (e.g., ER setting 214), firstelectronic device 206 transitions to displaying, ondisplay 210 of the device, an ER setting that is shared with second user 204 (e.g., ER setting 218), the target offirst user 202's gaze, where the ER setting that is shared withsecond user 204 includes an avatar (e.g., avatar 222) corresponding tosecond user 204 within the shared ER setting. Alternatively, in other embodiments, in response to detecting the gaze byfirst user 202 directed tosecond user 204, if firstelectronic device 206 was displaying an ER setting that was not shared with second user 204 (e.g., ER setting 214), firstelectronic device 206 displays, ondisplay 210, a user interface affordance for transitioning to the ER setting that is shared with second user 204 (e.g., ER setting 218), the target offirst user 202′s gaze, without first displaying the shared ER setting, and transitions to displaying the shared ER setting (e.g., ER setting 218) in response to detectingfirst user 202′s selection or activation of the user interface affordance. In some embodiments, iffirst user 202′s selection or activation of the user interface affordance is not detected (e.g., within a predetermined amount of time), firstelectronic device 206 forgoes transitioning to the shared ER setting (e.g., ER setting 218) and instead maintains display of the ER setting (e.g., ER setting 214) that was being displayed. In some examples, the user interface affordance for proceeding with the initiation of the shared ER setting includes an indication that another user (e.g., second user 204) is nearby and asks whether to initialize a shared ER setting that includes the other user detected nearby. - On the other hand, in some embodiments, in response to detecting that the gaze by
first user 202 directed tosecond user 204, if firstelectronic device 206 was not displaying an ER setting, firstelectronic device 206 displays, ondisplay 210 of the device, the ER setting shared with second user 204 (e.g., ER setting 218). - Further, in some embodiments, in response to detecting the gaze by
first user 202 directed tosecond user 204, firstelectronic device 206 causes secondelectronic device 208 to also display the ER setting that is shared betweenfirst user 202 and second user 204 (e.g., ER setting 218). As such, from the perspective ofsecond user 204, secondelectronic device 208 displays, ondisplay 212 of the device, the ER setting that is shared with first user 202 (e.g., ER setting 218) that includes an avatar (e.g., avatar 220) corresponding tofirst user 202 within the shared ER setting. -
FIG. 2E illustrates another example method for two or more users (e.g.,first user 202 and second user 204) to enter into an ER setting that is shared by the two or more users. InFIG. 2E , firstelectronic device 206 detects (e.g., using one or more cameras of the device; using one or more downward-facing cameras of the device if the device is a HIVID) thatfirst user 202 is engaging in a handshake withsecond user 204. In some embodiments, in response to detecting thatfirst user 202 is engaging in the handshake withsecond user 204, if firstelectronic device 206 was displaying an ER setting that was not shared with second user 204 (e.g., ER setting 214), firstelectronic device 206 transitions to displaying an ER setting that is shared with second user 204 (e.g., ER setting 218), where the ER setting that is shared withsecond user 204 includes an avatar (e.g., avatar 222) corresponding tosecond user 204 within the shared ER setting. - On the other hand, in some embodiments, in response to detecting that
first user 202 is engaging in the handshake withsecond user 204, if firstelectronic device 206 was not displaying an ER setting, firstelectronic device 206 displays, ondisplay 210 of the device, the ER setting that is shared with second user 204 (e.g., ER setting 218). - Similarly, in
FIG. 2E , secondelectronic device 206 detects (e.g., using one or more cameras of the device; using one or more downward-facing cameras of the device if the device is a HIVID) thatsecond user 204 is engaging in the handshake withfirst user 202. In some embodiments, in response to detecting thatsecond user 204 is engaging in the handshake withfirst user 202, if secondelectronic device 208 was displaying an ER setting that was not shared with first user 202 (e.g., ER setting 218), secondelectronic device 208 transitions to displaying an ER setting that is shared with first user 202 (e.g., ER setting 218), where the ER setting that is shared withfirst user 202 includes an avatar (e.g., avatar 220) corresponding tofirst user 202 within the shared ER setting. - On the other hand, in some embodiments, in response to detecting that
second user 204 is engaging in the handshake withfirst user 202, if secondelectronic device 208 was not displaying an ER setting, secondelectronic device 208 displays, ondisplay 212 of the device, the ER setting that is shared with first user 202 (e.g., ER setting 218). -
FIG. 2F illustrates another example method for two or more users (e.g.,first user 202 and second user 204) to enter into an ER setting that is shared by the two or more users. InFIG. 2F , firstelectronic device 206 detects (e.g., using one or more mics of the device) thatfirst user 202 is engaging in a conversation with second user 204 (e.g., an engaging conversation that has been ongoing for at least a predetermined amount of time, such as 10 seconds, 30 seconds, or 1 minute). In some embodiments, in response to detecting thatfirst user 202 is engaging in the conversation withsecond user 204, if firstelectronic device 206 was displaying an ER setting that was not shared with second user 204 (e.g., ER setting 214), firstelectronic device 206 transitions to displaying an ER setting that is shared with second user 204 (e.g., ER setting 218), where the ER setting that is shared withsecond user 204 includes an avatar (e.g., avatar 222) corresponding tosecond user 204 within the shared ER setting. - On the other hand, in some embodiments, in response to detecting that
first user 202 is engaging in the conversation withsecond user 204, if firstelectronic device 206 was not displaying an ER setting, firstelectronic device 206 displays, ondisplay 210 of the device, the ER setting that is shared with second user 204 (e.g., ER setting 218). - Similarly, in
FIG. 2F , secondelectronic device 206 detects (e.g., using one or more mics of the device) thatsecond user 204 is engaging in the conversation withfirst user 202. In some embodiments, in response to detecting thatsecond user 204 is engaging in the handshake withfirst user 202, if secondelectronic device 208 was displaying an ER setting that was not shared with first user 202 (e.g., ER setting 218), secondelectronic device 208 transitions to displaying an ER setting that is shared with first user 202 (e.g., ER setting 218), where the ER setting that is shared withfirst user 202 includes an avatar (e.g., avatar 220) corresponding tofirst user 202 within the shared ER setting. - On the other hand, in some embodiments, in response to detecting that
second user 204 is engaging in the conversation withfirst user 202, if secondelectronic device 208 was not displaying an ER setting, secondelectronic device 208 displays, ondisplay 212 of the device, the ER setting that is shared with first user 202 (e.g., ER setting 218). - As previously discussed,
FIG. 2G illustrates ER setting 220 (e.g., a desert) that includes bothavatar 220 corresponding tofirst user 202 andavatar 222 corresponding tosecond user 204, shown from the perspective offirst user 202 indisplay 210 of firstelectronic device 206 and from the perspective ofsecond user 204 indisplay 212 of secondelectronic device 208. ER setting 218 is an enhanced reality setting that was initialized (e.g., via one or more of the processes described above with reference toFIGS. 2C-2G ) to include bothfirst user 202 andsecond user 204. Thus, as shown inFIG. 2G , in contrast to ER setting 214 and ER setting 216 depicted inFIG. 2B ,first user 202 can see, within ER setting 218,avatar 222 corresponding tosecond user 204 and, similarly,second user 204 can see, within ER setting 218,avatar 220 corresponding tofirst user 202. - Additionally, ER setting 218 is not limited to being shared by two users. In some embodiments, ER setting 218 can be shared by a plurality of users, including three or more users. For example, three or more users can initialize a shared ER setting, such as ER setting 218, by satisfying the proximity criterion described above with reference to
FIG. 2C . - Furthermore, methods to initialize a shared ER setting is not limited to the triggering events (e.g., satisfying a proximity criterion, detecting a gaze, detecting a handshake, detecting an engaging conversation) described. For another example, the initialization of an ER setting (e.g., ER setting 218) shared by two or more users (e.g.,
first user 202 and second user 204) can be triggered by firstelectronic device 206 detecting (e.g., using an eye tracking sensor of the device) a wink made byfirst user 202 directed towardssecond user 204 and/or by secondelectronic device 208 detecting (e.g., using an eye tracking sensor of the device) a wink made bysecond user 204 directed towardsfirst user 202. For another example, the initialization of an ER setting (e.g., ER setting 218) shared by two or more users (e.g.,first user 202 and second user 204) can be triggered by firstelectronic device 206 detecting (e.g., using an orientation sensor and/or movement sensor of the device) a nod made byfirst user 202 directed towardssecond user 204 and/or by secondelectronic device 208 detecting (e.g., using an orientation sensor and/or movement sensor of the device) a nod made bysecond user 204 directed towardsfirst user 202. - It is noted that users of
electronic devices device 206 may allow its user to choose whether the device is discoverable by other devices such asdevice 208. Ifdevice 206 is not discoverable,device 208 will not alert itsuser 208 to the presence ofdevice 206 even if the two devices are in proximity. As another example,device 206 may allowuser 202 to provide a list of trusted contacts with whom shared experiences in ER settings may be allowed. Conversely,device 206 may allowuser 202 to specify limits, e.g., a blocked list, of users with whom ER settings are not to be allowed. As another example,device 206 may expressly confirm withuser 202 as to whether to allow or disallow a particular ER session. In these ways, the present techniques preserve user privacy while improving a user's ability to easily invite others to share an ER setting as appropriate. -
FIG. 3 is a flow diagram for initializing members of a shared ER setting, in accordance with some embodiments. In some embodiments, the flow ofFIG. 3 is performed using an electronic device (e.g.,electronic device 100a,electronic device 206,electronic device 208; a mobile electronic device such as a smartphone or a tablet computer; a HIVID) that is in communication with another electronic device (e.g.,electronic device 100a,electronic device 206,electronic device 208; a mobile electronic device such as a smartphone or a tablet computer; a HIVID) and/or an external server. - In some embodiments, flow 300 of
FIG. 3 is performed at a first electronic device (e.g., 206) having one or more sensors and one or more displays adapted to present an ER setting (e.g., 212, 214, 218; a VR setting; a MR setting, including an AR setting) and communicate with a second electronic device (e.g., 208) adapted to present the ER setting. - The first electronic device (e.g., 206) receives, at
block 302, a request to present the ER setting (e.g., 218). - The first electronic device (e.g., 206) determines, at
block 304, whether a first user (e.g., 202) of the first electronic device (e.g., 206) and a second user (e.g., 204) of the second electronic device (e.g., 208) satisfy a physical interaction criterion (e.g., a criterion based on user actions in a physical setting (e.g., 200), such as proximity between the first user and the second user or a social action (e.g., a handshake, gaze) between the first user and the second user). - In accordance with a determination, at
block 306, that the physical interaction criterion is satisfied, the first electronic device (e.g., 206) presents, atblock 308, the ER setting (e.g., 218), wherein presenting the ER setting includes displaying an avatar (e.g., 222) representing the second user (e.g., 204) in the ER setting. - In accordance with a determination, at
block 310, that the physical interaction criterion is not satisfied, the first electronic device (e.g., 206) presents, atblock 312, the ER setting (e.g., 214) without the avatar (e.g., 222) representing the second user (e.g., 204). - In some embodiments, presenting the ER setting (e.g., 218) comprises: displaying, at the first electronic device (e.g., 206), a confirmation affordance requesting confirmation for entering the ER setting with the second user (e.g., 204); and in response to detecting confirmation by the first user (e.g., 202), presenting the ER setting that includes the avatar (e.g., 222) representing the second user.
- In some embodiments, the physical interaction criterion being satisfied includes the first user (e.g., 202) and the second user (e.g., 204) being within a predetermined distance of one another within a physical setting (e.g., 200).
- In some embodiments, the physical interaction criterion being satisfied includes a social signal between the first user (e.g., 202) and the second user (e.g., 204) being detected within the physical setting (e.g., 200).
- In some embodiments, the social signal is a handshake performed between the first user (e.g., 202) and the second user (e.g., 204) within the physical setting (e.g., 200).
- In some embodiments, the social signal is a conversation performed between the first user (e.g., 202) and the second user (e.g., 204) within the physical setting (e.g., 200). In some embodiments, the conversation is an engaging conversation such that the first user and the second user have been engaging in a continuous conversation for at least a predetermined amount of time.
- In some embodiments, the social signal is a gaze (e.g., that is maintained for at least a predetermined amount of time) by the first user (e.g., 202) directed to the second user (e.g., 204) within the physical setting (e.g., 200). In some embodiments, the social signal is a nod by the first user directed to the second user within the physical setting. In some embodiments, the social signal is a wink by the first user directed to the second user within the physical setting.
- In some embodiments, the first electronic device (e.g., 206) includes one or more sensors (e.g., one or more cameras, one or more movement sensors) for detecting a movement corresponding to a signal action (e.g., movement to perform a handshake, gazing at the second user, winking at the second user, nodding at the second user) made by the first user (e.g., 202) within the physical setting (e.g., 200).
-
FIGS. 4A-4I depict an exemplary process for forming a private sub-space within an ER setting that includes a subset of the avatars within the ER setting (e.g., a sub-enhanced reality (sub-ER) setting within the ER setting) and example features of the private sub-space. -
FIG. 4A illustrates an ER setting 400 (e.g., a virtual room initialized using one or more methods described above with reference toFIGS. 2A-2G ), where ER setting 400 includes a plurality of avatars, including afirst avatar 402 corresponding to a first user of a first electronic device (e.g., similar to firstelectronic device 206 and second electronic device 208), asecond avatar 404 corresponding to a second user of a second electronic device (e.g., similar to firstelectronic device 206 and second electronic device 208), and athird avatar 406 corresponding to a third user of a third electronic device (e.g., similar to firstelectronic device 206 and second electronic device 208). That is,first avatar 402 is a virtual representation of, and is controlled by, the first user,second avatar 404 is a virtual representation of, and is controlled by, the second user, andthird avatar 406 is a virtual representation of, and is controlled by, the third user. - In
FIG. 4A ,first avatar 402,second avatar 404, andthird avatar 406 are included within ER setting 400 such that interactions supported by ER setting 400 are made available toavatars FIG. 4A , audio corresponding to an audio input (e.g., a spoken input, a voice input) made by the second user (e.g., captured via a mic of the second electronic device of the second user) viasecond avatar 404 is played at the devices of both the first user offirst avatar 402 and the third user ofthird avatar 406. Further, inFIG. 4A , movements made bysecond avatar 404 within ER setting 400 is displayed to the first user offirst avatar 402 via a display of the first device, and displayed to the third user ofthird avatar 406 via a display of the third device. - In
FIG. 4B , while the three avatars are included in ER setting 400, a determination is made (e.g., by an external server in communication with the first device of the first user, the second device of the second user, and the third device of the third user and/or by one or more of the first device, second device, and third device) thatsecond avatar 404 is gazing at (e.g., looking at for at least predetermined time period, such as at least 10 seconds, 5 seconds, or 3 seconds)first avatar 402. In some embodiments, the gaze triggers the formation of a private sub-space within the ER setting (e.g., a sub-ER setting within the ER setting that a subset of, but not all of, the avatars present within the ER setting is included in). - In
FIG. 4C , in response to the determination thatsecond avatar 404 is gazing atfirst avatar 402, aprivate sub-space 408 that includesfirst avatar 402 andsecond avatar 404 but does not includethird avatar 406 is formed within ER setting 400. In some embodiments, as shown inFIG. 4C , avisual indication 410 of the formed private sub-space (e.g., a halo or a partially transparent cone that encompasses the avatars within the private sub-space) is further provided to indicate to the users of other avatars within ER setting 400 (e.g., the user of third avatar 406) thatfirst avatar 402 andsecond avatar 404 are engaging in a private conversation in a sub-space within the ER setting (e.g., an ER-equivalent of texting under the table). -
FIG. 4D illustrates another example method forfirst avatar 402 andsecond avatar 404 to enter into a private sub-space, such asprivate sub-space 408, within ER setting 400. In some embodiments, each avatar of the plurality of avatars in ER setting 400 (e.g.,first avatar 402,second avatar 404, and third avatar 406) have a corresponding selectable affordance (e.g., displayed above or adjacent to the respective avatar), as shown inFIG. 4D . For example,first avatar 402 has aselectable affordance 402A,second avatar 404 has a selectable affordance 404A, andthird avatar 406 has aselectable affordance 406A. InFIG. 4D ,second avatar 404 selects/activatesselectable affordance 402A offirst avatar 402. In some embodiments,second avatar 404′s selection/activation ofselectable affordance 402A offirst avatar 402 triggers the formation ofprivate sub-space 408 that includesfirst avatar 402 andsecond avatar 404 but does not includethird avatar 406, as previously shown inFIG. 4C . - Methods for the creation of a private sub-space (e.g., private sub-space 408) is not limited to the two examples illustrated in
FIG. 4B and 4D . Another example method forfirst avatar 402 andsecond avatar 404 to form a private sub-space includesfirst avatar 402 andsecond avatar 404 engaging in a particular type of social interaction (e.g., a nod, a wink, physical contact). For example, the formation of a private sub-space betweenfirst avatar 402 andsecond avatar 404 can be triggered by a nod made bysecond avatar 404 directed tofirst avatar 402. For another example, the formation of a private sub-space betweenfirst avatar 402 andsecond avatar 404 and be triggered by a wink made bysecond avatar 404 directed tofirst avatar 402. For another example, the formation of a private sub-space betweenfirst avatar 402 andsecond avatar 404 can be triggered by a physical contact (e.g.,second avatar 404 putting a hand on the shoulder of first avatar 402) between the two avatars. - Furthermore, another example method for
first avatar 402 andsecond avatar 404 to form a private sub-space (e.g., private sub-space 408) includesfirst avatar 402 andsecond avatar 404 satisfying a proximity criterion. For example, the formation of a private sub-space can be triggered byfirst avatar 402 andsecond avatar 404 moving to be within a predetermined distance of one another within ER setting 400 (e.g., and also maintaining the proximity for at least a predetermined time period). - Additionally, a private sub-space (e.g., private sub-space 408) within an ER setting is not limited to being shared by two avatars. In some embodiments, a private sub-space can be shared by a plurality of avatars, including three or more avatars. For example, three avatars (e.g.,
first avatar 402,second avatar 404, and third avatar 406) can form a private sub-space, such asprivate sub-space 408, by moving to be within a predetermined distance of one another within the ER setting, thereby satisfying a proximity criterion to trigger the formation of a sub-space encompassing the three avatars. -
FIG. 4E illustrates ER setting 400 that includesprivate sub-space 408 includingfirst avatar 402 andsecond avatar 404 but not includingthird avatar 406, as previously depicted inFIG. 4C . InFIG. 4E , audio 412 corresponding to an audio input (e.g., a spoken input, a voice input) made by the first user (e.g., captured via a mic of the first device of the first user) viafirst avatar 402 is played at the second device of the second user ofsecond avatar 404, asfirst avatar 402 andsecond avatar 404 are included inprivate sub-space 408. However,audio 412 is not played (e.g., it is prevented/blocked from being played) at the third device of the third user of third avatar 406 (and at the device of any other user of any other avatar within ER setting 400), asthird avatar 406 is not included inprivate sub-space 408. - In some embodiments, as shown in
FIG. 4E , whileaudio 412 corresponding to the audio input made by the first user viafirst avatar 412 is being played at the second device of the second user ofsecond avatar 404 and not being played at the third device of the third user ofthird avatar 406, movements offirst avatar 402 corresponding to audio 412 (e.g., moving lips and/or other physical gestures such as hand gestures) can still be seen by third avatar 406 (and any other avatar in ER setting 400) from outsideprivate sub-space 408. Thus, in some embodiments, a user of an avatar that is outside of the private sub-space (e.g., the third user of third avatar 406) cannot hear audio corresponding toaudio 412 but can still see movements (e.g., moving lips, hand gestures) corresponding toaudio 412 via the device of the user. - In some embodiments, as shown in
FIG. 4F , whileaudio 412 corresponding to the audio input made by the first user viafirst avatar 402 is being played at the second device of the second user ofsecond avatar 404 and not being played at the third device of the third user ofthird avatar 406, movements offirst avatar 402 corresponding toaudio 412 cannot be seen by third avatar 406 (and by any other avatar in ER setting 400) from outside ofprivate sub-space 408. Thus, in some embodiments, a user of an avatar that is outside of the private sub-space (e.g., the third user of third avatar 406) cannot hear audio corresponding toaudio 412 nor see movements corresponding toaudio 412. Further, in some embodiments, a user of an avatar that is outside of the private sub-space (e.g., third avatar 406) cannot see any movements of avatars within the private sub-space (e.g.,first avatar 402 and second avatar 404) via the device of the user. -
FIG. 4G illustrates ER setting 400 withfirst avatar 402 andsecond avatar 404 in aprivate sub-space 414 andthird avatar 406 not included inprivate sub-space 414. In some embodiments, as shown inFIG. 4G ,private sub-space 414 does not includevisual indication 410 ofprivate sub-space 408 that indicates to outside avatars (e.g., third avatar 406) thatfirst avatar 402 andsecond avatar 404 is in a private sub-space. Instead, in some embodiments,private sub-space 414 includes avisual indication 416A above or adjacent to first avatar 402 (e.g., a glowing/shining light source) and a similarvisual indication 416B above or adjacent tosecond avatar 404, thereby indicating to outside avatars (e.g., third avatar 406) thatfirst avatar 402 andsecond avatar 404 are in a private sub-space. -
FIG. 4H illustrates an ER setting 420 that includes a first avatar 422 (corresponding to a first user of a first electronic device), a second avatar 424 (corresponding to a second user of a second electronic device), a third avatar 426 (corresponding to a third user of a third electronic device), and a fourth avatar 428 (corresponding to a fourth user of a fourth electronic device). As shown inFIG. 4H ,first avatar 422 andsecond avatar 424 is in a firstprivate sub-space 430 within ER setting 420 andthird avatar 426 andfourth avatar 428 is in a secondprivate sub-space 432. - In
FIG. 41 ,audio 434 corresponding to an audio input (e.g., a spoken input, a voice input) made by the second user (e.g., captured via a mic of the second device of the second user) viasecond avatar 424 is played at the first device of the first user offirst avatar 422, asfirst avatar 422 andsecond avatar 424 are included in the sameprivate sub-space 430. However,audio 434 is not played (e.g., it is prevented/blocked from being played) at the third device of the third user ofthird avatar 426 and at the fourth device of the fourth user offourth avatar 428, as boththird avatar 426 andfourth avatar 428 are not included in first private sub-space 430 (and are instead included in a different private sub-space 432). - Further in
FIG. 41 ,audio 436 corresponding to an audio input (e.g., a spoken input, a voice input) made by the third user (e.g., captured via a mic of the third device of the third user) viathird avatar 426 is played at the fourth device of the fourth user offourth avatar 428, asthird avatar 426 andfourth avatar 428 are included in the sameprivate sub-space 432. However,audio 436 is not played (e.g., it is prevented/blocked from being played) at the first device of the first user offirst avatar 422 and at the second device of the second user ofsecond avatar 424, as bothfirst avatar 422 andsecond avatar 424 are not included in second private sub-space 432 (and are instead included in a different private sub-space 430). -
FIG. 5 is a flow diagram for forming and maintaining a private sub-space within an ER setting, in accordance with some embodiments. In some embodiments, flow 500 ofFIG. 5 is performed by a system that includes one or more electronic devices (e.g.,electronic device 100 a,electronic device 206,electronic device 208; a mobile electronic device such as a smartphone or a tablet computer; a HMD) and/or an external server, where the one or more electronic devices are in communication with each other and/or with the external server. - At
block 502, the system (e.g., 100a, 206, 208) displays (e.g., at an electronic device, such aselectronic device - At
block 504, while displaying the ER setting (e.g., 400), the system selects, atblock 506, a first avatar (e.g., 404) of the plurality of avatars (e.g., 402, 404, 406) as a recipient of received audio input (e.g., 412, an audio input made by a user corresponding to a third avatar of the plurality of avatars). - At
block 504, while displaying the ER setting (e.g. 400), the system also selects, atblock 508, a second avatar (e.g., 406) of the plurality of avatars (e.g., 402, 404, 406) as a non-recipient of the received audio input. - At
block 504, while displaying the ER setting (e.g., 400), the system also receives, atblock 510, via a microphone (e.g., of the electronic device of the current user), audio input (e.g., 412). - At
block 504, while displaying the ER setting (e.g., 400), the system also causes, atblock 512, playback and a corresponding visual avatar movement for the first avatar (e.g., 404). - At
block 504, while displaying ER setting (e.g., 400), the system also forgoes causing, atblock 514, playback of the audio (e.g., 412) for the second avatar (e.g., 406). - In some embodiments, causing playback and the corresponding visual avatar movement for the first avatar (e.g., 404) comprises causing playback and the corresponding visual avatar movement for the first avatar in accordance with a determination that the first avatar is within (e.g., sharing with a current avatar) a sub-ER setting (e.g., 408) that includes the first avatar and does not include the second avatar (e.g., 406).
- In some embodiments, the determination that the first avatar (e.g., 404) is within the sub-ER setting (e.g., 408) that includes the first avatar and does not include the second avatar is based on whether the first avatar satisfied a sub-ER setting criterion (e.g., with a third avatar of the plurality of avatars). In some embodiments, satisfying the sub-ER setting criterion with a third avatar of the plurality of avatars enables the first avatar to enter and/or be associated with the sub-ER setting with the third avatar.
- In some embodiments, the sub-ER setting criterion being satisfied includes the first avatar being within a predetermined distance from a third avatar (e.g., 402) of the plurality of avatars within the ER setting (e.g., 400).
- In some embodiments, the sub-ER setting criterion being satisfied includes a gaze by the first avatar (e.g., 404) directed to a third avatar (e.g., 402) of the plurality of avatars being detected within the ER setting (e.g., 400).
- In some embodiments, the sub-ER setting criterion being satisfied includes a nod by the first avatar (e.g., 404) directed to a third avatar (e.g., 402) of the plurality of avatars being detected within the ER setting (e.g., 400).
- In some embodiments, the sub-ER setting criterion being satisfied includes a wink by the first avatar (e.g., 404) directed to a third avatar (e.g., 402) of the plurality of avatars being detected within the ER setting (e.g., 400).
- In some embodiments, the determination that the first avatar (e.g., 404) is within the sub-ER setting (e.g., 408) that includes the first avatar (e.g., 404) and does not include the second avatar (e.g., 406) is based on whether an affordance (e.g., 404A) associated with the first avatar has been selected (e.g., by a third avatar of the plurality of avatars). In some embodiments, additionally or alternatively, the determination that the first avatar is within the sub-ER setting that includes the first avatar and does not include the second avatar is based on whether an affordance associated with a third avatar of the plurality of avatars has been selected by the first avatar.
- In some embodiments, while displaying the ER setting (e.g., 400), the system presents (e.g., to a fourth avatar of the plurality of avatars) a visual indication (e.g.,416B, a light shown above or proximate to the first avatar) for the first avatar indicating that the first avatar is a recipient of the received user input. In some embodiments, while displaying the ER setting, the system forgoes presenting (e.g., to the fourth avatar of the plurality of avatars) the visual indication for the second avatar (e.g., because the second avatar is a non-recipient of the received user input).
- In some embodiments, the visual indication (e.g., 416B) for the first avatar is a light source shown proximate to (e.g., above the head of) the first avatar (e.g., 404) within the ER setting (e.g., 400).
- Various processes defined herein consider the option of obtaining and utilizing a user's personal information. For example, such personal information may be utilized in order to provide an improved experience for members of a shared enhanced reality setting on one or more electronic devices. However, to the extent such personal information is collected, such information should be obtained with the user's informed consent. As described herein, the user should have knowledge of and control over the use of their personal information.
- Personal information will be utilized by appropriate parties only for legitimate and reasonable purposes. Those parties utilizing such information will adhere to privacy policies and practices that are at least in accordance with appropriate laws and regulations. In addition, such policies are to be well-established, user-accessible, and recognized as in compliance with or above governmental/industry standards. Moreover, these parties will not distribute, sell, or otherwise share such information outside of any reasonable and legitimate purposes.
- Users may, however, limit the degree to which such parties may access or otherwise obtain personal information. For instance, settings or other preferences may be adjusted such that users can decide whether their personal information can be accessed by various entities. Furthermore, while some features defined herein are described in the context of using personal information, various aspects of these features can be implemented without the need to use such information. As an example, if user preferences, account names, and/or location history are gathered, this information can be obscured or otherwise generalized such that the information does not identify the respective user.
Claims (24)
1. A method, comprising:
at a first electronic device having one or more sensors and one or more displays and adapted to communicate with a second electronic device:
receiving a request to present an enhanced reality setting; and
determining whether a first user of the first electronic device and a second user of the second electronic device satisfy a physical interaction criterion:
in accordance with a determination that the physical interaction criterion is satisfied, presenting the enhanced reality setting, wherein presenting the enhanced reality setting includes displaying an avatar representing the second user in the enhanced reality setting; and
in accordance with a determination that the physical interaction criterion is not satisfied, presenting the enhanced reality setting without the avatar representing the second user.
2. The method of claim 1 , wherein presenting the enhanced reality setting comprises:
displaying, at the first electronic device, a confirmation affordance requesting confirmation for entering the enhanced reality setting with the second user; and
in response to detecting confirmation by the first user, presenting the enhanced reality setting that includes the avatar representing the second user.
3. The method of claim 1 , wherein the physical interaction criterion being satisfied includes the first user and the second user being within a predetermined distance of one another within a physical setting.
4. The method of claim 1 , wherein the physical interaction criterion being satisfied includes a social signal between the first user and the second user being detected within a physical setting.
5. The method of claim 4 , wherein the social signal is a handshake performed between the first user and the second user within the physical setting.
6. The method of claim 4 , wherein the social signal is a conversation performed between the first user and the second user within the physical setting.
7. The method of claim 4 , wherein the social signal is a gaze by the first user directed to the second user within the physical setting.
8. The method of claim 1 , wherein the first electronic device includes one or more sensors for detecting a movement corresponding to a signal action made by the first user within the physical setting.
9. A first electronic device, comprising:
one or more sensors;
one or more displays;
a wireless communication radio configured to communicate with a second electronic device;
one or more processors; and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving a request to present an enhanced reality setting; and
determining whether a first user of the first electronic device and a second user of the second electronic device satisfy a physical interaction criterion:
in accordance with a determination that the physical interaction criterion is satisfied, presenting the enhanced reality setting, wherein presenting the enhanced reality setting includes displaying an avatar representing the second user in the enhanced reality setting; and
in accordance with a determination that the physical interaction criterion is not satisfied, presenting the enhanced reality setting without the avatar representing the second user.
10. The first electronic device of claim 9 , wherein presenting the enhanced reality setting comprises:
displaying, at the first electronic device, a confirmation affordance requesting confirmation for entering the enhanced reality setting with the second user; and
in response to detecting confirmation by the first user, presenting the enhanced reality setting that includes the avatar representing the second user.
11. The first electronic device of claim 9 , wherein the physical interaction criterion being satisfied includes the first user and the second user being within a predetermined distance of one another within a physical setting.
12. The first electronic device of claim 9 , wherein the physical interaction criterion being satisfied includes a social signal between the first user and the second user being detected within a physical setting.
13. The first electronic device of claim 12 , wherein the social signal is a handshake performed between the first user and the second user within the physical setting.
14. The first electronic device of claim 12 , wherein the social signal is a conversation performed between the first user and the second user within the physical setting.
15. The first electronic device of claim 12 , wherein the social signal is a gaze by the first user directed to the second user within the physical setting.
16. The first electronic device of claim 9 , wherein the first electronic device includes one or more sensors for detecting a movement corresponding to a signal action made by the first user within the physical setting.
17. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first electronic device having one or more sensors and one or more displays and adapted to communicate with a second electronic device, the one or more programs including instructions for:
receiving a request to present an enhanced reality setting; and
determining whether a first user of the first electronic device and a second user of the second electronic device satisfy a physical interaction criterion:
in accordance with a determination that the physical interaction criterion is satisfied, presenting the enhanced reality setting, wherein presenting the enhanced reality setting includes displaying an avatar representing the second user in the enhanced reality setting; and
in accordance with a determination that the physical interaction criterion is not satisfied, presenting the enhanced reality setting without the avatar representing the second user.
18. The non-transitory computer-readable storage medium of claim 17 , wherein presenting the enhanced reality setting comprises:
displaying, at the first electronic device, a confirmation affordance requesting confirmation for entering the enhanced reality setting with the second user; and
in response to detecting confirmation by the first user, presenting the enhanced reality setting that includes the avatar representing the second user.
19. The non-transitory computer-readable storage medium of claim 17 , wherein the physical interaction criterion being satisfied includes the first user and the second user being within a predetermined distance of one another within a physical setting.
20. The non-transitory computer-readable storage medium of claim 17 , wherein the physical interaction criterion being satisfied includes a social signal between the first user and the second user being detected within a physical setting.
21. The non-transitory computer-readable storage medium of claim 20 , wherein the social signal is a handshake performed between the first user and the second user within the physical setting.
22. The non-transitory computer-readable storage medium of claim 20 , wherein the social signal is a conversation performed between the first user and the second user within the physical setting.
23. The non-transitory computer-readable storage medium of claim 20 , wherein the social signal is a gaze by the first user directed to the second user within the physical setting.
24. The non-transitory computer-readable storage medium claim 17 , wherein the first electronic device includes one or more sensors for detecting a movement corresponding to a signal action made by the first user within the physical setting.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/359,169 US20210327140A1 (en) | 2019-04-10 | 2021-06-25 | Techniques for participation in a shared setting |
US17/900,657 US11908086B2 (en) | 2019-04-10 | 2022-08-31 | Techniques for participation in a shared setting |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201962832161P | 2019-04-10 | 2019-04-10 | |
PCT/US2020/027186 WO2020210298A1 (en) | 2019-04-10 | 2020-04-08 | Techniques for participation in a shared setting |
US17/359,169 US20210327140A1 (en) | 2019-04-10 | 2021-06-25 | Techniques for participation in a shared setting |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2020/027186 Continuation WO2020210298A1 (en) | 2019-04-10 | 2020-04-08 | Techniques for participation in a shared setting |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/900,657 Continuation US11908086B2 (en) | 2019-04-10 | 2022-08-31 | Techniques for participation in a shared setting |
Publications (1)
Publication Number | Publication Date |
---|---|
US20210327140A1 true US20210327140A1 (en) | 2021-10-21 |
Family
ID=70476464
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/359,169 Abandoned US20210327140A1 (en) | 2019-04-10 | 2021-06-25 | Techniques for participation in a shared setting |
US17/900,657 Active US11908086B2 (en) | 2019-04-10 | 2022-08-31 | Techniques for participation in a shared setting |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/900,657 Active US11908086B2 (en) | 2019-04-10 | 2022-08-31 | Techniques for participation in a shared setting |
Country Status (3)
Country | Link |
---|---|
US (2) | US20210327140A1 (en) |
CN (1) | CN113646731A (en) |
WO (1) | WO2020210298A1 (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11733956B2 (en) * | 2018-09-04 | 2023-08-22 | Apple Inc. | Display device sharing and interactivity |
EP4234055A1 (en) * | 2022-02-28 | 2023-08-30 | Apple Inc. | System and method of three-dimensional placement and refinement in multi-user communication sessions |
US11812194B1 (en) | 2019-06-21 | 2023-11-07 | Apple Inc. | Private conversations in a virtual setting |
US11908086B2 (en) | 2019-04-10 | 2024-02-20 | Apple Inc. | Techniques for participation in a shared setting |
US11995230B2 (en) | 2022-02-11 | 2024-05-28 | Apple Inc. | Methods for presenting and sharing content in an environment |
Family Cites Families (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5736982A (en) * | 1994-08-03 | 1998-04-07 | Nippon Telegraph And Telephone Corporation | Virtual space apparatus with avatars and speech |
US20090089685A1 (en) * | 2007-09-28 | 2009-04-02 | Mordecai Nicole Y | System and Method of Communicating Between A Virtual World and Real World |
WO2009104564A1 (en) * | 2008-02-20 | 2009-08-27 | インターナショナル・ビジネス・マシーンズ・コーポレーション | Conversation server in virtual space, method for conversation and computer program |
US8875026B2 (en) * | 2008-05-01 | 2014-10-28 | International Business Machines Corporation | Directed communication in a virtual environment |
US20090303984A1 (en) | 2008-06-09 | 2009-12-10 | Clark Jason T | System and method for private conversation in a public space of a virtual world |
US20100169796A1 (en) * | 2008-12-28 | 2010-07-01 | Nortel Networks Limited | Visual Indication of Audio Context in a Computer-Generated Virtual Environment |
EP2462466B1 (en) | 2009-08-05 | 2016-01-06 | Ford Global Technologies, LLC | System and method for transmitting vehicle information to an occupant communication device |
US8954517B2 (en) | 2010-12-29 | 2015-02-10 | Avaya Inc. | Method and apparatus for delegating a message |
US8725197B2 (en) | 2011-12-13 | 2014-05-13 | Motorola Mobility Llc | Method and apparatus for controlling an electronic device |
US9524588B2 (en) | 2014-01-24 | 2016-12-20 | Avaya Inc. | Enhanced communication between remote participants using augmented and virtual reality |
US9922667B2 (en) | 2014-04-17 | 2018-03-20 | Microsoft Technology Licensing, Llc | Conversation, presence and context detection for hologram suppression |
EP3134847A1 (en) * | 2014-04-23 | 2017-03-01 | Google, Inc. | User interface control using gaze tracking |
US9818225B2 (en) | 2014-09-30 | 2017-11-14 | Sony Interactive Entertainment Inc. | Synchronizing multiple head-mounted displays to a unified space and correlating movement of objects in the unified space |
WO2016118656A1 (en) | 2015-01-21 | 2016-07-28 | Harman International Industries, Incorporated | Techniques for amplifying sound based on directions of interest |
WO2018005673A1 (en) * | 2016-06-28 | 2018-01-04 | Against Gravity Corp. | Systems and methods providing temporary decoupling of user avatar synchronicity for presence enhancing experiences |
US10657701B2 (en) | 2016-06-30 | 2020-05-19 | Sony Interactive Entertainment Inc. | Dynamic entering and leaving of virtual-reality environments navigated by different HMD users |
US10572005B2 (en) * | 2016-07-29 | 2020-02-25 | Microsoft Technology Licensing, Llc | Private communication with gazing |
US10375125B2 (en) * | 2017-04-27 | 2019-08-06 | Cisco Technology, Inc. | Automatically joining devices to a video conference |
EP3757728A1 (en) * | 2017-05-16 | 2020-12-30 | Apple Inc. | Image data for enhanced user interactions |
CN111164540B (en) * | 2017-09-28 | 2022-04-15 | 苹果公司 | Method and apparatus for presenting physical environment interactions during a simulated reality session |
US10732710B2 (en) * | 2018-06-26 | 2020-08-04 | Sony Interactive Entertainment Inc. | Privacy chat trigger using mutual eye contact |
CN113646731A (en) | 2019-04-10 | 2021-11-12 | 苹果公司 | Techniques for participating in a shared setting |
-
2020
- 2020-04-08 CN CN202080023769.9A patent/CN113646731A/en active Pending
- 2020-04-08 WO PCT/US2020/027186 patent/WO2020210298A1/en active Application Filing
-
2021
- 2021-06-25 US US17/359,169 patent/US20210327140A1/en not_active Abandoned
-
2022
- 2022-08-31 US US17/900,657 patent/US11908086B2/en active Active
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11733956B2 (en) * | 2018-09-04 | 2023-08-22 | Apple Inc. | Display device sharing and interactivity |
US11908086B2 (en) | 2019-04-10 | 2024-02-20 | Apple Inc. | Techniques for participation in a shared setting |
US11812194B1 (en) | 2019-06-21 | 2023-11-07 | Apple Inc. | Private conversations in a virtual setting |
US11995230B2 (en) | 2022-02-11 | 2024-05-28 | Apple Inc. | Methods for presenting and sharing content in an environment |
EP4234055A1 (en) * | 2022-02-28 | 2023-08-30 | Apple Inc. | System and method of three-dimensional placement and refinement in multi-user communication sessions |
Also Published As
Publication number | Publication date |
---|---|
WO2020210298A1 (en) | 2020-10-15 |
US11908086B2 (en) | 2024-02-20 |
US20230260217A1 (en) | 2023-08-17 |
CN113646731A (en) | 2021-11-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11086581B2 (en) | Controlling external devices using reality interfaces | |
US11908086B2 (en) | Techniques for participation in a shared setting | |
US11800059B2 (en) | Environment for remote communication | |
EP3676745B1 (en) | Privacy screen for computer simulated reality | |
US11836282B2 (en) | Method and device for surfacing physical environment interactions during simulated reality sessions | |
US11880911B2 (en) | Transitioning between imagery and sounds of a virtual environment and a real environment | |
US11222454B1 (en) | Specifying effects for entering or exiting a computer-generated reality environment | |
US11694658B2 (en) | Transferring a virtual object | |
US20230343049A1 (en) | Obstructed objects in a three-dimensional environment | |
US11968056B2 (en) | Avatar spatial modes | |
US20230308495A1 (en) | Asymmetric Presentation of an Environment | |
US11947733B2 (en) | Muting mode for a virtual object representing one or more physical elements | |
US20240104877A1 (en) | Methods for time of day adjustments for environments and environment presentation during communication sessions |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |