CN113785260A - Controlling representations of virtual objects in a computer-generated real-world environment - Google Patents

Abstract

According to some embodiments, an example process for controlling a representation of a virtual object based on a user context of a location in a Computer Generated Reality (CGR) environment is described. According to other embodiments, an exemplary process for controlling the simultaneous display of representations of one or more virtual objects within a CGR environment is described. According to other embodiments, an exemplary process for controlling a representation of a virtual object in a CGR environment based on characteristics of an input mechanism is described.

Description

Controlling representations of virtual objects in a computer-generated real-world environment

Cross Reference to Related Applications

The present application claims priority from U.S. provisional patent application Ser. No. 63/058,217 entitled "control resetting OF LOCATIONS OF VIRTUAL OBJECTS IN A COMPLEMENT-GENERATED REALITY ENVIRONMENT" filed ON 29.7.2020 and U.S. provisional patent application Ser. No. 62/907,216 entitled "control resetting OF LOCATIONS OF VIRTUAL OBJECTS BASED ON USE OF CONTEXT OF LOCATIONS IN A COMPLEMENT-GENERATED REALITY ENVIRONMENT" filed ON 27.9.2019. The contents of the aforementioned application are hereby incorporated by reference in their entirety.

Technical Field

The present disclosure relates generally to computer-generated reality environments, and more particularly to controlling the display of virtual objects in computer-generated reality environments.

Background

A computer-generated reality (CGR) environment is an environment generated by a computer in which at least some objects displayed for viewing by a user are displayed. In some applications, a user may interact with a virtual object. The user may move the virtual object to a different location within the computer-generated reality environment. However, robust functionality for controlling how virtual objects are represented at different locations within a computer-generated reality environment is lacking. Further, functionality is lacking for controlling representations of virtual objects in a computer-generated environment based on characteristics of input mechanisms available to a user for interacting with the virtual objects.

Further, the virtual object may be represented as a two-dimensional object and/or a three-dimensional object within the computer-generated reality environment. However, robust functionality for controlling simultaneous display of representations of virtual objects is lacking.

Disclosure of Invention

According to some embodiments, a method comprises: displaying, via a display of an electronic device, a first representation of a virtual object at a first location within a CGR environment, wherein the first location corresponds to a first usage context of a plurality of usage contexts; receiving a request to move a first representation within the CGR environment to a second location different from the first location; and in response to receiving the request: in accordance with a determination that the second location corresponds to a second usage context of the plurality of usage contexts, displaying, via the display of the electronic device, a second representation of the virtual object at the second location based on the second usage context, wherein the second representation is different from the first representation; and in accordance with a determination that the second location corresponds to a third usage context of the plurality of usage contexts, displaying, via the display of the electronic device, a third representation of the virtual object at the second location based on the third usage context, wherein the third representation is different from the first representation and the second representation.

According to some embodiments, a system comprises: a display; one or more processors; and memory storing one or more programs configured for execution by the one or more processors. The one or more programs include instructions for: displaying, via a display, a first representation of a virtual object at a first location within the CGR environment, wherein the first location corresponds to a first usage context of a plurality of usage contexts; receiving a request to move a first representation within the CGR environment to a second location different from the first location; and in response to receiving the request: in accordance with a determination that the second location corresponds to a second usage context of the plurality of usage contexts, displaying, via the display, a second representation of the virtual object at the second location based on the second usage context, wherein the second representation is different from the first representation; and in accordance with a determination that the second location corresponds to a third usage context of the plurality of usage contexts, displaying, via the display, a third representation of the virtual object at the second location based on the third usage context, wherein the third representation is different from the first representation and the second representation.

According to some embodiments, a non-transitory computer readable storage medium stores one or more programs configured for execution by one or more processors of a system with a display, the one or more programs including instructions for: displaying, via a display, a first representation of a virtual object at a first location within the CGR environment, wherein the first location corresponds to a first usage context of a plurality of usage contexts; receiving a request to move a first representation within the CGR environment to a second location different from the first location; and in response to receiving the request: in accordance with a determination that the second location corresponds to a second usage context of the plurality of usage contexts, displaying, via the display, a second representation of the virtual object at the second location based on the second usage context, wherein the second representation is different from the first representation; and in accordance with a determination that the second location corresponds to a third usage context of the plurality of usage contexts, displaying, via the display, a third representation of the virtual object at the second location based on the third usage context, wherein the third representation is different from the first representation and the second representation.

According to some embodiments, a transitory computer readable storage medium stores one or more programs configured for execution by one or more processors of a system with a display, the one or more programs including instructions for: displaying, via a display, a first representation of a virtual object at a first location within the CGR environment, wherein the first location corresponds to a first usage context of a plurality of usage contexts; receiving a request to move a first representation within the CGR environment to a second location different from the first location; and in response to receiving the request: in accordance with a determination that the second location corresponds to a second usage context of the plurality of usage contexts, displaying, via the display, a second representation of the virtual object at the second location based on the second usage context, wherein the second representation is different from the first representation; and in accordance with a determination that the second location corresponds to a third usage context of the plurality of usage contexts, displaying, via the display, a third representation of the virtual object at the second location based on the third usage context, wherein the third representation is different from the first representation and the second representation.

According to some embodiments, a system comprises: a display; means for displaying, via a display, a first representation of a virtual object at a first location within the CGR environment, wherein the first location corresponds to a first usage context of a plurality of usage contexts; means for receiving a request to move a first representation within the CGR environment to a second location different from the first location; and means, responsive to receiving the request, for: in accordance with a determination that the second location corresponds to a second usage context of the plurality of usage contexts, displaying, via the display, a second representation of the virtual object at the second location based on the second usage context, wherein the second representation is different from the first representation; and in accordance with a determination that the second location corresponds to a third usage context of the plurality of usage contexts, displaying, via the display, a third representation of the virtual object at the second location based on the third usage context, wherein the third representation is different from the first representation and the second representation.

According to some embodiments, a method comprises: displaying, via a display of an electronic device, a two-dimensional (2D) representation of a virtual object at a first location within a CGR environment; receiving a request to simultaneously display a three-dimensional (3D) representation and a 2D representation of a virtual object; and in response to the request, simultaneously displaying, via a display of the wearable electronic device, a 2D representation at a first location and a 3D representation at a second location of the CGR environment, wherein the second location is different from the first location.

According to some embodiments, a system comprises: a display; one or more processors; and memory storing one or more programs configured for execution by the one or more processors. The one or more programs include instructions for: displaying, via a display, a 2D representation of a virtual object at a first location of a CGR environment; receiving a request to simultaneously display a 3D representation and a 2D representation of a virtual object; and in response to the request, simultaneously displaying, via the display, a 2D representation at a first location and a 3D representation at a second location of the CGR environment, wherein the second location is different from the first location.

According to some embodiments, a non-transitory computer readable storage medium storing one or more programs configured for execution by one or more processors of a system including a display, the one or more programs including instructions for: displaying, via a display, a 2D representation of a virtual object at a first location of a CGR environment; receiving a request to simultaneously display a 3D representation and a 2D representation of a virtual object; and in response to the request, simultaneously displaying, via the display, a 2D representation at a first location and a 3D representation at a second location of the CGR environment, wherein the second location is different from the first location.

According to some embodiments, a transitory computer-readable storage medium storing one or more programs configured for execution by one or more processors of a system including a display, the one or more programs including instructions for: displaying, via a display, a 2D representation of a virtual object at a first location of a CGR environment; receiving a request to simultaneously display a 3D representation and a 2D representation of a virtual object; and in response to the request, simultaneously displaying, via the display, a 2D representation at a first location and a 3D representation at a second location of the CGR environment, wherein the second location is different from the first location.

According to some embodiments, a system comprises: a display; means for displaying, via a display, a 2D representation of a virtual object at a first location within a CGR environment; means for receiving a request to simultaneously display a 3D representation and a 2D representation of a virtual object; and means for simultaneously displaying, via the display, a 2D representation at a first location and a 3D representation at a second location of the CGR environment in response to the request, wherein the second location is different from the first location.

According to some embodiments, a method comprises: displaying, via a display of a wearable electronic device, a first representation of a virtual object within a CGR environment; in response to the detected movement of the input mechanism: in accordance with a determination that the current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, displaying, via a display of the wearable electronic device, a second representation of the virtual object within the CGR environment, wherein the second representation is different from the first representation; and in accordance with a determination that the current position of the input mechanism is not within the predetermined distance from the first representation of the virtual object, maintaining display of the first representation and not display of the second representation.

According to some embodiments, a system comprises: a display; one or more processors; and memory storing one or more programs configured for execution by the one or more processors. The one or more programs include instructions for: displaying, via a display, a first representation of a virtual object within a CGR environment; and in response to the detected movement of the input mechanism: in accordance with a determination that the current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, displaying, via the display, a second representation of the virtual object within the CGR environment, wherein the second representation is different from the first representation; and in accordance with a determination that the current position of the input mechanism is not within the predetermined distance from the first representation of the virtual object, maintaining display of the first representation and not display of the second representation.

According to some embodiments, a non-transitory computer readable storage medium storing one or more programs configured for execution by one or more processors of a system including a display, the one or more programs including instructions for: displaying, via a display, a first representation of a virtual object within a CGR environment; and in response to the detected movement of the input mechanism: in accordance with a determination that the current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, displaying, via the display, a second representation of the virtual object within the CGR environment, wherein the second representation is different from the first representation; and in accordance with a determination that the current position of the input mechanism is not within the predetermined distance from the first representation of the virtual object, maintaining display of the first representation and not display of the second representation.

According to some embodiments, a transitory computer-readable storage medium storing one or more programs configured for execution by one or more processors of a system including a display, the one or more programs including instructions for: displaying, via a display, a first representation of a virtual object within a CGR environment; and in response to the detected movement of the input mechanism: in accordance with a determination that the current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, displaying, via the display, a second representation of the virtual object within the CGR environment, wherein the second representation is different from the first representation; and in accordance with a determination that the current position of the input mechanism is not within the predetermined distance from the first representation of the virtual object, maintaining display of the first representation and not display of the second representation.

According to some embodiments, a system comprises: a display; means for displaying, via a display, a first representation of a virtual object within a CGR environment; and in response to the detected movement of the input mechanism: means for displaying, via the display, a second representation of the virtual object within the CGR environment in accordance with a determination that the current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, wherein the second representation is different from the first representation; and means for, in accordance with a determination that the current position of the input mechanism is not within the predetermined distance from the first representation of the virtual object, maintaining display of the first representation and not display of the second representation.

Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. Executable instructions for performing these functions are optionally included in a transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

Drawings

In the following description, reference is made to the accompanying drawings which form a part hereof and which illustrate several examples of the present disclosure. It is to be understood that other examples may be utilized and structural and operational changes may be made without departing from the scope of the present disclosure. The use of the same reference symbols in different drawings indicates similar or identical items.

Fig. 1A-1B depict exemplary systems used in various computer-generated reality technologies.

Fig. 2A-2E depict example techniques for controlling representations of virtual objects based on usage context at different locations within a CGR environment, in accordance with aspects of the present disclosure.

Fig. 3A-3C depict another example of a technique for controlling representations of virtual objects based on usage context at different locations within a CGR environment, in accordance with aspects of the present disclosure.

Fig. 4A-4C depict yet another example of a technique for controlling a representation of a virtual object based on usage context at different locations within a CGR environment, in accordance with aspects of the present disclosure.

FIG. 5 is a flow chart illustrating a method performed to implement aspects of the present disclosure.

Fig. 6A-6C depict exemplary techniques for controlling the simultaneous display of representations of virtual objects within a CGR environment, according to aspects of the present disclosure.

FIG. 7 is a flow chart illustrating a method performed to implement aspects of the present disclosure.

Fig. 8A-8B depict an exemplary technique for controlling representations of virtual objects within a CGR environment based on characteristics of an input mechanism, according to aspects of the present disclosure.

Fig. 9A-9B depict another example of a technique for controlling representations of virtual objects within a CGR environment based on characteristics of an input mechanism, in accordance with aspects of the present disclosure.

Fig. 10A-10B depict yet another example of a technique for controlling representations of virtual objects within a CGR environment based on characteristics of an input mechanism, in accordance with aspects of the present disclosure.

11A-11B depict yet another example of a technique for controlling representations of virtual objects within a CGR environment based on characteristics of an input mechanism, in accordance with aspects of the present disclosure.

12A-12B depict another example of a technique for controlling representations of virtual objects within a CGR environment based on characteristics of an input mechanism, according to aspects of the present disclosure.

FIG. 13 is a flow chart illustrating a method performed to implement aspects of the present disclosure.

Detailed Description

Various examples of electronic systems related to various CGR techniques and techniques for using such systems are described.

A physical environment (or real environment) refers to the physical world in which people can sense and/or interact without the aid of an electronic system. A physical environment such as a physical park includes physical objects (or physical objects or real objects) such as physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

In contrast, a CGR environment refers to a fully or partially simulated environment in which people sense and/or interact via an electronic system. In CGR, a subset of the human's physical movements, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one laws of physics. For example, the CGR system may detect head rotations of a person and in response adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in the physical environment. In some cases (e.g., for accessibility reasons), adjustments to the characteristics of virtual objects in the CGR environment may be made in response to representations of physical motion (e.g., voice commands).

A person may utilize any of their senses to sense and/or interact with CGR objects, including vision, hearing, touch, taste, and smell. For example, a person may sense and/or interact with an audio object that creates a (3D) or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sound from a physical environment with or without computer-generated audio. In some CGR environments, a person may sense and/or interact only with audio objects.

Examples of CGR include virtual reality and mixed reality.

A Virtual Reality (VR) environment (virtual environment) refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. The VR environment includes a plurality of virtual objects that a person can sense and/or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person may sense and/or interact with a virtual object in the VR environment through simulation of the presence of the person within the computer-generated environment and/or through simulation of a subset of the physical movements of the person within the computer-generated environment.

In contrast to VR environments that are designed to be based entirely on computer-generated sensory inputs, a Mixed Reality (MR) environment refers to a simulated environment that is designed to introduce sensory inputs from a physical environment or representations thereof in addition to computer-generated sensory inputs (e.g., virtual objects). On a virtual continuum, an MR environment is anything between the full physical environment as one end and the VR environment as the other end, but not both ends.

In some MR environments, computer-generated sensory inputs may be responsive to changes in sensory inputs from the physical environment. Additionally, some electronic systems for presenting MR environments may track position and/or orientation relative to a physical environment to enable virtual objects to interact with real objects (i.e., physical objects or representations thereof from the physical environment). For example, the system may cause motion such that the virtual trees appear to be stationary relative to the physical ground.

Examples of MR include augmented reality and augmented virtual.

An Augmented Reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed over a physical environment or representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person may directly view the physical environment. The system may be configured to present the virtual object on a transparent or translucent display such that the human perceives the virtual object superimposed over the physical environment with the system. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the image or video with the virtual object and presents the combination on the opaque display. A person utilizes the system to indirectly view the physical environment via an image or video of the physical environment and perceive a virtual object superimposed over the physical environment. As used herein, video of the physical environment displayed on the opaque display is referred to as "pass-through video," meaning that the system captures images of the physical environment using one or more image sensors and uses those images when rendering the AR environment on the opaque display. Further alternatively, the system may have a projection system that projects the virtual object into the physical environment, for example as a hologram or on a physical surface, so that a person perceives the virtual object superimposed on the physical environment with the system.

An AR environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., viewpoint) that is different from the perspective captured by the imaging sensor. As another example, a representation of a physical environment may be transformed by graphically modifying (e.g., magnifying) a portion thereof, such that the modified portion may be a representative but not real version of the original captured image. As another example, a representation of a physical environment may be transformed by graphically eliminating portions thereof or blurring portions thereof.

An enhanced virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the face of a person is realistically reproduced from an image taken of a physical person. As another example, the virtual object may take the shape or color of the physical object imaged by the one or more imaging sensors. As another example, the virtual object may employ a shadow that conforms to the positioning of the sun in the physical environment.

There are many different types of electronic systems that enable a person to sense and/or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), display-integrated vehicle windshields, display-integrated windows, displays formed as lenses designed for placement on a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smart phones, tablets, and desktop/laptop computers. The head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, the head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). The head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment, and/or one or more microphones for capturing audio of the physical environment. The head mounted system may have a transparent or translucent display instead of an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eye. The display may utilize digital light projection, OLED, LED, uuled, liquid crystal on silicon, laser scanning light sources, or any combination of these technologies. The medium may be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one example, a transparent or translucent display may be configured to selectively become opaque. Projection-based systems may employ retinal projection techniques that project a graphical image onto a person's retina. The projection system may also be configured to project the virtual object into the physical environment, for example as a hologram or on a physical surface.

Fig. 1A and 1B depict an exemplary system 100 for use in various CGR techniques.

In some examples, as shown in fig. 1A, system 100 includes device 100 a. Device 100a includes various components, such as a processor 102, RF circuitry 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch-sensitive surface 122. These components optionally communicate via a communication bus 150 of device 100 a.

In some examples, elements of system 100 are implemented in a base station device (e.g., a computing device such as a remote server, mobile device, or laptop computer), and other elements of system 100 are implemented in a Head Mounted Display (HMD) device designed to be worn by a user, where the HMD device is in communication with the base station device. In some examples, device 100a is implemented in a base station device or HMD device.

As shown in fig. 1B, in some examples, system 100 includes two (or more) devices in communication, such as through a wired connection or a wireless connection. The first device 100b (e.g., a base station device) includes a processor 102, RF circuitry 104, and a memory 106. These components optionally communicate via a communication bus 150 of the device 100 b. Second device 100c (e.g., HMD) includes various components, such as processor 102, RF circuitry 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch-sensitive surface 122. These components optionally communicate via a communication bus 150 of the device 100 c.

In some examples, system 100 is a mobile device. In some examples, system 100 is an HDM device. In some examples, device 100 is a wearable HUD device.

The system 100 includes a processor 102 and a memory 106. The processor 102 includes one or more general purpose processors, one or more graphics processors, and/or one or more digital signal processors. In some examples, the memory 106 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) that store computer-readable instructions configured to be executed by the processor 102 to perform the techniques described below.

System 100 includes RF circuitry 104. The RF circuitry 104 optionally includes circuitry for communicating with electronic devices, networks (such as the internet, intranets), and/or wireless networks (such as cellular networks and wireless Local Area Networks (LANs)). The RF circuitry 104 may optionally include circuitry for using near field communications and/or short range communications (such as

) A circuit to perform the communication.

The system 100 includes a display 120. In some examples, the display 120 includes a first display (e.g., a left-eye display panel) and a second display (e.g., a right-eye display panel), each for displaying an image to a respective eye of a user. Corresponding images are simultaneously displayed on the first display and the second display. Optionally, the corresponding images comprise representations of the same virtual object and/or the same physical object from different viewpoints, thereby creating a parallax effect that provides a user with a stereoscopic effect of the objects on the display. In some examples, display 120 includes a single display. For each eye of the user, a corresponding image is simultaneously displayed on the first and second regions of the single display. Optionally, the corresponding images comprise representations of the same virtual object and/or the same physical object from different viewpoints, thereby creating a parallax effect that provides the user with a stereoscopic effect of the objects on a single display.

In some examples, the system 100 includes a touch-sensitive surface 122 for receiving user input, such as tap input and swipe input. In some examples, display 120 and touch-sensitive surface 122 form a touch-sensitive display.

The system 100 includes an image sensor 108. The image sensor 108 optionally includes one or more visible light image sensors, such as a Charge Coupled Device (CCD) sensor, and/or a Complementary Metal Oxide Semiconductor (CMOS) sensor operable to obtain images of physical objects from a real environment. The image sensor also optionally includes one or more Infrared (IR) sensors, such as passive IR sensors or active IR sensors, for detecting infrared light from the real environment. For example, an active IR sensor comprises an IR emitter, such as an IR spot emitter, for emitting infrared light into the real environment. The image sensor 108 also optionally includes one or more event cameras configured to capture movement of physical objects in the real environment. The image sensor 108 also optionally includes one or more depth sensors configured to detect the distance of a physical object from the system 100. In some examples, the system 100 uses a CCD sensor, an event camera, and a depth sensor in combination to detect the physical environment surrounding the system 100. In some examples, the image sensor 108 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical objects in the real environment from two different perspectives. In some examples, the system 100 uses the image sensor 108 to receive user input, such as gestures. In some examples, the system 100 uses the image sensor 108 to detect the position and orientation of the system 100 and/or the display 120 in the real environment. For example, the system 100 uses the image sensor 108 to track the position and orientation of the display 120 relative to one or more fixed objects in the real environment.

In some examples, system 100 includes a microphone 112. The system 100 uses the microphone 112 to detect sounds from the user and/or the user's real environment. In some examples, the microphones 112 include a microphone array (including a plurality of microphones) that optionally operates in series to identify ambient noise or to localize sound sources in space in the real environment.

The system 100 includes an orientation sensor 110 for detecting orientation and/or movement of the system 100 and/or the display 120. For example, the system 100 uses the orientation sensor 110 to track changes in the position and/or orientation of the system 100 and/or the display 120, such as with respect to physical objects in a real environment. The orientation sensor 110 optionally includes one or more gyroscopes and/or one or more accelerometers.

Aspects of the present disclosure relate to systems and techniques that provide functionality for controlling the representation of virtual objects within a CGR environment.

In particular, aspects of the present disclosure relate to systems and techniques that provide functionality for controlling representations of virtual objects based on a usage context associated with a location of the virtual object within a CGR environment. The systems and techniques described herein allow representations of virtual objects to be adapted to a particular use context associated with a location within a CGR environment.

Fig. 2A-2E illustrate an example technique for controlling a representation of a virtual object of a CGR environment based on a usage context associated with a location of the virtual object within the CGR environment, in accordance with aspects of the present disclosure. In particular, fig. 2A shows a user 202 and an electronic device 200. In some embodiments, the electronic device 200 may be a wearable electronic device (e.g., HMD). Examples of wearable electronic devices are described herein, such as the electronic device 100a described above with reference to fig. 1A and 1B.

As shown in fig. 2A, a user 202 wears an electronic device 200 configured to make the user 202 perceive a CGR environment 290. As described above, CGR environment 290 may include a physical object or representation thereof and a virtual object, where the virtual object is superimposed on the physical object (e.g., in AR implementations) or the physical object is superimposed on the virtual object (e.g., in AV implementations) to present a coherent CGR environment to user 202. In some embodiments, the CGR environment 290 may be a fully virtual environment (e.g., in a VR implementation), where each object within the CGR environment 290 is a virtual object. Whether fully or partially virtual, in the example shown in fig. 2A, virtual object 210 may be a representation of a rendering application (e.g., an application configured to facilitate multimedia rendering) and may be rendered to user 202 within CGR environment 290.

In an implementation, the virtual object 210 may be located at any location within the CGR environment 290. In the specific example shown in fig. 2A to 2E: CGR environment 290 may include at least locations 220, 222, 224, 226, and 228. It should be understood that these locations are described for illustrative purposes and are not intended to be limiting in any way. That is, any other location within CGR environment 290 may be suitable for the features and functions described herein.

In aspects, location 220 may correspond to a location on a representation of an electronic device within CGR environment 290. For example, location 220 may correspond to a location on display 240 (e.g., a display, a screen, a surface or housing of an electronic device). The display 240 may be a display such as a computer, laptop, tablet, telephone, display, projector display, or the like. The display 240 may be an actual physical device (e.g., a physical object) or may be a virtual representation of a display (e.g., a virtual object) within the CGR environment 290.

Location 222 may correspond to a location on a vertical plane of CGR environment 290 (e.g., a primary vertical plane, a structure such as a vertical plane, a wall, a surface corresponding to a wall-like structure, such as a side of a building, a bedroom wall, a fence, a vertical or secondary vertical plane, etc.). In the particular example shown in fig. 2A, the location 222 corresponds to a location on a wall of the CGR environment 290.

Position 224 and/or position 228 may correspond to a position on a horizontal plane of CGR environment 290 (e.g., a primarily horizontal plane such as a horizontal plane, a desktop, a table, a countertop, a shelf, a floor, an elevated horizontal plane such as a horizontal plane above another horizontal plane within the CGR environment, a non-elevated horizontal plane, etc.). In the particular example shown in FIG. 2A, locations 224 and 228 correspond to locations on desktop 242, which may be physical objects or virtual objects.

Location 226 may correspond to a location on the horizontal plane of CGR environment 290, but of a different type than locations 224 and/or 228. For example, location 226 may be a location on a primarily horizontal plane, such as a structure that is a horizontal plane, a floor, a sidewalk, grass, a lawn, a surface on which one or more people stand, a non-elevated horizontal plane (such as a horizontal plane below another horizontal plane within the CGR), and so forth. In the particular example shown in fig. 2A, the location 226 corresponds to a location on the floor of the CGR environment 290.

As shown in fig. 2A, a virtual object 210 may be displayed (e.g., by the electronic device 200) at a location 220. In some embodiments, a location (e.g., location 220) within CGR environment 290 may be associated with or otherwise correspond to at least one of a plurality of usage contexts. In embodiments, the usage context may relate to the type of surface (e.g., desk, wall, computer screen, floor, etc.) or the type of material of the surface (e.g., sand, grass, concrete, carpet, etc.) on which the virtual object is to be placed and/or may relate to the manner in which the virtual object is to be used (e.g., manipulated, interacted with) or displayed (e.g., rendered) in the CGR environment.

In aspects, the location 220 may be associated with a first usage context. For example, as described above, the location 220 may be a location on the display 240. The display 240 may be a representation of an electronic device. In this case, the first usage context associated with the location 220 may be a type of surface or object that is the location 220 of the electronic device. Thus, in this case, the first usage context can be satisfied when the location 220 is determined to be a location on the representation of the electronic device. In other embodiments, the first usage context associated with the location 220 may be the manner in which the virtual object 210 will be used when in the location 220. For example, it may be determined that at location 220 (which is an electronic device), virtual object 220 will serve as an application for multimedia presentation on display 240. In this case, it may be determined that the virtual object is to be represented as a two-dimensional (2D) window based on the manner in which the virtual object is to be used.

It is noted that, as used herein, a representation of a virtual object may include the content, size, functionality, user interface object, form, shape, design, graphical presentation, etc. of the virtual object within the CGR environment. For example, the virtual object may be represented as a 2D object (e.g., an application icon, an application window, an image, a user interface of an application, etc.). In other examples, the virtual object may be represented as a 3D object within the CGR environment. In some embodiments, the first representation of the virtual object may be a 3D object including specific content, and the second, different representation of the virtual object may be a 3D object including content different from the specific content in the first representation.

In some embodiments, the representation of the virtual object within the CGR environment may include audio features. For example, one representation may include a particular sound, noise, spoken words, etc., and a second representation may include a different sound, noise, spoken words, etc. In some cases, the representations of the virtual objects may also include sound levels, where one representation of the virtual object may include one sound level and a different representation may include a higher or lower sound level.

In accordance with the foregoing, when the virtual object 210 is at least partially located at the location 220, whether by moving or dragging to the location 220 or by displaying at the location 220, the virtual object 210 is displayed as a 2D window (e.g., by the electronic device 200) on the display 240 based on determining that the location 220 is associated with a usage context that is satisfied by determining that the location 220 is on the display 240, the display 240 being an electronic device.

In some embodiments, the virtual object 210 may be configured such that the user 202 may interact with the virtual object 210. As described above, interaction with the virtual object 220 may be via an input sensor configured to detect user input interacting with the virtual object of the CGR environment 290. In some embodiments, the input sensor may include a mouse, stylus, touch-sensitive surface, image sensor (e.g., to perform hand tracking), etc., which may be configured to allow the user 202 to grab, move, drag, click, select, and/or otherwise select the virtual object 210. Thus, in an implementation, a request to move a virtual object 210 to a location within the CGR environment 290 may be received.

In the example shown in fig. 2A, the request to move the virtual object 210 from the location 220 to another location within the CGR environment 290 may include the user 202 grabbing or otherwise selecting the virtual object 210 for movement from the location 220 and may cause the virtual object 210 to leave the location 220. In some embodiments, once the virtual object 210 is removed from the location (e.g., location 220), the current representation of the virtual object 210 may change. For example, once the virtual object 210 is removed from the location 220, the current representation of the virtual object 210, which is a 2D window of the multimedia presentation application, may change to another representation. In some implementations, the current representation of the virtual object 210 can change to some transition representation, which may not be associated with a particular usage context, but rather may be a default representation indicating that the virtual object 210 is transitioning from one location to another. In other implementations, when the virtual object 210 is removed from the location, the current representation of the virtual object 210 may not change, but the current representation of the virtual object 210 may remain unchanged until the virtual object is positioned at another location determined to be associated with a usage context that can determine to display a different representation of the virtual object 210. In this case, the current representation of the virtual object 210 may be maintained during the transition of the virtual object 210 from the current location to the new location.

FIG. 2B illustrates an example of a virtual object 210 displayed (e.g., by the electronic device 200) at a location 224. In this example, in response to a request to move a virtual object to the location 224, at least one usage context corresponding to the location 224 can be determined. For example, the location 224 may correspond to a location on the desktop 242. In this case, it may be determined that the location 224 is associated with a usage context that is satisfied by the location type (e.g., surface type, air) of the location 224, the location 224 being a location on the desktop 242 (e.g., a location on a horizontal plane). In an alternative or additional embodiment, the location 224 on the desktop 242 may be determined as, for example, the location at which the user 202 can annotate with respect to the multimedia presentation using the virtual object 210. In either case, whether because location 224 is a location on the desktop or because location 224 is a location where a virtual object may be used to annotate a multimedia presentation, virtual object 210 may be represented as a 3D object (e.g., a notepad, notebook, book, or any other 3D representation) that is configured to facilitate a user annotating and/or making notepads on the multimedia presentation.

Although not shown, the virtual object 210 may move from location 224 on the desktop 242 to location 228, which is also on the desktop 242. In an embodiment, the representation of the virtual object 210 (e.g., a 3D virtual notepad) may remain the same in location 228 as in location 224, as both locations may be associated with the same usage context. Alternatively, although locations 224 and 228 are both on desktop 242 (e.g., the same type of surface), the representation of the virtual object while on location 228 may be different than the representation while on location 224. For example, the representation of the virtual object when at location 228 may have a different size (e.g., smaller or larger) or a different orientation than the representation when at location 224, because location 228 may be determined to be unable to accommodate the size and/or orientation of the representation of virtual object 210 when at location 224. In some embodiments, different locations within the same type of surface (e.g., different locations on the desktop 242, on the wall 222, etc.) may be configured for different usage contexts. For example, a particular location on the desktop 242 may be configured with a usage context in which the representation of the virtual object 210 may be in a particular language, and another location on the desktop 242 may be configured with a usage context in which the representation of the virtual object 210 may be in a different language.

FIG. 2C shows an example of a virtual object 210 displayed at a location 222. For example, a request to move the virtual object 210 to the location 222 may be received. The request may include a request to move the virtual object 210 from any other location within the CGR environment 290 (e.g., location 220, location 224, location 226, etc.). In response to the request, the virtual object 210 may be moved to the location 222, and a representation of the virtual object 210 to be displayed at the location 222 may be determined. In this example, in response to a request to move a virtual object to the location 222, at least one usage context corresponding to the location 222 can be determined. For example, the location 222 may correspond to a location on a vertical plane (e.g., a wall) of the CGR environment 290. In this case, it may be determined that location 222 is associated with a usage context that is satisfied by the location type (e.g., surface type) of location 222, location 222 being a location on a wall of CGR environment 290. In an alternative or additional embodiment, the location 222 on the wall of the CGR environment 290 may be determined as the location at which the virtual object 210 may be used to render the multimedia presentation. In either case, whether because the location 222 is a location on a wall or because the location 222 is a location at which the virtual object is available to render the multimedia presentation, the virtual object 210 may be represented (e.g., displayed by the electronic device 200) as a large window object configured to facilitate rendering of the multimedia presentation. For example, a large window object may be a 3D representation of a 2D window or a large monitor, which is displayed as being fixed against a wall. In some implementations, the size of the large window object against the wall may be determined based on the distance of the wall displaying the large window object against relative to the position of the user 202 within the CGR environment 290.

In some embodiments, the content (e.g., information and/or information arrangement) of the representation of the virtual object 210 at the location 222 may be different than the content in the representations of the virtual object 210 at other locations. For example, when at location 224, the 3D notepad used as a representation of virtual object 210 may include information arranged in a particular arrangement within the 3D notepad. Although at location 222, the large window display against the wall, which serves as a representation of the virtual object 210, may include different information that may be arranged in different arrangements within the large window display.

FIG. 2D illustrates an example of a virtual object 210 displayed (e.g., by the electronic device 200) at a location 226. For example, a request to move the virtual object 210 to the location 226 may be received. The request may include a request to move the virtual object 210 from any other location within the CGR environment 290 (e.g., location 220, location 222, location 224, location 228, etc.). In response to the request, the virtual object 210 may be moved to the location 226 and a representation of the virtual object 210 to be displayed (e.g., by the electronic device 200) at the location 226 may be determined. In this example, in response to a request to move a virtual object to the location 226, at least one usage context corresponding to the location 226 can be determined. For example, the location 226 may correspond to a location on a horizontal plane (e.g., floor) of the CGR environment 290. Note that in this example, the location 226 corresponds to a different type of location on the horizontal plane than the horizontal plane corresponding to the location 224, which is a location on the desktop 242. In this case, it may be determined that the location 226 is associated with a usage context that is satisfied by the location type (e.g., surface type) of the location 226, the location 226 being a location on the floor of the CGR environment 290. In an alternative or additional embodiment, the location 226 on the floor of the CGR environment 290 may be determined as a location where the virtual object 210 may be used to render the multimedia presentation at least partially immersive (e.g., from a first person view mode). In either case, whether because location 226 is a location on a wall or because location 226 is a location where a virtual object may be used to at least partially immerse rendering a multimedia presentation, virtual object 210 may be represented as a 3D pod placed on or near location 226 that is configured to facilitate user 202 rendering of the multimedia presentation from the pod. In some embodiments, the representation of the virtual object 210 at the location 226 may include content 212 (e.g., notepad, annotation, presentation content, etc.) related to the multimedia presentation, and may be presented on top of a podium where the user 202 may perceive the content 212.

Fig. 2E illustrates an example of a virtual object 210 displayed in a fully immersive mode (e.g., by electronic device 200). In some embodiments, the particular location may be associated with a fully immersive usage context. For example, a location (such as location 226 on the floor of CGR environment 290) may be associated with a usage context in which the presentation is to be presented as a fully immersive experience. In response to a request to move virtual object 210 to location 226, the virtual object may be moved to location 226 and a fully immersive representation of virtual object 210 may be displayed. In this case, displaying the virtual object 210 as a fully immersive representation may include displaying the entire CGR environment 290 as a virtual hall configured to present multimedia applications.

In some embodiments, a representation of the virtual object 210 associated with a particular usage context may be displayed without having to move the virtual object to a particular location. For example, referring back to FIG. 2A, in some embodiments, the affordance 214 may be presented within a CGR environment 290. The affordances 214 may be virtual objects (e.g., buttons, affordances, user interface elements, interactive elements, etc.) configured to allow a user (e.g., user 202) to interact with. The affordance 214 can correspond to at least one usage context. In some embodiments, affordances 214 may also be associated with virtual objects 214 (e.g., associated with a particular application of virtual object 214, such as a multimedia presentation, calculator, weather, etc.). When the user 202 selects the affordance 214 of the virtual object 210, the usage context corresponding to the affordance 214 may be considered satisfied, and the associated representation (e.g., the representation of the virtual object 210 associated with the usage context) may be caused to be displayed. For example, where the affordance 214 corresponds to a usage context associated with a location 224 (e.g., a desktop), as shown in FIG. 2B, a representation of a virtual object 210 (e.g., a 3D notepad) may be displayed by the electronic device 200. In some cases, the representation of the virtual object 210 may be displayed at a location associated with the usage context (e.g., without having to move the virtual object 210 from its current location to a location corresponding to the usage context associated with the affordance 214), or may be displayed at any location where the virtual object 210 is currently being displayed. In some embodiments, displaying the representation of the virtual object 210 at the location associated with the usage context corresponding to the affordance 214 may include moving the virtual object 210 from its current location to the location associated with the usage context. In these cases, the movement of the virtual object 210 to the location associated with the usage context may be displayed in an animated manner.

As another example, where affordance 214 corresponds to a usage context associated with a fully immersive usage context, as shown in fig. 2E, a representation of virtual object 210 as a fully immersive experience may be displayed by electronic device 200 in response to user 202 selecting affordance 214.

In some embodiments, the affordance 214 may include a plurality of affordances, each of which corresponds to a particular use context. In these embodiments, each of the plurality of affordances may be a selectable affordance that, when selected, may cause the corresponding usage context to be deemed satisfied, and may cause an associated representation (e.g., a representation of the virtual object 210 associated with the satisfied usage context) to be displayed in accordance with the foregoing.

It is noted that while the present disclosure describes embodiments in which virtual objects are displayed at a single location within a CGR environment at a time, this is done for illustrative purposes and should not be construed as being limiting in any way. Indeed, in some implementations, separate and in some cases different representations of the same virtual object may be displayed at more than one location within the CGR environment at the same time. In embodiments, the separate representations at different locations may all be different (e.g., may include different information or may have different shapes and/or forms, as described above), or some representations at different locations may be the same while other representations at other locations may be different. In some embodiments, a change to the configuration of a virtual object (e.g., a change to an application associated with the virtual object) may trigger a change to all representations at all locations, or may trigger a change to some representations at some locations instead of all representations at all locations. In some cases, a change in a representation at one location within the CGR environment (e.g., a change caused in response to a user interaction and/or by a change in an associated application) may trigger at least one change in at least one representation of a virtual object at another location within the CGR environment.

Fig. 3A-3C illustrate examples of functionality for controlling representations of virtual objects based on a usage context associated with a location within a CGR environment, in accordance with aspects of the present disclosure. In particular, fig. 3A shows a user 202 wearing an electronic device 200 that may be configured to allow the user 202 to view a CGR environment 290. In some implementations, the electronic device 200 may be similar to the electronic device 100a described above with reference to fig. 1A and 1B.

CGR environment 290 includes a display 340, which may be a physical display or a virtual representation of a display. In any case, the representation of the virtual object 310 may be displayed at a location 320 (e.g., by the electronic device 200), which is a location on the display 340. In the example shown in FIG. 3A, the virtual object 310 may be a calculator application. In this case, it may be determined that the location 320 corresponds to at least one usage context (e.g., a type of location, surface, material, etc., and/or a type of usage of the virtual object at the location). For example, location 320 may be determined as a location on an electronic device (e.g., a physical device or a computer-generated simulation of a physical device) of CGR environment 290. In this case, it may be determined that the location 320 is associated with a usage context that is satisfied by the location type (e.g., surface type) of the location 320, the location 320 being a location on the electronic device. Based on determining that the location 320 is a location on the electronic device, the virtual object 310 may be displayed as a 2D window or desktop applet for a calculator application on the display 340 (e.g., via the electronic device 200). Thus, it should be understood that the representation of the virtual object 310 at the location 320 is based on the usage context corresponding to the location 320.

FIG. 3B illustrates the user 202 interacting with the virtual object 310 at the location 320. The interaction of the user 202 with the virtual object 310 at the location 320 may include a request to move the virtual object 310 to another location (e.g., location 324). FIG. 3C illustrates that in response to a request to move the virtual object 310, the virtual object 310 has been moved to a location 324. In this example, at least one usage context associated with the location 324 can be determined. For example, location 324 is a location on desktop 342. In this case, it may be determined that the location 324 is associated with a usage context that is satisfied by the location type (e.g., surface type) of the location 324, the location 324 being a location on the desktop 342 (e.g., a location on a horizontal plane). In an alternative or additional embodiment, the location 324 on the desktop 342 may be determined as a location at which the virtual object 310 (e.g., a calculator application) may be used, for example, by the user 202, to manipulate the calculator application in a manner that causes input into the calculator application as in a real-world physical calculator, for example, by using the hand of the user 202 or a virtual representation thereof. In either case, whether because location 324 is a location on the desktop or because location 324 is a location in which a virtual object is available to enter into the calculator using the user's hand or a virtual representation thereof, virtual object 310 may be represented as a 3D object (e.g., a 3D representation of a physical calculator) that is configured to facilitate the user entering an item into the calculator application.

Fig. 4A-4C illustrate another example of a representation of a virtual object of a CGR environment based on a usage context associated with a location of the virtual object within the CGR environment, in accordance with aspects of the present disclosure. In particular, fig. 4A shows a user 202 wearing an electronic device 200 configured to allow the user 202 to view a CGR environment 290. As described above, in some embodiments, the electronic device 200 may be similar to the electronic device 100a described above with reference to fig. 1A and 1B.

CGR environment 290 includes display 440. As described above, display 440 may be a physical display or a virtual representation of a display. The representation of the virtual object 410 may be displayed by the electronic device 200 at a location 420, which is a location on the display 440. In the example shown in FIG. 4A, virtual object 410 may be an application for rendering an interactive and/or animated robot. It should be understood that the description of the animated robot herein is for illustrative purposes only and should not be construed as limiting in any way. Indeed, the techniques herein are applicable to any application within a CGR environment that may be represented as a virtual object. In this example, location 420 may be determined as a location on a representation of the electronic device (e.g., a representation of a display of a physical computer). Based on determining that location 420 is a location on a representation of the electronic device, virtual object 410 may be displayed (e.g., by electronic device 200) as a 2D window or desktop applet on display 440.

FIG. 4B shows that virtual object 410 has been moved to location 424. In aspects, the virtual object 410 may move to the location 424 in response to a user's request (e.g., the user interacting with the virtual object 410 to drag or otherwise cause the virtual object 410 to move to the location 424). In this example, at least one usage context associated with the location 424 can be determined. For example, location 424 is a location on desktop 442. In this case, it may be determined that the location 424 is associated with a usage context that is satisfied by the location type (e.g., surface type) of the location 424, which is a location on the desktop 442 (e.g., a location on a horizontal plane). Based on the usage context corresponding to the location 424, the virtual object 410 may be represented (e.g., displayed by the electronic device 200) as a 3D object (e.g., a 3D representation of an animated robot). In an embodiment, the representation of the virtual object 410 at location 424 may include different functionality than the representation of the virtual object at location 420. For example, an animated 3D robot on the desktop 442 may be configured to move around the desktop 442 in more than one axis. Additionally or alternatively, the animated 3D robot on the desktop 442 can rotate about its own axis. Additionally or alternatively, the animated 3D robot on the desktop 442 may be configured to have a larger size than when in position 420.

FIG. 4C shows that virtual object 410 has been moved to location 426. In aspects, the virtual object 410 may move to the location 426 in response to a user's request (e.g., the user interacting with the virtual object 410 to drag or otherwise cause the virtual object 410 to move to the location 426). In this example, at least one usage context associated with location 426 may be determined. For example, location 426 is a location on the floor of CGR environment 290. Note that in this example, location 426 corresponds to a different type of location on the horizontal plane than the horizontal plane corresponding to location 424, which is a location on desktop 442. In this case, it may be determined that location 426 is associated with a usage context that is satisfied by the location type (e.g., surface type) of location 426, location 426 being a location on the floor of CGR environment 290. Based on the usage context corresponding to the location 424, the virtual object 410 may be represented (e.g., displayed by the electronic device 200) as a 3D object (e.g., a 3D representation of an animated robot) on the floor of the CGR environment 290. In an embodiment, the representation of the virtual object at location 426 may be different than the representation of the virtual object at location 424. For example, an animated 3D robot on the floor of the CGR environment 290 may be larger than the animated 3D robot at location 424 on the desktop 442. Further, the animated 3D robot on the floor of the CGR environment 290 may be configured to move at a faster rate than the animated 3D robot at location 424 on the desktop 442.

In some embodiments, some locations within CGR environment 290 may not be associated with the usage context of a particular application or may be prohibited locations relative to virtual objects associated with a particular application. For example, the location 422 may be a location on a vertical plane (e.g., a wall) of the CGR environment 290. In this example, location 422 may not have an associated usage context. If user 202 attempts to move virtual object 210 to location 422, movement may not be allowed because, for example, the 3D robot may not be able to navigate on a vertical surface. Alternatively, a default representation of the virtual object (e.g., a 2D image or a 2D application window) may be displayed.

Fig. 5 is a flow diagram illustrating a method 500 for controlling a representation of a virtual object of a CGR environment based on a usage context associated with a location of the virtual object within the CGR environment. In some embodiments, the method 500 may be performed by the system 100 or a portion of the system 100. In some embodiments, method 500 may be performed by one or more external systems and/or devices. In some embodiments, method 500 may be performed by system 100 (or a portion of system 100) in conjunction with one or more external systems and/or devices.

At block 502, the system displays, via a display of an electronic device (e.g., a wearable electronic device, an HMD device, etc.), a first representation of a virtual object at a first location within a CGR environment. For example, the first representation of the virtual object may be displayed via a first display (e.g., a left eye display panel) or a second display (e.g., a second eye display panel) of the electronic device.

In an embodiment, the first location may correspond to a first usage context of a plurality of usage contexts. In an embodiment, the plurality of usage contexts may include usage contexts related to a type of surface (e.g., desk, wall, computer screen, floor, etc.) and/or a type of material (e.g., sand, grass, concrete, carpet, etc.) on which the virtual object is to be placed, and/or usage contexts corresponding to how the virtual object is to be used (e.g., manipulated, interacted with) or displayed (e.g., rendered) in the first location of the CGR environment. In some embodiments, the system may be part of an electronic device, or the electronic device may be part of a system.

In some embodiments, when the representation of the virtual object is displayed at the first location, the representation of the virtual object may be displayed on a first type of surface (e.g., a desktop, a wall, a computer screen, a floor, etc.), and the representation of the virtual object may be displayed based on the first location (e.g., the type of surface corresponding to the first location). In some embodiments, one or more of the plurality of usage contexts may be predefined. For example, one or more of a plurality of usage contexts can be predefined based on a particular application corresponding to the virtual object. In some embodiments, the first application may have a first number of predefined usage contexts and the second application may have a second number of predefined usage contexts different from the first number of predefined usage contexts. In some embodiments, the second application may have a different context of use than the first application, and vice versa.

At block 504, the system receives a request to move a first representation within the CGR environment to a second location different from the first location. In some embodiments, the request may be received or detected by the system based on detecting movement of the first representation from the first location to the second location. In some embodiments, one or more user inputs may be detected, and in response to detecting these user inputs, the system may receive a request to move the representation to the second location. In some embodiments, the request to move the first representation from the first location to the second location may be received based on one or more determinations of the external application, wherein the request to move the first representation from the first location to the second location is received based on the one or more determinations.

At block 506, in response to receiving the request and in accordance with a determination that the second location corresponds to a second usage context of the plurality of usage contexts (e.g., the second usage context is different from the first usage context), the system displays, via a display of the electronic device, a second representation of the virtual object based on the second usage context and/or based on one or more applications associated with the virtual object at, near, and/or on a surface corresponding to the second location. In an embodiment, the second representation may be different from the first representation. For example, the second representation may have a different size, shape, user interface object, function, audio characteristic, surface material, etc., and/or may be configured to have a different and/or additional operation than the first representation.

In some embodiments, a second usage context of the plurality of usage contexts may include a usage context that is satisfied when the second location is determined to correspond to a location (e.g., a display, a screen, a surface or a housing of the electronic device) on the electronic device (e.g., a computer, a laptop, a tablet, a phone, a display, a projector display). In some embodiments, in accordance with a determination that the second location corresponds to a second usage context of the plurality of usage contexts, the system displays, within the CGR environment, a 2D representation of the virtual object on the electronic device as part of displaying a second representation of the virtual object based on the second usage context. In some embodiments, the second representation of the virtual object may be a 2D representation on the electronic device. In some implementations, the second representation can be moved (e.g., dragged away from a display of the electronic device) to a location in the virtual environment that corresponds to a physical surface in the physical environment. In some embodiments, the 2D application may be manipulated as a 3D application on the electronic device. In some embodiments, a second usage context of the plurality of usage contexts may include a usage context that is satisfied when the second location is determined to correspond to a location on an electronic device (e.g., a computer, a laptop, a tablet, a phone, a display, a projector display). In these embodiments, in accordance with a determination that the second location corresponds to a second usage context of the plurality of usage contexts, displaying a second representation of the virtual object based on the second usage context may include displaying a 3D representation on the electronic device within the CGR environment. In some implementations, the representation may vary depending on the type of electronic device (e.g., display (e.g., monitor), tablet, personal computer, laptop).

In some embodiments, a second use context of the plurality of use contexts can include a use context that is satisfied when it is determined that the second location corresponds to a location on a vertical plane (e.g., a wall, a surface corresponding to a wall-like structure, a building side, a bedroom wall, a fence, etc.). In some embodiments, in accordance with a determination that the second location corresponds to a second usage context of the plurality of usage contexts, as part of displaying a second representation of the virtual object based on the second usage context, the system displays the 2D representation on a vertical plane (e.g., on a wall) within the CGR environment. In some embodiments, the second representation of the virtual object may be a 2D representation on the electronic device. In some implementations, a 2D representation displayed on a vertical plane (e.g., on a wall) within the CGR environment is compared to a 2D representation displayed on an electronic device: may be larger, may have more visual content, and may include one or more additional (or different) user interface objects. In some embodiments, the representation may vary according to the type of vertical plane (e.g., side of a building, bedroom wall, fence) and/or one or more characteristics of the vertical plane (e.g., virtual or physical), such as size, shape (e.g., circular, rectangular), material (e.g., brick, wood, metal), texture (e.g., rough, worn), color, opacity, and the like.

In some implementations, the size of the second representation may be based on a distance between a display of the electronic device and a vertical plane within the CGR environment. In some embodiments, the 2D representation may be smaller when the vertical plane is closer to the display of the electronic device and larger when the vertical plane is further from the display of the electronic device. In some embodiments, after the initial display of the 2D representation, the size of the 2D representation may be maintained as the user moves farther away or closer to the 2D representation. In some embodiments, after the initial display of the 2D representation, the size of the 2D representation may be changed as the user moves farther away or closer to the 2D representation. In some embodiments, the size of the 2D representation may be based on whether the distance is in a certain category (e.g., category of distances (e.g., distance, near, average distance), where the distance of each category corresponds to a different size representation (e.g., super large, small, medium).

In some embodiments, a second use context of the plurality of use contexts comprises a use context that is satisfied when the second location is determined to correspond to a location on a horizontal plane within the CGR environment (e.g., a desktop, a table, a countertop, a shelf, a floor, an elevated horizontal plane, a horizontal plane above another horizontal plane, a non-elevated horizontal plane, etc.). In some embodiments, in accordance with a determination that the second location corresponds to a second usage context of the plurality of usage contexts, the system may display the 3D representation on a horizontal plane within the CGR environment as part of displaying a second representation of the virtual object based on the second usage context. In some embodiments, the second representation of the virtual object may be a 3D representation on a horizontal plane. In some embodiments, the representation may vary according to the type of horizontal plane (e.g., desktop, table, countertop, shelf) and/or one or more features of the horizontal plane (e.g., virtual or physical), such as size, shape (e.g., circular, rectangular), material (e.g., brick, wood, metal), texture (e.g., rough, worn), color, opacity, and the like.

In some embodiments, the 3D representation may be a representation of a first size in accordance with a determination that the horizontal plane is a horizontal plane of a first type. In some embodiments, the 3D representation may be a representation of a second size different from (e.g., larger than) the first size in accordance with a determination that the horizontal plane is a horizontal plane of the second type. In embodiments, the first type of horizontal plane and the second type of horizontal plane may be selected from horizontal planes that may include the following types: a mainly horizontal plane, a structure of a horizontal plane, a floor, a crosswalk, grass, lawn, a surface on which one or more persons stand, a non-elevated horizontal plane, a horizontal plane below another horizontal plane within the CGR environment, etc.

In some embodiments, a 3D representation displayed on a horizontal plane of a first type (e.g., a desktop, a table, a tabletop, a shelf) within a CGR environment may be larger, may have more visual content, may include one or more additional (or different) user interface objects than a 3D representation displayed on a horizontal plane of a second type (e.g., a floor, a sidewalk, a grass, a lawn, a surface on which one or more people stand).

In some embodiments, the second of the plurality of usage contexts may comprise a usage context that is satisfied when the maximize view criteria is satisfied. For example, the maximized view criteria may be satisfied when a user interface element (e.g., a button, an affordance, and/or any other interactive element) is selected based on a room in which the application may be running, based on a second location (e.g., a location where a virtual object is moved to or dropped), a location on a body part of a user of the device that corresponds to the maximized criteria being satisfied (e.g., a location on a hand), a gesture, and/or the like. In these embodiments, as part of displaying the second representation of the virtual object based on the second usage context, the system displays a plurality of representations of the virtual object on a plurality of planes within the CGR environment. In some embodiments, displaying multiple representations of virtual objects on multiple planes within a CGR environment may include changing one or more aspects of the physical environment and/or the CGR environment to create a fully or partially immersive experience. For example, a room (e.g., physical or virtual) within the CGR environment may be converted to a virtual hall when the application is a rendering application; transitioning to a virtual stadium (e.g., football stadium) when the application is a sports viewing application (e.g., fantasy sports application, live sports application); is converted into a virtual store while shopping on a shopping application or the like. In some embodiments, the maximized view may be displayed via a companion application (e.g., fantasy sports application, live sports application, shopping application, presentation application, etc.). In some embodiments, the companion application may correspond to a virtual object and/or may be a companion application of an application corresponding to a virtual object.

In some embodiments, a selectable virtual object corresponding to the maximized view representational (e.g., a selectable virtual object currently displayed with a representation of the virtual object, such as the first representation) may be displayed. In some embodiments, the maximized view criteria may include a criterion that is met when a selectable virtual object corresponding to the maximized view representable (e.g., a tap or swipe on the virtual object) is selected.

In some embodiments, it may be determined that the second location corresponds to a second usage context of the plurality of usage contexts. In some embodiments, the first representation may include first visual content (e.g., representations of text, buttons, audio/video, user interface elements, etc.). In some implementations, the second representation may not include the first visual content.

In some embodiments, it may be determined that the second location corresponds to a second usage context of the plurality of usage contexts. In some embodiments, the first representation may include third visual content displayed in a third size. In some embodiments, the second representation may include third visual content displayed in a fourth size that is different from the third size (e.g., larger or smaller representations of text, buttons, audio/video, user interface elements, etc.).

In some embodiments, it may be determined that the second location corresponds to a second usage context of the plurality of usage contexts. In some embodiments, the first representation may include a first selectable object (e.g., one or more selectable user interface elements). In some embodiments, the second representation may not include the first selectable object.

In some embodiments, it may be determined that the second location corresponds to a second usage context of the plurality of usage contexts. In some embodiments, the first representation is a fourth dimension. In some embodiments, the second representation is a fifth dimension that is different (e.g., larger or smaller) than the fourth dimension.

In some embodiments, as part of displaying the second representation of the virtual object based on the second usage context, the system may transition the display of the first representation to the display of the second representation when the first representation is at a predetermined distance from the second location (e.g., a distance near the second location when the first representation reaches the second location). In some embodiments, the display of the first representation is maintained as the first representation is moved from the first position until the first representation reaches the second position or is within a certain distance from the second position.

In some embodiments, in accordance with a determination that the second location corresponds to a fourth usage context of the plurality of usage contexts, wherein the fourth usage context is satisfied when the second location corresponds to a prohibited location (e.g., a location prohibited by an application to which the virtual object corresponds and/or one or more other applications and/or the system), the system forgoes displaying the representation of the virtual object within the CGR environment based on the fourth usage context. In some embodiments, the first representation can continue to remain displayed even when the second location corresponds to a location that satisfies the usage context (e.g., the second usage context) but prohibits display of a different usage context because representations that differ from the first representation are prohibited from being displayed and/or representations that correspond to the usage context (e.g., the second usage context) that will satisfy but prohibit display of a different representation are prohibited from being displayed.

In some embodiments, in accordance with a determination that the second location corresponds to a fourth usage context of the plurality of usage contexts, the system may display (e.g., display a message or symbol to note that a representation corresponding to the fourth usage context cannot be displayed or inhibited) within the CGR environment an indication that the second location is an inhibited location (e.g., a location that an application and/or one or more other applications and/or systems corresponding to the virtual object are inhibited).

At block 508, in response to receiving the request and in accordance with a determination that the second location corresponds to a third usage context of the plurality of usage contexts (e.g., the third usage context is different from the first and second usage contexts), the system may display, via a display of the electronic device, a third representation of the virtual object based on the third usage context (and/or based on one or more applications associated with the virtual object) at the second location (e.g., on a surface corresponding to the second location), wherein the third representation is different from the first representation and the second representation.

Aspects of the present disclosure relate to systems and techniques that provide functionality for controlling the simultaneous display of representations of virtual objects within a CGR environment. In an embodiment, controlling the simultaneous display of representations of virtual objects may include displaying a first representation on a first surface (e.g., a physical or virtual surface) of the CGR environment and displaying a second representation on a second surface of the CGR environment different from the first surface. In an embodiment, a control may be provided for requesting that the second representation of the virtual object be displayed concurrently with the first representation of the virtual object.

Fig. 6A-6C illustrate an exemplary technique for controlling the simultaneous display of representations of virtual objects within a CGR environment, according to aspects of the present disclosure. In particular, fig. 6A shows a user 202 wearing an electronic device 200 configured to allow the user 202 to view a CGR environment 290. As described above, in some embodiments, the electronic device 200 may be similar to the electronic device 100a described above with reference to fig. 1A and 1B.

As shown in FIG. 6A, CGR environment 290 includes a display 640. As described above, the display 640 may be a physical display or a virtual representation of a display. A first representation 620 of the virtual object 610 may be displayed by the electronic device 200 at a first surface of the CGR environment. For example, the first representation 620 of the virtual object 610 may be displayed on the display 640. In the example shown in fig. 6A, the first representation 620 is a 2D representation displayed on the display 640. In an embodiment, the first representation 620 may be displayed on any surface (e.g., physical or virtual) within the CGR environment 290. The first representation 620 may include various graphical elements associated with the virtual object. For example, as shown, virtual object 610 is associated with a calculator application and includes various graphical elements associated with the calculator application. It should be understood that the use of a calculator application to illustrate the virtual object 610 is for illustrative purposes and is not intended to be limiting in any way. Accordingly, virtual object 610 may be associated with any other type of application (e.g., calendar, multimedia application, presentation, etc.).

In some embodiments, a control may be provided for requesting display of the second representation of the virtual object 610. A user (e.g., user 202) may request simultaneous display and the request may be received by device 200. The request to display the second representation of the virtual object 610 may include a request to display the second representation of the virtual object 610 concurrently with the first representation 620. The controls for requesting simultaneous display may include any technique for providing a selection (e.g., by user 202). For example, in some embodiments, the control for requesting simultaneous display may include an affordance 611 presented within CGR environment 290. In some embodiments, the affordance 611 may be disposed within the first representation 620 or may be disposed outside of the first representation 620. In some embodiments, affordances 611 may be virtual objects (e.g., buttons, affordances, user interface elements, interactive elements, etc.) displayed within CGR environment 290 and configured to allow a user (e.g., user 202) to interact with. In other embodiments, affordance 611 may be a graphical element (e.g., rather than a virtual element) displayed on a physical display.

In an embodiment, the control for requesting simultaneous display may comprise a gesture, which may include moving or dragging virtual object 610 off of display 640. For example, the user 202 may perform a gesture (e.g., using a appendage, input sensor, etc.), wherein the virtual object 610 may be dragged out of or moved out of the display 640. The drag gesture may be determined as a request to display the second representation of the virtual object 610 concurrently with the first representation 620. In some embodiments, user 202 may drag virtual object 610 out of display 640 and may continue to drag the virtual object to a location within CGR environment 290 where a second representation of virtual object 610 is to be displayed.

In some implementations, in response to receiving a request to simultaneously display representations of virtual object 610, a second representation of virtual object 610 may be displayed within CGR environment 290. In an embodiment, a request to simultaneously display representations of virtual object 610 may result in an animation in which a second representation of virtual object 610 appears (e.g., pops up) of first representation 620. This is shown in fig. 6B.

FIG. 6C illustrates a second representation 621 of a virtual object 610 displayed within the CGR environment 290 in response to receiving a request to simultaneously display representations of the virtual object 610. In an embodiment, the second representation 621 may be displayed on any surface (e.g., physical or virtual) within the CGR environment 290. In an embodiment, the second representation 621 may be separate or different from the first representation 620. For example, as shown in FIG. 6C, the first representation 620 may be a 2D representation of the virtual object 610 displayed on the display 640, and the second representation 621 may be a 3D representation of the virtual object 610 displayed outside the display 640 on a second and different surface of the CGR environment 290. In some embodiments, a 2D representation of an object (e.g., an object within a particular application or a particular type of application (e.g., a calculator application or a key fob presentation application, a media or entertainment application, a productivity application) may be displayed concurrently with a 3D representation of the object.

In some embodiments, the first representation 620 and the second representation 621, although associated with the same virtual object, may provide different or the same functionality. For example, the first representation 620 and the second representation 621 may share a common set of UI elements. In this example, the first representation 620 may be a 2D representation of an application (e.g., a calculator) that includes a particular set of UI elements for user interaction with the application. The second representation 621 may be a 3D representation of an application (e.g., a calculator) that includes the same set of specific UI elements for user interaction as the first representation 620. However, in some embodiments, the first representation 620 and the second representation 621 may have different sets of UI elements. For example, the first representation 620 may include a particular set of UI elements, while the second representation 621 may include a different set of UI elements. In an embodiment, one set of UI elements in the different sets of UI elements may include at least one UI element not included in the other set of UI elements. In other embodiments, different groups of UI elements do not have UI elements in common. It will be appreciated that displaying representations of virtual objects simultaneously by providing different functionality provides an improved system in that the system may be configured to adjust representations (e.g., 2D representations or 3D representations) of virtual objects having functionality dependent on the type of representation.

In some embodiments, one representation of a virtual object may be a virtual representation, while another representation of the virtual object may not be a virtual representation. For example, the display 640 may be a physical display and the first representation 620 may be a graphical representation of the virtual object 610 displayed on the physical display 640. In this case, the first representation 620 may not be a virtual representation because the first representation 620 is actually displayed in the real world on a physical display and perceived by the user 202 via a transparent or semi-transparent display of the electronic device 200. In this example, the second representation 621 may be a virtual representation of the virtual object 610, because the second representation 621 is not actually displayed in the real world on the physical display, but is displayed on the display of the electronic device 200 and superimposed on the real world physical display. In this way, a user may be provided with the ability to request display of a 3D representation of a virtual object by interacting with a control provided in a 2D representation of the same virtual object. In some embodiments, both the first representation 620 and the second representation 621 may be virtual representations.

In an embodiment, a modification to one representation of a virtual object may selectively result in a modification to another representation of the virtual object. For example, a request to modify the first representation 620 may be received while the first representation 620 and the second representation 621 are simultaneously displayed. In an embodiment, a request to modify the first representation 620, such as modifying a size, UI element, shape, theme, etc., may be received (e.g., from the user 202). In an embodiment, a request (e.g., user input) to modify the first representation 620 may result in a corresponding modification to the second representation 621 (e.g., size, UI element, shape, theme, etc.). In aspects, both the first representation 620 and the second representation 621 may be modified according to the modification request. In some embodiments, whenever a modification to the first representation 621 is requested, a corresponding modification is made to the second representation 621. In other embodiments, a first request to modify the first representation 621 may result in a corresponding modification to the second representation 621. However, a second request to modify the first representation 621 does not result in a corresponding modification of the second representation 621. In this case, when a second request to modify the first representation 620 is received, the modification to the second representation 621 is aborted. It is noted that although the foregoing discussion describes selectively modifying the second representation 621 based on a request to modify the first representation 620, this is for illustrative purposes and not by way of limitation. Thus, the same technique may be used to selectively modify the first representation 620 based on a request to modify the second representation 621.

FIG. 7 is a flow diagram illustrating a method 700 for controlling the simultaneous display of representations of virtual objects within a CGR environment. In some embodiments, method 700 may be performed by system 100 or a portion of system 100. In some embodiments, method 700 may be performed by one or more external systems and/or devices. In some embodiments, method 700 may be performed by system 100 (or a portion of system 100) in conjunction with one or more external systems and/or devices.

At block 702, the system displays, via a display of an electronic device (e.g., a wearable electronic device, an HMD device, etc.), a 2D representation of a virtual object at a first surface (and/or location) of a CGR environment. For example, a first representation of a virtual object may be displayed on a representation of a display within a CGR environment via a first display (e.g., a left eye display panel) or a second display (e.g., a second eye display panel) of an electronic device. In some embodiments, the first surface may be a virtual surface within the CGR environment. For example, the first surface may be a virtual representation of a physical display. In other embodiments, the first surface may be a real-world physical surface of the CGR environment. For example, the first surface may be a surface of a physical display. The 2D representation of the virtual object may be a virtual representation (e.g., a virtual representation superimposed over the first surface via a translucent display of the electronic device) or may be a real-world graphical representation (e.g., a real-world graphical representation displayed on a real-world physical display).

In some embodiments, the 2D representation of the virtual object may include a set of UI elements for user interaction with the virtual object. In an embodiment, the 2D representation of the virtual object may further include at least one control for requesting simultaneous display of the second representation of the virtual object.

At block 704, the system receives a request to display a 3D representation of a virtual object concurrently with a 2D representation. In an embodiment, the request for simultaneous display may include user input. The request may be input by the user using a control element (e.g., a button, an affordance, a user interface element, an interactive element, etc.) displayed with the 2D representation (e.g., within the 2D representation or outside of the 2D representation). For example, a user may select a control element, and the selection may cause the system to receive a request for simultaneous display.

In some embodiments, the request to simultaneously display the 2D representation and the 3D representation may include a gesture to move or drag the 2D representation out of or off of the first surface. For example, the user 202 may grab, click, and/or otherwise select (e.g., using an appendage, an input device, an input sensor, etc.) the 2D representation displayed at the first surface, and may move or drag the 2D representation away from the first surface. In some aspects, the drag gesture may be determined as a request for simultaneous display.

In an embodiment, a request to simultaneously display a 3D representation of a virtual object with a 2D representation may cause an animation to be played, wherein the 3D representation is configured to pop out (or pop up) from the 2D representation. In an embodiment, the animation may include sound that may be played during the animation.

At block 706, in response to the request to simultaneously display, the system simultaneously displays, via a display of the electronic device, the 2D representation at the first surface and the 3D representation at the second surface of the CGR environment. In some embodiments, the second surface may be different from the first surface. In an embodiment, the second surface may be a virtual surface, or may be a real-world physical surface within the CGR environment. For example, the second surface may be a physical real-world surface of a desk, or may be a virtual representation of a surface of a physical desk.

In an embodiment, the second surface on which the 3D representation may be displayed may be determined by user input. For example, a user may drag the 2D representation out of the first surface and continue to drag to the second surface. As such, the 3D representation may be displayed in any surface within the CGR environment where the drag gesture stopped. In other implementations, the second surface may be predetermined, for example, where a control element in the 2D representation is used to request simultaneous display. In some implementations, the user can indicate that the surface of the 3D representation is to be displayed before requesting the simultaneous display. For example, a user may first indicate (e.g., via user input (e.g., user input detected using an input sensor that may include a mouse, stylus, touch-sensitive surface, image sensor (e.g., to perform hand tracking), etc.)) surfaces within the CGR environment other than the first surface. Upon request for simultaneous display, the 3D representation may be displayed at a surface indicated by the user.

In some embodiments, the 3D representation of the virtual object may include a set of UI elements for user interaction. In an embodiment, the set of UI elements of the 3D representation may be different from the set of UI elements of the 2D representation. For example, one set of UI elements may include UI elements not included in another set of UI elements.

Aspects of the present disclosure relate to systems and techniques that provide functionality for controlling a representation of a virtual object based on characteristics of an input mechanism. In an implementation, the representation of the virtual object may be based on characteristics of the input mechanism relative to the virtual object (e.g., direction of movement of the input mechanism, distance, gesture type, etc.). For example, in an embodiment, the representation of the virtual object may be modified or maintained depending on whether an input mechanism associated with the virtual object is within a predetermined distance from the first representation of the virtual object. In other embodiments, for example, the representation of the virtual object may be modified or maintained in accordance with a determination of whether an input mechanism associated with the virtual object moves toward or away from the first representation of the virtual object. In other embodiments, for example, the representation of the virtual object may be modified or maintained to indicate a likelihood of the user interacting with the first representation of the virtual object based on whether a gesture associated with the input mechanism is determined. It will be appreciated that the functionality provided by the systems and techniques described herein provide an advantageous system in which the representation of a virtual object may be adapted to the characteristics of an input mechanism, thereby providing an improved user interface.

Fig. 8A and 8B illustrate an exemplary technique for controlling representations of virtual objects within a CGR environment based on characteristics of an input mechanism, according to aspects of the present disclosure. In particular, FIG. 8A shows a CGR environment 890 including an input mechanism 800 and virtual objects 810. In an embodiment, CGR environment 890 may be presented to a user (e.g., user 202) wearing an electronic device (e.g., electronic device 200) configured to allow user 202 to view CGR environment 890. As described above, in some embodiments, the electronic device 200 may be similar to the electronic device 100a described above with reference to fig. 1A and 1B.

As shown in FIG. 8A, a first representation 810 of a virtual object may be displayed by the electronic device 200. In an embodiment, the first representation 810 may be a 3D representation of a virtual object, and the virtual object may be associated with a particular application. For example, as shown in FIG. 8A, a first representation 810 may be associated with a calculator application. It should be understood that the use of a particular application (e.g., a calculator application) to illustrate the first representation 810 and other representations of virtual objects is for illustrative purposes and is not intended to be limiting in any way. Accordingly, first representation 810 may be associated with any type of application (e.g., calendar, multimedia application, presentation, etc.).

In an embodiment, the first representation 810 may be configured to facilitate indirect interaction between a user and the first representation 810. As used herein, indirect interaction may refer to user interaction with a representation of a virtual object that does not directly manipulate elements of the representation of the virtual object. A non-limiting example of indirect interaction may be user perception of information provided by a User Interface (UI) element of a representation of a virtual object without requiring the user to directly manipulate the UI element. In contrast, as used herein, direct interaction may refer to user interaction with a representation of a virtual object, where UI elements of the representation of the virtual object representation may be directly manipulated by a user. For example, a user may press a button, may interact with an interactive element, may click on a selectable item and/or affordance, and so forth.

The first representation 810 may include UI elements 811 and 815. In an embodiment, the UI element 815 may represent at least one UI element configured to provide (e.g., output) information associated with the virtual object represented by the first representation 810. For example, the UI element 815 may be a display of the first representation 810. Thus, the UI element 815 may be configured for indirect interaction such that a user may perceive output without directly manipulating the UI element 815. The UI element 811 may represent at least one UI element that may be configured to facilitate a configuration of user interactions (e.g., direct interactions or indirect interactions). For example, the UI elements 811 may be buttons, affordances, user interface elements, interactive elements, and the like, and/or any combination thereof. When the UI element 811 is configured to facilitate direct interaction, a user may select, click, select, and/or otherwise manipulate the UI element 811. In some embodiments, the UI element 811 may be configured to facilitate indirect interaction by displaying the UI element as a 3D element. In this case, the user may perceive the UI element 811 as a 3D element.

In an embodiment, input mechanism 800 may include a mechanism configured to facilitate interaction with a representation of a virtual object. For example, the input mechanism may include a mechanism for a user (e.g., user 202) to manipulate at least one element of the representation of the virtual object or to perceive data provided by the element of the representation of the virtual object. In an embodiment, the input mechanism 800 may include a representation of an appendage of the user (e.g., a finger, a hand, a leg, a foot, etc.), a gaze of the user (e.g., a head gaze, an eye gaze, etc.), an input device (e.g., a mouse, a stylus, etc.) (e.g., distinct from, in operative communication with, physically connected to (e.g., part of) an electronic device), and/or the like. In an embodiment, the representation of the appendage of the user may comprise a virtual representation of the appendage and/or may comprise data representing features of the appendage (e.g., location, orientation, distance from a particular point, etc.) within the CGR environment. In aspects, the input mechanism 800 may be detected using an input sensor (e.g., a touch-sensitive surface, an image sensor, etc.) configured to perform hand tracking, head gaze tracking, eye gaze tracking, finger tracking, etc. As shown in fig. 8A, the input mechanism may include an appendage (e.g., a finger) of the user.

As shown in FIG. 8A and described above, a first representation 810 may be displayed within CGR environment 890, and first representation 810 may be configured to facilitate indirect interaction by a user rather than direct interaction (e.g., by providing UI elements 811 and 815 configured for indirect interaction). As also shown in FIG. 8A, the input mechanism 800 may be located at a current position that is a distance 831 from the first representation 810. In some implementations, the predetermined distance 830 from the first representation 810 may be provided, but in some implementations, the predetermined distance 830 may not be shown within the CGR environment 890. The predetermined distance 830 may be configured to operate as a threshold such that the display of the first representation 810 may be maintained when the current position of the input mechanism is not within the predetermined distance 830 from the first representation 810. For example, because distance 831 may be determined to be greater than predetermined distance 830, the current location of input mechanism 800 may be determined not to be within predetermined distance 830 from first representation 810.

In an embodiment, whether the display of the first representation 810 may be modified or maintained may be based on characteristics of the input mechanism 800. In some implementations, the characteristics of the input mechanism 800 may include a direction of movement, a distance to the representation of the virtual object, a gesture type, and so on. In accordance with a determination that the current position of the input mechanism 800 is not within the predetermined distance 830 from the first representation 810, the display of the first representation 810 may be maintained without displaying another representation of the virtual object. Instead, as described below, and as shown in the example illustrated in FIG. 8B, in accordance with a determination that the current position of the input mechanism 800 is within a predetermined distance 830 from the first representation 810, the display of the first representation 810 may be modified, and the second representation of the virtual object may be displayed. In aspects, the second representation of the virtual object may be different from the first representation 810.

In some implementations, determining whether the position of the input mechanism 800 is within a predetermined distance 830 from the first representation 810 may be performed in response to detecting movement of the input mechanism 800. In these cases, if no movement of the input mechanism 800 is detected, a determination of whether the position of the input mechanism 800 is within the predetermined distance 830 from the first representation 810 may not be performed. In some embodiments, when it is determined that the detected movement is toward the first representation 810, a determination of whether the position of the input mechanism 800 is within a predetermined distance 830 from the first representation 810 may be performed. In these cases, if the movement of the input mechanism 800 is determined to be away from the first representation 810, the determination of whether the position of the input mechanism 800 is within the predetermined distance 830 from the first representation 810 may not be performed even though the movement of the input mechanism 800 may be detected.

In some implementations, the first representation 810 may be initially displayed within the CGR environment 890 in response to determining that the input mechanism 800 is not within the predetermined distance 830 from the location where the first representation 810 is to be displayed. For example, a representation of a virtual object may be determined to be initially displayed at a first location within CGR environment 890. In this example, the first representation of the virtual object may be configured for indirect interaction. Further, in this example, the CGR environment 890 may not include any representation of the virtual object at the first location, but in some cases at least one other representation of the virtual object may be displayed at another location within the CGR environment 890. In response to determining that the representation of the virtual object is initially displayed at a first location within the CGR environment 890, it may be determined whether the current location of the input mechanism 800 is within a predetermined distance 830 from the first location. If it is determined that the current position of the input mechanism 800 is not within the predetermined distance 830 from the first position, a first representation (e.g., first representation 810) may be displayed at the first position. In some embodiments, if the current position of input mechanism 800 is determined to be within a predetermined distance 830 from the first location, a second representation (e.g., second representation 820 described below) configured for direct interaction may be displayed at the first location.

As shown in fig. 8B, the input mechanism 800 may be moved (e.g., in direction 833) from a previous position (e.g., as shown in fig. 8A) to a current position having a distance 832 to the first representation 810. Movement from a previous location to a current location may be detected (e.g., using an input sensor as described above). In response to detecting movement of the input mechanism 800 from the previous position to the current position, it may be determined whether the current position of the input mechanism 800 to the first representation 810 may be within a predetermined distance 830. For example, a distance 832 from the current position of the input mechanism 800 to the first representation 810 may be compared to the predetermined distance 830. In accordance with a determination that distance 832 is greater than predetermined distance 830, it may be determined that the current position of input mechanism 800 is not within predetermined distance 830 from first representation 810. Conversely, in accordance with a determination that distance 832 is not greater than predetermined distance 830, the current position of input mechanism 800 may be determined to be within predetermined distance 830 from first representation 810.

In an embodiment, the display of the first representation 810 may be modified in accordance with a determination that the current position of the input mechanism 800 is within a predetermined distance 830 from the first representation 810. In an embodiment, modifying the display of the first representation 810 may include ceasing to display the first representation 810 and displaying the second representation 820, wherein the second representation 820 may be different from the first representation 810. In some embodiments, the second representation 820 may be displayed at the same location and/or on the same surface at which the first representation 810 is displayed.

In an embodiment, the second representation 820 may be configured for direct interaction between a user (e.g., user 202) and the second representation 820 (e.g., an element of the second representation 820). For example, as shown in fig. 8A, while the first representation 810 includes a UI element 811 configured for indirect interaction (e.g., a UI element displayed as a highlighted 3D UI element), the second representation 820 may include a UI element 821 configured for direct interaction. In this example, the UI element 821 may include at least one UI element displayed as a flat button or a 2D element, where the flat button may not protrude from the second representation 820. It should be appreciated that a flat 2D UI element (e.g., a 2D button) displayed on a physical table (e.g., on the same plane as the physical table) may more readily provide physical feedback when a user manipulates the 2D element. For example, when a user manipulates a 2D element, the user receives feedback provided by the physical table on which the virtual 2D element I is displayed. Moreover, displaying second representation 820 configured for direct interaction may also encourage a user (e.g., user 202) to interact with second representation 820.

In some embodiments, modifying the first representation 810 (which may include displaying the second representation 820) may include animating the modification. For example, one of the differences between the first representation 810 and the second representation 820 may be that the UI element 811 of the first representation 810 is displayed as a highlighted 3D UI element and the UI element 821 of the second representation 820 is displayed as a flat 2D UI element. In this example, the modification of the first representation 810 may include animating the UI elements such that the protruding 3D UI elements of the first representation 810 are rendered back into the flat 2D UI elements of the second representation 820. In an embodiment, the animation may also include sounds that may be played while the animation is occurring.

In another embodiment, modifying the first representation of the virtual object may include moving the first representation to a location closer to the user (e.g., user 202). For example, based on a characteristic of the input mechanism 800 (e.g., the current location of the input mechanism 800 is within a predetermined distance (e.g., predetermined distance 830) from the current location of the first representation (e.g., first representation 810)), the second representation of the virtual object may be displayed. In an embodiment, the second representation of the virtual object may be the same as the first representation, but in a position closer to the user than the current position of the first representation. In some embodiments, for example, the second representation displayed at the new location may be a different representation of the first representation in accordance with the above description.

In further embodiments, determining the characteristics of the input mechanism upon which to modify or maintain the first representation 810 may include determining whether a direction of movement of the input mechanism 800 is toward or away from the first representation 810. For example, as shown in FIG. 8B, the input mechanism 800 may be moved in a direction 833, which is a direction toward the first representation 810. In this case, in accordance with a determination that the direction of movement of the input mechanism 800 is toward the first representation 810, the display of the first representation 810 may be modified, and a second representation of the virtual object (e.g., a second representation 820 configured to facilitate direct interaction by the user) may be displayed. Conversely, in accordance with a determination that the direction of movement of the input mechanism 800 is away from the first representation 810, the display of the first representation 810 may be maintained without displaying another representation (e.g., the second representation 820) of the virtual object. In aspects, the second representation of the virtual object may be different from the first representation 810.

In further embodiments, the characteristic of the input mechanism on which it is determined to modify or maintain the first representation 810 may include determining whether the input mechanism 800 has made a particular type of gesture. In aspects, a particular type of gesture may be a gesture that may indicate a likelihood for direct user interaction. For example, as shown in FIG. 8B, input mechanism 800 may be a pointing hand. In an embodiment, the pointing hand may be considered to be a type of gesture that indicates the likelihood of user interaction. It should be appreciated that a user desiring to use a finger to interact with a virtual object, such as a virtual object for user input represented by a UI element, may do so by having his or her hand form a pointing hand with the finger pointing outward. In this sense, the pointing hand may indicate that the user intends or desires to interact with the virtual object. Accordingly, when it is determined that the input mechanism has made a gesture indicating a likelihood of user interaction (e.g., pointing hand, grabbing hand, etc.), it may be determined to modify the current representation (e.g., first representation 810) configured for indirect interaction to a representation (e.g., second representation 820) configured for direct interaction. In aspects, modifying a current representation configured for indirect interaction to a representation configured for direct interaction may be in accordance with the foregoing description.

In another example, the determination to maintain display of the first representation 810 configured for indirect interaction may be based on a gesture that does not indicate a likelihood of user interaction. For example, a gesture can be detected, which can include a user (e.g., user 202) crossing his or her arm and/or leaning back. In this case, the gesture may be considered a gesture type that does not indicate a likelihood of user interaction. Thus, when it is determined that the user has crossed his or her arm and/or has tipped back, it may be determined that the current representation (e.g., first representation 810) configured for indirect interaction remains without displaying the representation (e.g., second representation 820) configured for direct interaction. In some embodiments, detecting a gesture that does not indicate a likelihood of user interaction may result in determining to modify a current representation (e.g., second representation 820) configured for direct interaction to a representation (e.g., first representation 810) configured for indirect interaction.

It is noted that while the foregoing examples and the examples that follow may focus on the description of the modification of a representation of a virtual object configured for indirect interaction into a representation of a virtual object configured for direct interaction, this is done for illustrative purposes and is not intended to be limiting in any way. In some embodiments, the representation of the virtual object configured for direct interaction may be modified to a representation of the virtual object configured for indirect interaction based on a characteristic of the input mechanism. For example, in some implementations, based on the detected movement of the input mechanism, based on a characteristic of the input mechanism (e.g., in accordance with a determination that the position of the input mechanism is not within a predetermined distance from a representation configured for direct interaction (e.g., the first representation 810), a display of a representation configured for direct interaction (e.g., the first representation 810 described above) may be modified to display a representation configured for indirect interaction (e.g., the second representation 820 described above). Accordingly, the present disclosure provides techniques for selectively and dynamically configuring representations of virtual object augmented interactions (e.g., direct or indirect) based on characteristics of an input mechanism. Thus, when input mechanism-based features are more advantageous, the representation of the virtual object may be configured for direct or indirect interaction.

Additionally, while the foregoing discussion describes the second representation 820 as being configured for direct interaction with a flat 2D UI element, it should be understood that this is for illustrative purposes and not by way of limitation. It should be understood that the representation of the virtual object may be configured for direct interaction by other methods (e.g., orientation, size, angle, shape, color, brightness, language, position, distance, direction, etc.). For example, in an embodiment, based on a feature of the input mechanism (e.g., in accordance with a determination that a current position of the input mechanism is within a predetermined distance from a first representation of the virtual object), display of the first representation may be modified, and modifying may include displaying a second representation that is different from the first representation. In these embodiments, the second representation may include a different orientation, size, angle, shape, color, brightness, language, location, distance, direction, etc. than the first representation, wherein the modification may be configured to allow, encourage, enable, and/or otherwise facilitate direct interaction with the second representation of the virtual object. Some of these embodiments are described in more detail below.

Fig. 9A and 9B illustrate another example of a technique for controlling representations of virtual objects within a CGR environment based on characteristics of an input mechanism, in accordance with aspects of the present disclosure. As shown in FIG. 9A, a first representation 910 of a virtual object may be displayed via a display of the electronic device 200. In an embodiment, the first representation 910 may be a 3D representation of a virtual object, and the virtual object may be associated with a particular application program (e.g., calendar, multimedia application, presentation, etc.), as described above. In the example shown in FIG. 9A, the first representation 910 may be associated with a calculator application.

In an embodiment, the first representation 910 may be configured to facilitate indirect interaction with an associated virtual object. For example, the first representation 910 may include UI elements 911 and 915. In an embodiment, the UI element 915 may represent at least one UI element configured to provide (e.g., output) information associated with the virtual object represented by the first representation 910. For example, the UI element 915 may be a display (e.g., a virtual display) of the first representation 910. In this case, the first representation 910 may be configured to facilitate the indirect interaction of the user (e.g., user 202) by being displayed in an orientation that facilitates the indirect interaction of the user with the UI element 915. For example, the first representation 910 may include an orientation display of the angle 912. In an embodiment, the angle 912 may be an angle configured to place the first representation 910 at an orientation that enables a user to see, hear, or otherwise perceive the UI element 915. As such, the angle 912 facilitates indirect interaction of the user with the UI element 915. In an embodiment, the angle 912 may be measured relative to a surface (e.g., surface 916) on which the first representation 910 is displayed.

In an embodiment, the orientation at which the first representation 910 may be displayed may be determined based on the user's location. For example, a user's gaze (e.g., head gaze and/or eye gaze) may be determined (e.g., by detecting a location of the user's head and/or eyes and then determining the user's gaze), and then the determined user's gaze may be used to determine an orientation in which to display the first representation 910 such that a UI element configured for indirect interaction (e.g., UI element 915) faces the user's gaze.

In an embodiment, the UI element 911 of the first representation 910 may be configured for indirect interaction. In this case, the UI element 811 may be displayed as a highlight button, or as a 3D element, where a flat button may not be highlighted from the first representation 910. As such, UI element 911 (shown in FIG. 9A) is not configured for direct interaction.

As shown in fig. 9A, and as described above, the first representation 910 may be configured to facilitate indirect interaction by a user rather than direct interaction (e.g., by providing a prominent 3D UI element 911 and by orienting the first representation 910 at an angle 912). As also shown in FIG. 9A, the input mechanism 800 may be located at a current position a distance 931 from the first representation 910. In some implementations, a predetermined distance 930 may be provided from the first representation 910.

In an embodiment, the display of the first representation 910 may be maintained in accordance with a determination that the current position of the input mechanism 800 is not within the predetermined distance 930 from the first representation 910. For example, the first representation 910 configured for indirect interaction may continue to be displayed without displaying another representation of the virtual object and/or without changing the first representation 910. Instead, as described below, in accordance with a determination that the current position of the input mechanism 800 is within the predetermined distance 930 from the first representation 910, the display of the first representation 910 may be modified and the second representation of the virtual object may be displayed. In aspects, the second representation of the virtual object may be different from the first representation 910.

As shown in fig. 9B, input mechanism 800 may be moved (e.g., in direction 933) from a previous position (e.g., as shown in fig. 9A) to a current position having a distance 932 to first representation 910. Movement from a previous location to a current location may be detected (e.g., using an input sensor as described above). In response to detecting movement of the input mechanism 800 from the previous position to the current position, it may be determined whether the current position of the input mechanism 800 to the first representation 910 may be within a predetermined distance 930. For example, a distance 932 from the current position of the input mechanism 800 to the first representation 910 may be compared to the predetermined distance 930. In accordance with a determination that distance 932 is greater than predetermined distance 930, it may be determined that the current position of input mechanism 800 is not within predetermined distance 930 from first representation 910. Conversely, in accordance with a determination that distance 932 is not greater than predetermined distance 930, the current position of input mechanism 800 may be determined to be within predetermined distance 930 from first representation 910.

In an embodiment, the display of the first representation 910 may be modified in accordance with a determination that the current position of the input mechanism 800 is within a predetermined distance 930 from the first representation 910. In an embodiment, modifying the display of the first representation 910 may include ceasing to display the first representation 910 and displaying the second representation 920, wherein the second representation 920 may be different from the first representation 910. In some embodiments, the second representation 920 may be displayed at the same location and/or on the same surface at which the first representation 910 is displayed.

In an embodiment, the second representation 920 may be configured to facilitate direct interaction of a user (e.g., user 202) with an associated virtual object. For example, while the first representation 910 is displayed in an orientation having an angle 912 that facilitates a user being able to perceive (e.g., see, hear, etc.) information (e.g., indirect interaction) provided by the UI element 915, the second representation 920 may be displayed in an orientation that facilitates a user interacting directly (e.g., directly manipulating, selecting, clicking, dragging, and/or otherwise selecting) with a UI element (e.g., UI element 921) of the second representation 920. For example, second representation 920 may be displayed within CGR environment 890 in an orientation that is longitudinal to surface 916. Thus, the second representation 920 may be shown lying flat on the surface 916. It should be appreciated that a flat surface may interact with it more easily than an angled surface. Thus, by modifying the representation of the virtual object from an angled orientation to a flat orientation, or vice versa, the representation of the virtual object is selectively adapted to enhance the direct interaction based on the characteristics of the input mechanism. In some embodiments, the second representation 920 may be displayed in an orientation with respect to the surface 916 having a non-zero angle that is different than the angle 912.

Further, while the first representation 910 includes a UI element 911 configured for indirect interaction, as shown in fig. 9A (e.g., a UI element displayed as a highlighted 3D UI element, where the highlighted 3D UI element may be highlighted (or pop-up) from the first representation 910), the second representation 920 may include a UI element 921 configured for direct interaction, as previously described. For example, the UI element 921 may include at least one UI element displayed as a flat 2D UI element displayed on a physical object, which facilitates physical feedback when a user manipulates the 2D UI element.

In some embodiments, modifying the first representation 910 (which may include displaying the second representation 920) may include displaying the modification in an animated manner. For example, the modification of the first representation 910 may include animating a change in orientation of the first representation 910 such that the first representation 910 is displayed as moving from a current orientation (e.g., angled at an angle 912) to an orientation of the second representation 920 (e.g., flat on the surface 916). Additionally or alternatively, the modification of the first representation 910 may include animating the UI elements such that the protruding 3D UI elements of the first representation 910 are rendered back into the flat 2D UI elements of the second representation 920. In an embodiment, the animation may also include sounds that may be played while the animation is occurring.

Fig. 10A and 10B illustrate another example of a technique for controlling representations of virtual objects within a CGR environment based on characteristics of an input mechanism, in accordance with aspects of the present disclosure. In particular, fig. 10A and 10B illustrate an example in which a representation of a virtual object is modified based on characteristics of an input mechanism, and in which the modification includes adding a UI element for user interaction and changing a size of the representation.

As shown in fig. 10A, a first representation 1010 of a virtual object may be displayed via a display of the electronic device 200. In an embodiment, the first representation 1010 may be a 3D representation of a virtual object, and the virtual object may be associated with a particular application program (e.g., calendar, multimedia application, presentation, etc.), as described above. In the example shown in FIG. 10A, the first representation 1010 may be associated with a calculator application.

In an embodiment, the first representation 1010 may be configured to facilitate indirect interaction with an associated virtual object. For example, the first representation 910 may include a UI element 1012. The UI element 1012 may represent at least one UI element configured to provide (e.g., output) information associated with the virtual object represented by the first representation 1010. For example, the UI element 1012 may be a display (e.g., a virtual display) of the first representation 1010. In some embodiments, the first representation 1010 may have a size. In some embodiments, the first representation 1010 may not include any UI elements (e.g., buttons, affordances, user interface elements, interactive elements, etc.) configured for user input.

As shown in FIG. 10A and described above, a first representation 1010 may be displayed within CGR environment 890 and first representation 1010 may be configured to facilitate indirect rather than direct interaction by a user. Also as shown in FIG. 10A, the input mechanism 800 may be located at a current position that is a distance 1031 from the first representation 1010. In some implementations, a predetermined distance 1030 from the first representation 1010 can be provided.

In an embodiment, the display of the first representation 1010 may be maintained in accordance with a determination that the current position of the input mechanism 800 is not within a predetermined distance 1030 from the first representation 1010. For example, the first representation 1010 configured for indirect interaction may continue to be displayed without displaying another representation of the virtual object and/or without changing the first representation 1010. Instead, as described below, in accordance with a determination that the current position of the input mechanism 800 is within a predetermined distance 1030 from the first representation 1010, the display of the first representation 1010 may be modified and the second representation of the virtual object may be displayed. In aspects, the second representation of the virtual object may be different from the first representation 1010.

As shown in fig. 10B, the input mechanism 800 may be moved from a previous position (e.g., as shown in fig. 9A) to a current position having a distance 1030 to the first representation 1010. Movement from a previous location to a current location may be detected (e.g., using an input sensor as described above). In response to detecting movement of the input mechanism 800 from the previous position to the current position, it may be determined whether the current position of the input mechanism 800 to the first representation 1010 may be within a predetermined distance 1030. In accordance with a determination that the current position of the input mechanism 800 is within a predetermined distance 1030 from the first representation 1010, the display of the first representation 1010 may be modified. In an embodiment, modifying the display of the first representation 1010 may include ceasing to display the first representation 1010 and displaying the second representation 1020, wherein the second representation 1020 may be different from the first representation 1010. In some embodiments, the second representation 1020 may be displayed at the same location and/or on the same surface at which the first representation 1020 is displayed.

In an embodiment, the second representation 1020 may be configured to facilitate direct interaction of a user (e.g., user 202) with an associated virtual object. For example, while the first representation 1010 may not include a UI element 911 configured for user input, the second representation 1020 may include a UI element 1021 configured for user interaction, as previously described. For example, UI element 1021 may include at least one UI element that is displayed as a flat 2D UI element.

Further, the second representation 1020 may be displayed having a different size than the first representation 1010. For example, the second representation 1020 may be displayed at a size that is larger than the size of the first representation 1010. In some embodiments, the second representation 1020 may be displayed in a size that is smaller than the size of the first representation 1010.

As previously described, in some embodiments, modifying the first representation 1010 (which may include displaying the second representation 1020) may include displaying the modification in an animated manner. For example, the modification of the first representation 1010 may include animating a change in size of the first representation 1010 such that the first representation 1010 is displayed as being optionally increased or shrunk from a current size to a size of the second representation 1020. Additionally or alternatively, the modification of the first representation 1010 may include animating the UI elements such that the protruding 3D UI elements of the first representation 910 are rendered back into the flat 2D UI elements of the second representation 920. In an embodiment, the animation may also include sounds that may be played while the animation is occurring.

11A and 11B illustrate another example of a technique for controlling representations of virtual objects within a CGR environment based on characteristics of an input mechanism, in accordance with aspects of the present disclosure. In particular, fig. 11A and 11B illustrate examples of modifying a representation of a virtual object based on a characteristic of an input mechanism (e.g., a user's gaze).

As shown in FIG. 11A, a first representation 1110 of a virtual object may be displayed via a display of the electronic device 200. In an embodiment, the first representation 1110 may be a representation of a virtual object, and the virtual object may be associated with a particular application program (e.g., calendar, multimedia application, presentation, etc.), as described above. In the example shown in FIG. 11A, the first representation 1110 can be associated with a calendar application program.

In an embodiment, first representation 1110 may have dimensions and may be displayed at location 1152. In an embodiment, first representation 1110 may not be configured for user interaction, whether direct or indirect. For example, the size of the first representation 1110 may be a small size, and the small size may not enable a user to perceive or interact with any information from any UI element of the first representation 1110. In some embodiments, first representation 1110 may not include any UI elements.

As shown in fig. 11A, a gaze 1150 of a user 202 wearing the electronic device 200 may be detected. In aspects, the detected gaze 1150 may be a head gaze (e.g., a direction the user's head is facing), an eye gaze (e.g., a direction the user's eyes are looking at), a combination thereof, and the like. The gaze 1150 of the user 202 may be determined to be focused, placed, or otherwise directed at a location 1151, which may be different from the location 1152 at which the first representation 1110 is displayed. In aspects, in accordance with a determination that the gaze 1150 is directed to a location different from the location of the first representation 1110, the display of the first representation 1110 at the current location and having a size may be maintained without displaying another representation of the virtual object and/or without making any changes to the first representation 1110.

Fig. 11B shows that the gaze 1150 of the user 202 has changed to a direction different from the direction pointing to location 1151. In an implementation, a change in gaze may be detected (e.g., via an input sensor). In response to a detected change in the user's gaze, a direction of a new direction of gaze may be determined. For example, a new direction of gaze 1150 may be determined to point to location 1152. Location 1152 may be the location where the first representation 1110 is displayed. In an embodiment, in accordance with a determination that the gaze 1150 is directed to the same location as the location of the first representation 1110, the display of the first representation 1110 may be modified.

In some embodiments, determining to modify the display of the first representation 1110 in accordance with a determination that the gaze 1150 is directed to the same location as the location of the first representation 1110 may include determining that the gaze 1150 has remained directed to the same location as the location of the first representation 1110 for at least a predetermined period of time. When it is determined that the gaze 1150 has remained pointing at the same location as the location of the first representation 1110 for a period of time less than the predetermined period of time (e.g., the direction of the gaze 1150 moved to a different direction before the predetermined period of time expires), the display of the first representation 1110 may not be modified, but may instead be maintained without displaying another representation of the virtual object and/or without making any changes to the first representation 1110. When it is determined that gaze 1150 remains pointed at the same location as the location of first representation 1110 for at least the same period of time as the predetermined period of time (e.g., the direction of gaze 1150 has not moved to a different direction before the predetermined period of time expires), the display of first representation 1110 may be modified.

In an embodiment, modifying the display of the first representation 1110 may include ceasing to display the first representation 1110 and displaying the second representation 1120, wherein the second representation 1120 may be different from the first representation 1110. In some implementations, the second representation 1120 can be displayed at the same location and/or on the same surface at which the first representation 1120 is displayed.

In an embodiment, the second representation 1120 may be different from the first representation 1110, and the second representation 1120 may be configured to facilitate interaction by a user (e.g., user 202). For example, the second representation 1120 may be configured to include a UI element 1112. UI element 1112 may include at least one UI element, such as a display, configured for user interaction. In some embodiments, the second representation 1120 can alternatively or additionally have a different size than the first representation 1110. For example, the size of the second representation 1120 may be larger or smaller than the size of the first representation 1110. In an implementation, the size of the second representation 1120 may be based on the distance between the location of the second representation 1120 (e.g., location 1152) and the location of the user's head and/or eyes (e.g., location 1153).

In some embodiments, second representation 1120 may be configured for indirect interaction, but may not be configured for direct interaction. For example, the second representation 1120 may not include any UI elements (e.g., buttons, affordances, user interface elements, interactive elements, etc.) configured for direct interaction with a user. In this case, the techniques described above with respect to fig. 8A, 8B, 9A, 9B, 10A, and 10B may be used to selectively modify the second representation 1120 into a configuration for direct interaction based on features according to the input mechanisms disclosed herein (e.g., representations of an appendage, mouse, stylus, etc.). As such, the representation of the virtual object may be selectively and dynamically modified from the non-interactive configuration to the non-direct interactive configuration based on a characteristic of the input mechanism (e.g., a user's gaze) and then further modified from the non-direct interactive configuration to the direct interactive configuration based on another characteristic of the input mechanism or based on a characteristic of another input mechanism (e.g., a representation of an appendage, an input device, etc.).

Fig. 12A and 12B illustrate another example in which representations of virtual objects within a CGR environment are modified based on a user's gaze. In particular, fig. 12A shows a user 202 wearing an electronic device 200 configured to allow user 202 to view CGR environment 890. As shown in FIG. 12A, a first representation 1210 of a virtual object may be displayed at a particular size at a location 1251 via a display of the electronic device 200. In aspects, the location 1251 may be on a wall of the CGR environment 890. In an embodiment, the first representation 1210 may be a representation of a virtual object, and the virtual object may be associated with a particular application program (e.g., calendar, multimedia application, presentation, etc.), as described above. In the example shown in FIG. 12A, the first representation 1210 can be associated with a calendar application. In an embodiment, first representation 1210 may not be configured for user interaction, whether direct or indirect. For example, the size of the first representation 1210 may be a small size, and the small size may not enable a user to perceive or interact with any information from any UI element of the first representation 1210.

As shown in fig. 12A, it may be determined that the gaze 1250 of the user 202 is directed to a location 1252, which may be different from the location 1251 at which the first representation 1210 is displayed. In aspects, in accordance with a determination that the gaze 1250 is directed to a location different from the location of the first representation 1210, the display of the first representation 1210 may be maintained without displaying another representation of the virtual object and/or without making any changes to the first representation 1210.

Fig. 12B shows that the gaze 1250 of the user 202 has changed to a direction different from the direction pointing to the location 1252. In an implementation, a change in gaze may be detected (e.g., via an input sensor). In response to a detected change in the user's gaze, a direction of a new direction of gaze may be determined. For example, a new direction of gaze 1250 may be determined to point to location 1251. Position 1251 is the position at which the first representation 1210 is displayed. In an embodiment, in accordance with a determination that gaze 1250 is directed to the same location as the location of the first representation 1210, the display of the first representation 1210 may be modified. For example, the first representation 1210 may cease to be displayed and the second representation 1220 may be displayed, where the second representation 1220 may be different from the first representation 1210. In some embodiments, the second representation 1220 may be displayed at the same location and/or on the same surface at which the first representation 1210 is displayed.

In an embodiment, the second representation 1220 may be configured to include a UI element 1221. The UI elements 1221 may include at least one UI element, such as a display, configured for user interaction. In some embodiments, the second representation 1220 can alternatively or additionally have a different size than the first representation 1210. For example, the size of the second representation 1220 may be larger or smaller than the size of the first representation 1210. In an embodiment, the size of the second representation 1220 may be based on the distance between the location of the second representation 1220 (e.g., location 1251) and the location of the user's head and/or eyes. In some embodiments, second representation 1120 may be configured for indirect interaction, but may not be configured for direct interaction. For example, the second representation 1120 may not include any UI elements (e.g., buttons, affordances, user interface elements, interactive elements, etc.) configured for direct interaction with a user.

In some embodiments, determining to modify the display of the first representation 1210 in accordance with a determination that the gaze 1250 is directed to the same location as the location of the first representation 1210 may include determining that the gaze 1250 has remained directed to the same location as the location of the first representation 1210 for at least a predetermined period of time, as described with reference to fig. 11A and 11B.

As previously described, in an embodiment, modifying the first representation (which may include displaying the second representation) may include displaying the modification in an animated manner. For example, the modification of the first representation may include animating a change in size of the first representation such that the first representation is displayed as optionally increasing or shrinking from a current size to a size of the second representation. Additionally or alternatively, the modification of the first representation may include animating the UI element of the first representation such that the UI element is rendered to fall back into the first representation. In an embodiment, the animation may also include sounds that may be played while the animation is occurring.

It is noted that, in embodiments, implementations of the techniques described herein may include any combination of the features and functions described above. For example, the representation of the virtual object may be modified to have any of different sizes, different UI elements, different types of UI elements (e.g., flat UI elements, protruding UI elements, etc.), different orientations, different positions, different shapes, different intensities, etc., and/or any combination thereof.

FIG. 13 is a flow diagram illustrating a method 1300 for controlling a representation of a virtual object within a CGR environment based on characteristics of an input mechanism. In some embodiments, method 1300 may be performed by system 100 or a portion of system 100. In some embodiments, method 1300 may be performed by one or more external systems and/or devices. In some embodiments, method 1300 may be performed by system 100 (or a portion of system 100) in conjunction with one or more external systems and/or devices.

At block 1302, the system displays a first representation of a virtual object within the CGR environment via a display of an electronic device (e.g., a wearable electronic device, an HMD device, etc.). For example, a first representation of a virtual object may be displayed on a representation of a display within a CGR environment via a first display (e.g., a left eye display panel) or a second display (e.g., a second eye display panel) of an electronic device. In an embodiment, the first representation of the virtual object may be a virtual representation (e.g., a virtual representation superimposed on the first surface of the CGR environment via a translucent display of the electronic device).

In an embodiment, the first representation of the virtual object may be configured to facilitate indirect interaction with the virtual object. For example, the first representation of the virtual object may include at least one of the UI elements configured for indirect interaction such that a user may perceive interaction with the UI element without directly manipulating the UI element (e.g., the UI element configured for output).

In an embodiment, the first representation of the virtual object may include at least one of the UI elements that may be configured to facilitate indirect interaction but not configured to interact directly (e.g., the UI element may be displayed as a highlighted 3D UI element). For example, the UI elements may include buttons, affordances, user interface elements, interactive elements, and the like, and/or any combination thereof. When the UI element is configured to facilitate direct interaction, the user may select, click, select, and/or otherwise manipulate the UI element.

In embodiments, movement of the input mechanism may be detected. The input mechanism may include a mechanism configured to facilitate interaction with the virtual object. For example, the input mechanism may include a mechanism for a user to manipulate at least one element of a representation of a virtual object or to perceive data provided by the virtual object. In an embodiment, the input mechanism may include a representation of the appendage of the user (e.g., a finger, a hand, a leg, a foot, etc.), a gaze of the user (e.g., a head gaze, an eye gaze, etc.), an input device (e.g., a mouse, a stylus, etc.), and/or the like. In an embodiment, the representation of the appendage of the user may comprise a virtual representation of the appendage and/or may comprise data representing features of the appendage (e.g., location, orientation, distance from a particular point, etc.) within the CGR environment. In aspects, movement of the input mechanism may be detected using an input sensor (e.g., a touch-sensitive surface, an image sensor, etc.) configured to perform hand tracking, head gaze tracking, eye gaze tracking, finger tracking, etc. For example, the input mechanism may be moved from a previous position to a current position.

In an embodiment, in response to a detected movement of the input mechanism, it may be determined whether a current position of the input mechanism is within a predetermined distance from the first representation. However, when no movement of the input mechanism is detected, the determination of whether the current position of the input mechanism is within a predetermined distance from the first representation may not be performed. In some embodiments, the determination of whether the current position of the input mechanism is within a predetermined distance from the first representation may be performed when it is determined that the detected movement is towards the first representation. In these cases, if it is determined that the movement of the input mechanism is away from the first representation, the determination of whether the current position of the input mechanism is within a predetermined distance from the first representation may not be performed even though the movement of the input mechanism may be detected.

In accordance with a determination that the current position of the input mechanism is within the predetermined distance from the first representation of the virtual object, the system displays, via a display of the electronic device, a second representation of the virtual object within the CGR environment at block 1304. In an embodiment, the second representation of the virtual object may be different from the first representation of the virtual object.

In an embodiment, in response to displaying the second representation of the virtual object, the display of the first representation may cease. In some embodiments, the second representation may be displayed at the same location and/or on the same surface at which the first representation is displayed.

In an embodiment, the second representation may be configured to facilitate direct interaction of the user with the associated virtual object. For example, the second representation may include at least one of the UI elements configured for direct interaction. In an embodiment, the UI element may include at least one UI element displayed as a flat 2D UI element displayed on the physical object. In an embodiment, the UI elements may include any one and/or any combination of buttons, affordances, user interface elements, interactive elements, and the like.

In some embodiments, the second representation may have a different size than the first representation. For example, the size of the second representation may be larger than the size of the first representation. In an embodiment, the second representation may include a portion of the first representation, and the portion of the first representation included in the second representation may be larger than a size of the same portion in the first representation.

In some embodiments, the second representation of the virtual object may be displayed at a location different from the current location of the first representation. In an embodiment, the location at which the second representation of the virtual object may be displayed may be a location closer to the user than the current location of the first representation. In an embodiment, displaying the second representation at the new location may be the same representation as the first representation.

In some embodiments, the first representation may be a 3D representation of the virtual object and the second representation may be a 2D representation of the virtual object. In an embodiment, the second representation may include at least a portion of the virtual object that is not displayed in the first representation of the virtual object.

As described above, one aspect of the present technology is to collect and use data available from various sources to provide dedicated resource management for low power devices with additional displays (e.g., HMD devices with additional displays), thereby conserving battery life for users and providing dedicated content to users of low power devices. The present disclosure contemplates that, in some instances, this collected data may include personal information data that uniquely identifies or may be used to contact or locate a particular person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, twitter IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

The present disclosure recognizes that the use of such personal information data in the present technology may be useful to benefit the user. For example, personal information data may be used to conserve battery life of a user's low power device. Thus, for example, using such personal information data, the system properly manages resources to conserve battery life of the low power device. In addition, the present disclosure also contemplates other uses for which personal information data is beneficial to a user. For example, health and fitness data may be used to provide insight into the overall health condition of a user, or may be used as positive feedback for individuals using technology to pursue health goals.

The present disclosure contemplates that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information data will comply with established privacy policies and/or privacy practices. In particular, such entities should enforce and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining privacy and security of personal information data. Users can conveniently access such policies and should update as data is collected and/or used. Personal information from the user should be collected for legitimate and legitimate uses by the entity and not shared or sold outside of these legitimate uses. Furthermore, such acquisition/sharing should be performed after receiving user informed consent. Furthermore, such entities should consider taking any necessary steps to defend and secure access to such personal information data, and to ensure that others who have access to the personal information data comply with their privacy policies and procedures. In addition, such entities may subject themselves to third party evaluations to prove compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to the particular type of personal information data collected and/or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the united states, the collection or acquisition of certain health data may be governed by federal and/or state laws, such as the health insurance association and accountability act (HIPAA); while other countries may have health data subject to other regulations and policies and should be treated accordingly. Therefore, different privacy practices should be maintained for different personal data types in each country.

Regardless of the foregoing, the present disclosure also contemplates examples in which a user selectively prevents use or access to personal information data. That is, the present disclosure contemplates that hardware elements and/or software elements may be provided to prevent or block access to such personal information data. For example, with respect to managing resources of a low power device, the present techniques may be configured to allow a user to opt-in or opt-out of participating in the collection of personal information data at any time during or after registration service. In another example, the user may choose not to provide eye tracking data for specialized resource management, such as pupil location, pupil dilation, and/or blink rate. In yet another example, the user may choose to limit the length of time that the glasses tracking data is maintained or to inhibit the development of the baseline glasses tracking file altogether. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications related to accessing or using personal information. For example, the user may be notified that their personal information data is to be accessed when the application is downloaded, and then be reminded again just before the personal information data is accessed by the application.

Further, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of inadvertent or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by limiting data collection and deleting data. In addition, and when applicable, including in certain health-related applications, data de-identification may be used to protect the privacy of the user. De-identification may be facilitated by removing particular identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at a city level rather than at an address level), controlling how data is stored (e.g., aggregating data among users), and/or other methods, as appropriate.

Thus, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed examples, this disclosure also contemplates that various examples may also be implemented without having to access such personal information data. That is, various examples of the present technology do not fail to function properly due to lack of all or a portion of such personal information data. For example, the resources of the low power device can be managed and content (e.g., status updates and/or objects) can be selected and delivered to the user by inferring preferences based on non-personal information data or an absolute minimum of personal information, such as content requested by a device associated with the user, other non-personal information available to a system controlling the low power device, or publicly available information.

Claims (57)

1. A method, comprising:

displaying, via a display of a wearable electronic device, a first representation of a virtual object at a first location within a computer-generated reality (CGR) environment, wherein the first location corresponds to a first usage context of a plurality of usage contexts;

receiving a request to move the first representation within the CGR environment to a second location different from the first location; and

in response to receiving the request:

in accordance with a determination that the second location corresponds to a second usage context of the plurality of usage contexts, displaying, via the display of the wearable electronic device, a second representation of the virtual object at the second location based on the second usage context, wherein the second representation is different from the first representation; and

in accordance with a determination that the second location corresponds to a third usage context of the plurality of usage contexts, displaying, via the display of the wearable electronic device, a third representation of the virtual object at the second location based on the third usage context, wherein the third representation is different from the first representation and the second representation.

2. The method of claim 1, wherein:

the second one of the plurality of usage contexts comprises a usage context that is satisfied when it is determined that the second location corresponds to a location on a representation of an electronic device; and is

In accordance with the determination that the second location corresponds to the second usage context of the plurality of usage contexts, displaying the second representation of the virtual object based on the second usage context comprises displaying a two-dimensional representation on the representation of the electronic device within the CGR environment.

3. The method of any of claims 1-2, wherein:

the second one of the plurality of usage contexts comprises a usage context that is satisfied when it is determined that the second location corresponds to a location on a vertical plane; and is

In accordance with the determination that the second location corresponds to the second usage context of the plurality of usage contexts, displaying the second representation of the virtual object based on the second usage context comprises displaying a two-dimensional representation on the vertical plane within the CGR environment.

4. The method of claim 3, wherein a size of the second representation is based on a distance between the display of the electronic device and the vertical plane within the CGR environment.

5. The method of any of claims 1-4, wherein:

the second one of the plurality of usage contexts comprises a usage context that is satisfied when it is determined that the second location corresponds to a location on a horizontal plane within the CGR environment; and is

In accordance with the determination that the second location corresponds to the second usage context of the plurality of usage contexts, displaying the second representation of the virtual object based on the second usage context comprises displaying a three-dimensional representation on the horizontal plane within the CGR environment.

6. The method of any of claims 1-5, wherein:

in accordance with a determination that the horizontal plane is a first type of horizontal plane, the three-dimensional representation is a representation of a first size; and is

In accordance with a determination that the horizontal plane is a horizontal plane of a second type, the three-dimensional representation is a representation of a second size that is different from the first size.

7. The method of any of claims 1-6, wherein:

the second one of the plurality of usage contexts comprising a usage context that is satisfied when a maximize view criterion is satisfied, displaying the second representation of the virtual object based on the second usage context comprising displaying a plurality of representations of virtual objects on a plurality of planes within the CGR environment.

8. The method of claim 7, wherein selectable virtual objects corresponding to maximized visual representations are displayed, and wherein maximized view criteria include criteria that are met when the selectable virtual objects corresponding to maximized visual representations are selected.

9. The method of any one of claims 1 to 8, wherein:

determining that the second location corresponds to the second usage context of the plurality of usage contexts;

the first representation comprises first visual content; and is

The second representation does not include the first visual content.

10. The method of any one of claims 1 to 9, wherein:

determining that the second location corresponds to the second usage context of the plurality of usage contexts;

the first representation includes third visual content displayed in a third size; and is

The second representation includes the third visual content displayed in a fourth size different from the third size.

11. The method of any one of claims 1 to 10, wherein:

determining that the second location corresponds to the second usage context of the plurality of usage contexts;

the first representation comprises a first selectable object; and is

The second representation does not include the first selectable object.

12. The method of any one of claims 1 to 11, wherein:

determining that the second location corresponds to the second usage context of the plurality of usage contexts;

the first representation is a fourth size; and is

The second representation is a fifth size different from the fourth size.

13. The method of any of claims 1-12, wherein displaying the second representation of the virtual object based on the second usage context comprises transitioning display of the first representation to display of the second representation when the first representation is within a predetermined distance from the second location.

14. The method of any of claims 1 to 13, further comprising:

in accordance with a determination that the second location corresponds to a fourth usage context of the plurality of usage contexts, wherein the fourth usage context is satisfied when the second location corresponds to a prohibited location, forgoing displaying the representation of the virtual object within the CGR environment based on the fourth usage context.

15. The method of claim 14, further comprising:

in accordance with the determination that the second location corresponds to the fourth one of the plurality of usage contexts, displaying, within the CGR environment, an indication that the second location is a prohibited location.

16. A system, comprising:

a display;

one or more processors; and

memory storing one or more programs configured for execution by the one or more processors, the one or more programs including instructions for:

displaying, via the display, a first representation of a virtual object at a first location within a Computer Generated Reality (CGR) environment, wherein the first location corresponds to a first usage context of a plurality of usage contexts;

receiving a request to move the first representation within the CGR environment to a second location different from the first location; and

in response to receiving the request:

in accordance with a determination that the second location corresponds to a second usage context of the plurality of usage contexts, displaying, via the display, a second representation of the virtual object at the second location based on the second usage context, wherein the second representation is different from the first representation; and

in accordance with a determination that the second location corresponds to a third usage context of the plurality of usage contexts, displaying, via the display, a third representation of the virtual object at the second location based on the third usage context, wherein the third representation is different from the first representation and the second representation.

17. A non-transitory computer readable storage medium storing one or more programs configured for execution by one or more processors of a system with a display, the one or more programs comprising instructions for:

displaying, via the display, a first representation of a virtual object at a first location within a Computer Generated Reality (CGR) environment, wherein the first location corresponds to a first usage context of a plurality of usage contexts;

receiving a request to move the first representation within the CGR environment to a second location different from the first location; and

in response to receiving the request:

in accordance with a determination that the second location corresponds to a second usage context of the plurality of usage contexts, displaying, via the display, a second representation of the virtual object at the second location based on the second usage context, wherein the second representation is different from the first representation; and

in accordance with a determination that the second location corresponds to a third usage context of the plurality of usage contexts, displaying, via the display, a third representation of the virtual object at the second location based on the third usage context, wherein the third representation is different from the first representation and the second representation.

18. A system, comprising:

a display;

means for displaying, via the display, a first representation of a virtual object at a first location within a Computer Generated Reality (CGR) environment, wherein the first location corresponds to a first usage context of a plurality of usage contexts;

means for receiving a request to move the first representation within the CGR environment to a second location different from the first location; and

means, in response to receiving the request, for:

in accordance with a determination that the second location corresponds to a second usage context of the plurality of usage contexts, displaying, via the display, a second representation of the virtual object at the second location based on the second usage context, wherein the second representation is different from the first representation; and

in accordance with a determination that the second location corresponds to a third usage context of the plurality of usage contexts, displaying, via the display, a third representation of the virtual object at the second location based on the third usage context, wherein the third representation is different from the first representation and the second representation.

19. A system, comprising:

a display;

one or more processors; and

memory storing one or more programs configured for execution by the one or more processors, the one or more programs including instructions for performing the method of any of claims 1-15.

20. A non-transitory computer-readable storage medium storing one or more programs configured for execution by one or more processors of a wearable electronic device including a display, the one or more programs including instructions for performing the method of any of claims 1-15.

21. A system, comprising:

a display;

apparatus for performing the method of any one of claims 1 to 15.

22. A method, comprising:

displaying, via a display of a wearable electronic device, a two-dimensional (2D) representation of a virtual object at a first location within a computer-generated reality (CGR) environment;

receiving a request to simultaneously display a three-dimensional (3D) representation of the virtual object with the 2D representation; and

in response to the request, simultaneously displaying, via the display of the wearable electronic device, the 2D representation at the first location and the 3D representation at a second location of the CGR environment, wherein the second location is different from the first location.

23. The method of claim 22, wherein receiving the request to simultaneously display comprises receiving a request to drag the 2D representation of the virtual object from the first location to the second location of the CGR environment.

24. The method of any of claims 22 to 23, wherein the request for simultaneous display includes a selection of an interactive element displayed with the 2D representation of the virtual object.

25. The method of any of claims 22-24, wherein simultaneously displaying the 2D representation at the first location of the CGR environment and the 3D representation at a second location of the CGR environment comprises animating the 3D representation as popping up the 2D representation.

26. The method of any of claims 22 to 25, further comprising:

in response to receiving a first request to modify the 2D representation:

modifying the 3D representation in accordance with a modification corresponding to the first request to modify the 2D representation; and

in response to receiving a first request to modify the 3D representation:

modifying the 2D representation in accordance with a modification corresponding to the first request to modify the 3D representation.

27. The method of claim 26, further comprising:

in response to receiving a second request to modify the 2D representation:

forgoing modifying the 3D representation in accordance with a modification corresponding to the second request to modify the 2D representation; and

in response to receiving a second request to modify the 3D representation:

forgoing modifying the 2D representation in accordance with the modification corresponding to the second request to modify the 3D representation.

28. The method of any of claims 22 to 27, wherein the 2D representation of the virtual object comprises at least one interactive User Interface (UI) element.

29. The method of claim 28, wherein the at least one interactive UI element of the 2D representation is not included in the 3D representation of the virtual object.

30. A system, comprising:

a display;

one or more processors; and

memory storing one or more programs configured for execution by the one or more processors, the one or more programs including instructions for:

displaying, via the display, a two-dimensional (2D) representation of a virtual object at a first location within a computer-generated reality (CGR) environment;

receiving a request to simultaneously display a three-dimensional (3D) representation of the virtual object with the 2D representation; and

in response to the request, simultaneously displaying, via the display, the 2D representation at the first location and the 3D representation at a second location of the CGR environment, wherein the second location is different from the first location.

31. A non-transitory computer readable storage medium storing one or more programs configured for execution by one or more processors of a system with a display, the one or more programs comprising instructions for:

displaying, via the display, a two-dimensional (2D) representation of a virtual object at a first location within a computer-generated reality (CGR) environment;

receiving a request to simultaneously display a three-dimensional (3D) representation of the virtual object with the 2D representation; and

in response to the request, simultaneously displaying, via the display, the 2D representation at the first location and the 3D representation at a second location of the CGR environment, wherein the second location is different from the first location.

32. A system, comprising:

a display;

means for displaying, via the display, a two-dimensional (2D) representation of a virtual object at a first location within a computer-generated reality (CGR) environment;

means for receiving a request to simultaneously display a three-dimensional (3D) representation of the virtual object with the 2D representation; and

means for concurrently displaying, via the display, the 2D representation at the first location and the 3D representation at a second location of the CGR environment in response to the request, wherein the second location is different from the first location.

33. A system, comprising:

a display;

one or more processors; and

memory storing one or more programs configured for execution by the one or more processors, the one or more programs including instructions for performing the method of any of claims 22-29.

34. A non-transitory computer-readable storage medium storing one or more programs configured for execution by one or more processors of a wearable electronic device including a display, the one or more programs including instructions for performing the method of any of claims 22-29.

35. A system, comprising:

a display;

apparatus for performing the method of any of claims 22 to 29.

36. A method, comprising:

displaying, via a display of a wearable electronic device, a first representation of a virtual object within a computer-generated reality (CGR) environment; and

in response to the detected movement of the input mechanism:

in accordance with a determination that the current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, displaying, via the display of the wearable electronic device, a second representation of the virtual object within the CGR environment, wherein the second representation is different from the first representation; and

in accordance with a determination that the current position of the input mechanism is not within the predetermined distance from the first representation of the virtual object, maintaining display of the first representation and not display of the second representation.

37. The method of claim 36, wherein the detected movement of the input mechanism is detected at least in part by an image sensor.

38. The method of any of claims 36-37, wherein the input mechanism includes at least one of an appendage of a user of the wearable electronic device, a gaze of the user, and an input received on an input device different from the electronic device.

39. The method of claim 38, wherein the input mechanism is determined based on a position of the first representation of the virtual object within the CGR environment.

40. The method of any of claims 36-39, wherein displaying the second representation of the virtual object comprises displaying at least one interactive User Interface (UI) element of the second representation.

41. The method of claim 40, wherein the at least one interactive UI element is configured for direct interaction.

42. The method of claim 41, wherein the at least one interactive UI element configured for direct interaction corresponds to at least one UI element of the first representation, the at least one UI element of the first representation configured for indirect interaction.

43. The method of any of claims 40-42, wherein the at least one interactive UI element of the second representation corresponds to at least one UI element of the first representation, and wherein a size of the at least one interactive UI element of the second representation is different from a size of the corresponding at least one UI element of the first representation.

44. The method of any of claims 40-43, wherein the at least one interactive UI element of the second representation corresponds to at least one UI element of the first representation, wherein the at least one UI element of the first representation is displayed as a three-dimensional element in the first representation, and wherein the at least one interactive UI element of the second representation is displayed as a two-dimensional representation of the corresponding three-dimensional element.

45. The method of any of claims 40-44, wherein the at least one interactive UI element of the second representation corresponds to at least one UI element not displayed in the first representation.

46. The method of any of claims 36 to 45, wherein the second representation of the virtual object is a different size than the first representation of the virtual object.

47. The method of any of claims 36-46, wherein displaying the second representation of the virtual object includes displaying the second representation at a location different from the location at which the first representation is displayed.

48. The method of claim 47, wherein the location at which the second representation is displayed is a location closer to a user of the wearable electronic device than the location at which the first representation is displayed.

49. The method of any of claims 36 to 48, wherein the first representation is a three-dimensional (3D) representation of the virtual object and the second representation is a two-dimensional (2D) representation of the virtual object.

50. The method of any of claims 36-49, wherein displaying the second representation of the virtual object within the CGR environment includes animating at least one difference between the second representation and the first representation.

51. The method of claim 50, wherein animating the at least one difference between the second representation and the first representation comprises playing at least one sound associated with the at least one difference.

52. A system, comprising:

a display;

one or more processors; and

memory storing one or more programs configured for execution by the one or more processors, the one or more programs including instructions for:

displaying, via the display, a first representation of a virtual object within a computer-generated reality (CGR) environment; and

in response to the detected movement of the input mechanism:

in accordance with a determination that the current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, displaying, via the display, a second representation of the virtual object within the CGR environment, wherein the second representation is different from the first representation; and

in accordance with a determination that the current position of the input mechanism is not within the predetermined distance from the first representation of the virtual object, maintaining display of the first representation and not display of the second representation.

53. A non-transitory computer readable storage medium storing one or more programs configured for execution by one or more processors of a system with a display, the one or more programs comprising instructions for:

displaying, via the display, a first representation of a virtual object within a computer-generated reality (CGR) environment; and

in response to the detected movement of the input mechanism:

in accordance with a determination that the current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, displaying, via the display, a second representation of the virtual object within the CGR environment, wherein the second representation is different from the first representation; and

in accordance with a determination that the current position of the input mechanism is not within the predetermined distance from the first representation of the virtual object, maintaining display of the first representation and not display of the second representation.

54. A system, comprising:

a display;

means for displaying, via the display, a first representation of a virtual object within a computer-generated reality (CGR) environment; and

means for, in response to the detected movement of the input mechanism:

in accordance with a determination that the current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, displaying, via the display, a second representation of the virtual object within the CGR environment, wherein the second representation is different from the first representation; and

in accordance with a determination that the current position of the input mechanism is not within the predetermined distance from the first representation of the virtual object, maintaining display of the first representation and not display of the second representation.

55. A system, comprising:

a display;

one or more processors; and

memory storing one or more programs configured for execution by the one or more processors, the one or more programs including instructions for performing the method of any of claims 36-51.

56. A non-transitory computer-readable storage medium storing one or more programs configured for execution by one or more processors of a wearable electronic device including a display, the one or more programs including instructions for performing the method of any of claims 36-51.

57. A system, comprising:

a display;

apparatus for performing the method of any of claims 36 to 51.

Images