JP2023548795A - Eye movement-based extended reality environment interaction method and system - Google Patents

Abstract

A system and method for extended reality environment interaction is described. An extended reality environment including an object is generated for display, and a first sensor detects that gaze is shifted from a first portion of the extended reality environment to a second portion of the extended reality environment. The object is excluded from a first portion of the extended reality environment and included within a second portion of the extended reality environment. An indicator of a deviation in gaze is generated for display within the extended reality environment in response to detecting the gaze deviation, and a voice command is activated to the second sensor while the indicator is in the vicinity of the object. detected by. In response to detecting the voice command, an action corresponding to the voice command may be performed.

Description

The present disclosure relates to improved extended reality environment interaction, and in particular, systems and methods for detecting eye movements and performing actions within an extended reality environment based on the detected eye movements are disclosed. .

Advances in media technology have led to the development of extended reality (XR) technologies, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies. VR systems may fully immerse the user within a three-dimensional computer-generated environment (e.g., give the user the feeling of being within the environment) or partially immerse the user (e.g., give the user the sense of looking at the environment). give). The environment may include objects or items with which a user can interact. AR systems may provide a modified version of reality, such as augmented information overlaid over real-world objects. MR systems map interactive virtual objects to the real world. Such systems may utilize wearables, such as head-mounted devices, with stereoscopic displays or smart glasses.

XR systems introduce many challenges. For example, the wearable device being used to view the environment may not include an external device (e.g., a lens), so the XR system may can be difficult to detect. As another example, mydriasis and miosis may vary depending on what the user is viewing within the XR environment or the amount of light incident on the user's eyes; There may be no control and therefore monitoring the user's pupils may not be a reliable method for determining the user's gaze or field of view within the XR environment. Even worse, even if the user's field of view is accurately ascertained, it is difficult to determine which object the user wishes to interact with if there are multiple objects within the user's field of view. , can be difficult.

In addition, current approaches to XR suffer from certain drawbacks. In one approach, the user employs hand gestures or a joystick to navigate the XR environment. However, requiring such user input to interact with the XR environment is cumbersome for the user, not to mention detracts from the XR experience (i.e., reminding the user that the XR environment is not real). Or it can be inconvenient. Additionally, current approaches to XR may not allow users to conveniently obtain information about objects within their field of view or to interact within the XR environment.

To overcome these issues, we can identify objects within the user's field of view, detect the user's eyelid movements, and based on such detection, identify objects within the extended reality environment using a modified level of detail. Provided herein are systems and methods for regenerating for display. The systems and methods described herein also include matching the detected eyelid movement and the stored eyelid movement identifier, and performing an action on the object based on such matching. provide. In addition, generate an indicator to reflect the user's gaze shift to a new part of the extended reality environment, including the object, and perform an action when a voice command is received while the indicator is in the vicinity of the object. Systems and methods are provided for. The systems and methods described herein also include generating an opacity-based indicator for display within the extended reality environment in the vicinity of a portion of the extended reality environment that includes the object; and varying the opacity of such indicator based on the identified boundary. Additionally, systems and methods are provided for allowing users to conveniently obtain additional information about items within an extended reality environment.

In some aspects of the present disclosure, an extended reality system generates an extended reality environment for display that includes a first object and receives input from one or more sensors. Based on the received input, the system identifies a first object in the field of view, detects eyelid movement, and in response to detecting the eyelid movement uses a modified level of detail to identify a first object in the field of view. respawn the object for display. Accordingly, eyelid movement can be monitored to overcome the challenges associated with determining which objects in a user's visual field are of interest to the user. In addition, detecting such eyelid movements of the user allows the user to see finer details of objects that appear far away from the user in an extended reality environment, for example, and this , may improve the user experience in extended reality systems, especially for users with visual impairments.

The extended reality environment may include a plurality of objects, including a first object and a second object in the field of view, and the system determines that the detected eyelid movement is associated with the first object. In response, the first object may be regenerated for display using the modified level of detail. If the system determines that the detected eyelid movement is associated with the second object, the system may regenerate the second object for display using the modified level of detail. . The first object may be in one of the foreground or background within the field of view within the extended reality environment, and the second object may be within the other of the foreground or background within the field of view within the extended reality environment. You can.

In some embodiments, regenerating the first object for display using the modified level of detail includes presenting the object at a higher resolution. Additionally or alternatively, one or more actions may be performed on the first object based on the one or more detected eyelid movements.

In some aspects of the present disclosure, the system calculates distinct virtual distances of the plurality of objects with respect to the user, and identifying the first object within the field of view includes determining whether the first object is within the distinct virtual distances. determining being at the closest virtual distance to the user.

In some embodiments, detecting eyelid movement comprises determining an amount of eyelid movement, and/or detecting eyelid movement comprises determining one or more eyelid levels. including. The system detects that the user is navigating from a first position to a new position within the extended reality environment while the first object remains within the field of view, and based on the user's viewpoint at the new position, An updated version of the first object may be generated for display.

In some aspects of the present disclosure, an extended reality system generates for display an extended reality environment that includes objects, and a table of eyelid movement identifiers and corresponding actions that can be performed on objects in the extended reality environment. is stored in memory. Using the sensor, the system detects eyelid movement and matches the detected eyelid movement to one of the stored eyelid movement identifiers. In response to matching the detected eyelid movement to one of the stored eyelid movement identifiers, the system generates an updated image of the extended reality environment based on the action corresponding to the matched eyelid movement. Generate a version for display. Accordingly, eyelid movement can be monitored to overcome the challenges associated with determining objects within the visual field with which a user desires to interact. In addition, detecting such eyelid movements of the user allows the user to interact with objects that appear far away from the user within the extended reality environment, which is particularly useful for visual impairments. The user experience in an extended reality system may be improved for users with .

The object may be selected from the plurality of objects in the extended reality environment by detecting that the user's gaze is directed to the object. The system displays a subset of eyelid movement identifiers that can be performed on the object to which the user's gaze is directed (e.g., to remind or guide the user regarding the action that a certain eyelid movement causes to be performed). It may be generated for The actions of the plurality of actions may correspond to manipulating the object and/or modifying the appearance of the object. (For example, if the object is a book, the action may be turning the pages of the book, tilting the book, tearing the pages of the book, etc.). The system detects that the user is navigating from a first position to a new position within the extended reality environment while the user's gaze remains on the object, and determines a first position based on the user's viewpoint at the new position. An updated version of the object may be generated for display, the updated version of the object having an altered appearance.

In some embodiments, a user may be associated with a user profile that defines a relationship between eyelid movement identifiers and corresponding actions that can be performed on objects within the extended reality environment. The actions that can be performed on an object may vary based on the type of object. To detect eyelid movement, the system determines whether the eyelids remain closed for a predetermined period of time (e.g., to ensure that the eyelid movement is not an involuntary blink). In response to determining that the eyelids remain closed for a predetermined period of time, the detected eyelid movement may be matched to one of the stored eyelid movement identifiers.

In some aspects of the present disclosure, an extended reality system generates for display an extended reality environment comprising an object, and uses a first sensor to direct gaze from a first portion of the extended reality environment to an extended reality environment. Detecting a shift to a second portion of the reality environment, the object is excluded from the first portion of the extended reality environment and included within the second portion of the extended reality environment. In response to detecting the gaze shift, the system generates an indicator of the shift in gaze for display within the extended reality environment, and by using the second sensor, the system generates an indicator of the shift in gaze. Detect voice commands while in the vicinity. In response to detecting the voice command, the extended reality system performs an action corresponding to the voice command. Therefore, extended reality may be leveraged in combination with audio to improve the user experience. More specifically, users may conveniently use their eyes to navigate extended reality environments (e.g., as a proxy for how a mouse or trackpad is used in conjunction with a desktop, laptop, or mobile device). to perform desired actions in the environment via voice commands when the user's gaze indicator is in the vicinity of an object of interest within the extended reality environment. Good too.

An interactive media guide may be provided on the display, and the actions mentioned above may be instructions related to media assets accessible via the interactive media guide. The voice commands may include commands to identify a media asset and perform an action, and/or to present a new media asset on a display and/or to retrieve content associated with an entity; Objects are associated with entities.

In some embodiments, the extended reality system determines whether the rate of retinal movement exceeds a predetermined value, and in response to determining that the rate of retinal movement exceeds the predetermined value. When converting retinal movement to movement of an indicator on a display, the retinal movement may be normalized. The system detects the voice command while the indicator is in the vicinity of (e.g., overlaps with) the object in response to determining that gaze is directed at the object for at least a predetermined threshold period of time. Good too. The display is presented via a virtual reality head-mounted device.

In some aspects of the present disclosure, an extended reality system generates for display an extended reality environment comprising an object, and by using a sensor, gaze is directed to a first portion of the extended reality environment. The object may be detected to be within the first portion of the extended reality environment. The extended reality system generates a plurality of opacity-based indicators for display within the extended reality environment in the vicinity of the first portion of the extended reality environment to identify the boundaries of the object and to identify the object's identified boundaries. The opacity of at least one of the plurality of opacity-based indicators may be varied based on the boundary. Accordingly, the user may conveniently use his or her eyes to navigate the extended reality environment (e.g., as a proxy for the way a mouse or trackpad is used in conjunction with a desktop, laptop, or mobile device); One may receive real-time confirmation of the location of one's gaze, and the opacity of such real-time gaze indicators may be adjusted to avoid obscuring the user's field of view and degrading the user's experience. adjusted to.

The extended reality system may determine whether at least one of the opacity-based indicators overlaps a boundary of the object and vary the respective opacity of the opacity-based indicators that overlap the boundary. The opacity-based indicators are arrows directed towards the object. The extended reality system detects whether the gaze is shifted to a second portion of the extended reality environment by using the sensor and determines that the gaze is shifted to the second portion. In response to the display, a plurality of opacity-based indicators may be overlaid proximate the second portion of the display.

In some embodiments, the individual opacity is varied based on distance from the object. For example, the individual opacity of the indicator may increase as the distance between the indicator and the object decreases (e.g., to emphasize an object that the user is gazing at) or may increase as the distance between the indicator and the object decreases (e.g., to emphasize the object that the user is gazing at). may be increased as the distance between the indicator and the object decreases (to avoid obscuring the object).

In some embodiments, an interactive media guide may be provided on the display, and an action related to a media asset accessible via the interactive media guide is detected, at least in part. Received based on gaze. Such displays may be presented via a virtual reality head-mounted device or without the use of a virtual reality head-mounted device.

In some aspects of the present disclosure, an extended reality system generates an extended reality environment for display that includes an object, detects eye movements, and determines, based on the detection, that the object is within the field of view for at least a predetermined period of time. Determine whether there is. In response to determining that the object is within the field of view for at least a predetermined period of time, the system generates one or more items related to the object for display within the extended reality environment. Therefore, information regarding the object of interest may be conveniently displayed to the user based on detecting the user's eye movements related to the object of interest.

The one or more items associated with the object may comprise textual information, images, video, or any combination thereof. The system further determines that at least a second predetermined period of time has elapsed from the beginning of display of the one or more items without the object being within the field of view for the first predetermined period of time. , may stop displaying one or more items in response to such a determination. Extended reality environments may be presented via virtual reality head-mounted devices. In some embodiments, detecting eye movement includes monitoring eyelid movement or monitoring gaze.

The system determines that the object is in the field of view for a predetermined period of time in response to determining that the field of view is continuously (or discontinuously) over the object for a predetermined period of time during a virtual reality session. You may decide whether or not.

In some embodiments, the system determines that the new object has been in view for at least a predetermined period of time and, in response to such determination, selects one or more items associated with the new object. While being generated for display within the extended reality environment, one or more items associated with the object may continue to be generated for display within the extended reality environment.

The above and other objects and advantages of the present disclosure will become apparent upon consideration of the following detailed description, considered in conjunction with the accompanying drawings, in which like reference characters refer to like parts throughout. Probably.

FIG. 1 illustrates an example of regenerating objects for display within an extended reality environment in accordance with some embodiments of the present disclosure.

FIG. 2 illustrates an example of regenerating objects for display within an extended reality environment in accordance with some embodiments of the present disclosure.

FIG. 3 illustrates an example of performing actions on objects in an extended reality environment, according to some embodiments of the present disclosure.

4A-4B illustrate an example of receiving voice commands while an indicator is in proximity to an object in an extended reality environment, according to some embodiments of the present disclosure.

FIG. 5 illustrates an example of receiving voice commands while an indicator is in proximity to an object in an extended reality environment, according to some embodiments of the present disclosure.

FIG. 6 illustrates an example of receiving voice commands while an indicator is in proximity to an object in an extended reality environment, according to some embodiments of the present disclosure.

FIG. 7 illustrates an example of presenting information related to items in an extended reality environment, according to some embodiments of the present disclosure.

FIG. 8 is a block diagram of an exemplary device in an extended reality system, according to some embodiments of the present disclosure.

FIG. 9 is a block diagram of an illustrative extended reality system, according to some embodiments of the present disclosure.

FIG. 10 is a flowchart of a detailed illustrative process for regenerating objects for display within an extended reality environment, according to some embodiments of the present disclosure.

FIG. 11 is a flowchart of a detailed illustrative process for regenerating objects for display within an extended reality environment, according to some embodiments of the present disclosure.

FIG. 12 is a flowchart of a detailed illustrative process for performing actions on objects in an extended reality environment, according to some embodiments of the present disclosure.

FIG. 13 is a flowchart of a detailed illustrative process for receiving voice commands while an indicator is in proximity to an object in an extended reality environment, according to some embodiments of the present disclosure.

FIG. 14 is a flowchart of a detailed illustrative process for varying the opacity of an indicator in an extended reality environment, according to some embodiments of the present disclosure.

FIG. 15 is a flowchart of a detailed illustrative process for presenting additional information related to items within an extended reality environment, according to some embodiments of the present disclosure.

DETAILED DESCRIPTION FIG. 1 illustrates an example process for regenerating objects for display within an extended reality (XR) environment, according to some embodiments of the present disclosure. A head-mounted display 102 may generate a three-dimensional XR environment 100 for projecting images and immersing the user therein. A user may be fully or partially immersed within the XR environment 100, and such environment may be completely virtual. Head-mounted display 102 may alternatively be a wearable device (eg, smart glasses) or a computer or mobile device equipped with a camera and an XR application to facilitate the generation of environment 100. Environment 100 may alternatively be an augmented reality (AR) environment in which real-world objects are supplemented with computer-generated objects or information, or in which, for example, virtual objects interact with the real world or the real world differs. It may also be mixed reality (MR), in which virtual objects are connected to virtual objects. In some embodiments, the user's field of view or perspective of the environment 100 changes as the user moves his or her head, and other features (e.g., audio) are suitably modified to reflect the physical world. simulate. Environment 100 may be used for entertainment purposes (e.g., video games, movies, videos, sports, etc.), communication (e.g., social media), educational purposes (e.g., virtual classrooms), vocational purposes (e.g., training simulations), medical purposes, etc. It may be for the purpose.

The XR system may identify one or more objects within the user's field of view. A field of view is a portion of the XR environment 100 (eg, an angle in a 360 degree spherical environment) that is presented to the user at a given time by the display 102. In the case of a VR device, the field of view may comprise a pair of 2D images to generate stereoscopic vision; in the case of an AR device (e.g., smart glasses), the field of view may comprise a pair of 2D images to generate stereoscopic vision, and in the case of an AR device (e.g., smart glasses), the field of view may comprise a pair of 2D images to generate stereoscopic vision; It may comprise a 3D or 2D image that may include a mixture of virtual objects (e.g., for smart glasses, a photo captured using a camera and content added by the smart glasses) overlaid on top of the real object. If the XR environment has a single degree of freedom, e.g. 360 degrees of rotation, then any field of view can be defined by edge angular coordinates (e.g. +135 degrees, +225 degrees) or by a single degree of field combined with a known angular opening of the field. It may be defined either by one angular coordinate (eg -55 degrees). If the XR environment has 6 degrees of freedom, ie, 3 rotations of 360 degrees and 3 spatial positions, any field of view may be defined by 3 angular coordinates and 3 spatial coordinates. A field of view may thus be understood as the part of the XR environment that is displayed when the user is at a particular location within the XR environment and orienting the XR set in a particular direction.

An XR system (e.g., system 900 of FIG. 9) may generate a data structure related to a field of view that includes object identifiers associated with virtual objects within the field of view; It may also include coordinates representing the location. The system may determine the current field of view and identify objects within the user's field of view based on data structures and/or images captured by the XR device 102. As shown in the example of FIG. 1, the user's detected field of view within environment 100 includes object 104, depicted as a car, but one skilled in the art will appreciate that any number or combination of different types of objects may be included. , may be included within environment 100. The XR system may generate the object 104 for display at a default level of detail (eg, a default resolution or number of pixels displayed, or a default size or appearance). For example, objects in environment 100 may be presented at 4K resolution (3,840×2,160) or any other suitable resolution by default. The resolution of objects within environment 100 may be the same or vary from user to eye. In some embodiments, the level of detail may refer to the size or appearance of an object; for example, the object may be generated at a default size or color.

In some embodiments, in response to determining the one or more objects 104 within the user's field of view, the XR system may cause the user to perform certain eyelid movements regarding how to cause certain actions to be performed on the objects 104. An identifier 106 (e.g., "blink once to modify vehicle details") may be generated for display, which may indicate or otherwise provide guidance. In some embodiments, the XR system stores a table (e.g., which may be stored in storage device 808 of FIG. , table 316 in FIG. 3). For example, the table may additionally include an identifier (e.g., blinking twice) that may correspond to increasing or decreasing the size of the object displayed to the user in response to detecting the indicated eyelid movement. You may memorize it.

Once the object of interest within the field of view is identified, the XR system may detect the user's eyelid movements 108 by using a sensor (eg, a camera). In some embodiments, the XR system adjusts the eyelid movements to a predetermined range to avoid performing actions based on involuntary blinks (e.g., if such actions are not desired by the user). It may be detected whether a period of time (eg, 0.5 seconds or 1 second) is exceeded. In response to detecting a user's eyelid movement 108 (e.g., a single blink corresponding to an action that modifies details of the object of interest), the XR is provided to the user via the head-mounted display 102. , and regenerates object 104 for display. For example, object 104 is presented to the user at a higher resolution (e.g., 8K resolution, 7,680 x 4,320) than is initially provided (e.g., 4K resolution, 3,840 x 2,160). Good too. In some embodiments, the detected eyelid movement may cause the XR system to modify details of the object 104 in a different manner (e.g., increasing or decreasing the size of the object compared to the initial presentation of the object). , changing the color or texture of an object compared to its initial appearance, etc.).

In some embodiments, detecting eyelid movement comprises determining an amount of eyelid movement, or detecting eyelid movement comprises determining one or more eyelid levels. . For example, an XR system may use a sensor (e.g., a camera) to detect the amount of eyelid movement, where the detected amount and a threshold amount of movement over a predetermined period (e.g., detected over a 3 second period) The five eyelid movements performed) may be compared and the image may be modified or selected when the detected amount of eyelid movement exceeds a threshold amount of movement over a predetermined period of time. As another example, an XR system may use a sensor (e.g., a camera) to detect one or more eyelid levels (e.g., distinctly different eyelid levels) over a predetermined period of time, and detect may be compared to a threshold number of eyelid levels over a predetermined time period (e.g., five distinct eyelid levels detected over a 3 second period), such that the detected number of eyelid levels over a predetermined time period The image may be modified or selected when a threshold number of eye levels over the range is exceeded.

FIG. 2 shows an example process in which multiple objects are detected within a user's field of view. The XR system may detect that objects 204 and 208 are each within the field of view of a user who is viewing XR environment 200 via head-mounted display 202. As shown in the environment 200 in the top portion of FIG. 2, objects 204, 208 that are detected as being within the user's field of view are initially presented at a default level of detail (e.g., default resolution or default size). Good too. In response to detecting eyelid movement 210 (e.g., corresponding to "blink once to correct car details" indicated in identifier 206), the XR system uses a modified level of detail to Objects 204 within the field of view may be regenerated for display (eg, improving the resolution of objects 204). On the other hand, in response to detecting eyelid movement 212 (e.g., corresponding to "blink twice to correct airplane details" as indicated in identifier 206), the XR system detects a modified level of detail. may be used to regenerate objects 208 within the field of view for display (eg, to improve the resolution of objects 208).

In some embodiments, detecting further eyelid movement 212 may subsequently reverse the modifications performed in response to detecting eyelid movement 210 (e.g., object 204 initially The object 208 is presented using the modified details (while the user may be returned to the default resolution presented to the user). Alternatively, the object may be maintained in a modified state throughout the XR session and/or in future sessions. In some embodiments, detecting that the eyelid movement is being performed again may reverse the action (e.g., detecting the eyelid movement 210 a second time may cause the object 204 to be ). In some embodiments, of a plurality of objects that may be within the user's field of view, one such object (e.g., object 204) may be in the foreground of the display of the XR environment 200; The other of such objects may be in the background of a display (eg, object 208) within XR environment 200. Additionally, one or more actions may be performed on modified objects within the user's field of view (e.g., a particular eyelid movement may open the door of a car object 204, or an airplane (may correspond to interacting with object 208). The most recently modified object in the field of view may be a "selected" object so that actions can be performed on such object. Such aspects allow objects that are far from the user or otherwise too small to be seen in detail to be displayed in modified detail to allow the user to clearly see the objects. may be allowed to be regenerated for.

In some aspects of the present disclosure, the XR system may detect the user's eyelid movements that are consistent with objects in the user's field of view, calculate the relative degree to which the eyelids are closed, and calculate the relative degree to which the eyelids are closed. The object to focus on may be determined first. In some embodiments, when a user enters an XR environment, the XR system sets a default field of view, detects the number of objects in the environment and/or field of view, and assigns each detected object to the user. Individual virtual distances or focal lengths may be calculated. Objects may be at different virtual distances from the user. In some embodiments, identifying the object within the field of view includes determining that the object is at the closest virtual distance to the user at the respective virtual distance or focal distance. The virtual distance may be, for example, the perceived distance at which an object within the XR environment is located from the user, and may be calculated based on the coordinates of the object within the XR environment. The user's eyelid level may be calculated based, at least in part, on such virtual distance, and in response to detecting a change in the eyelid level, an object at a virtual distance closest to the user is detected. , may be selected as the object of interest, to which modifications may be performed.

In some embodiments, the XR system may detect movement of the user around the XR environment 200 and detect changes in the user's field of view as the user moves around. In such a situation, the user's field of view may be reset to determine the number of objects in the user's new field of view. On the other hand, if the XR system detects the user's movement around the XR environment 200, but the user's gaze remains fixed on a particular object, the display 200 detects the user's movement and the user's gaze remains fixed on a particular object. Such objects from consistent, varied viewpoints may be generated for display to maintain a simulated environment. In some embodiments, any changes in eyelid level detected by the virtual reality system may be used to determine the object in the user's field of view whose details should be modified. An XR system may track a user's movement within an XR environment by using sensors (eg, gyroscopes, accelerometers, cameras, etc. in combination with control circuitry).

FIG. 3 illustrates an example process for performing actions on objects in an XR environment, according to some embodiments of the present disclosure. Head-mounted display 302 may generate an XR environment 300 for display, including objects 304, 310, 312, 314. Four objects (a book object 304, a lamp object 310, a desk object 312, and a chair object 314) are shown in the environment 300, but any number and type of objects may be generated for display. I hope you understand what you get. The XR system (e.g., system 900 of FIG. 9) stores (e.g., in storage 808 of FIG. 8) a table 316 of eyelid movement identifiers and corresponding actions that can be performed on objects within the XR environment. Good too. For example, table 316 promotes the desired actions: "Blink once to turn a page in a book, Blink twice to tilt a book, Blink three times to tear a page from a book." A plurality of associations may be stored, including an association for the book object 304 as indicated by the identifier 306, that may be displayed to the user for the book object. Any number of actions and any type of identifier may be stored in table 316, and the actions may depend on the type of object (e.g., table 316 may turn on the virtual light bulb associated with lamp object 310). It should be understood that the identifier associated with the action may be stored). If the environment includes multiple objects, the object may be selected from the plurality by detecting (eg, using a sensor) that the user's gaze is directed toward the object.

XR systems detect a user's eyelid movements by using a sensor (eg, a camera). The system may, for example, analyze sensor output and compare such output to stored identifiers (e.g., predetermined number of blinks, blink pattern, amount of eyelid movement, eyelid level, etc.). It may be determined whether the detected eyelid movement matches any of the identifiers in table 316. In some embodiments, stored identifiers may include eyelid movements in combination with voice commands or other inputs. In some embodiments, the XR system adjusts the eyelid movements to a predetermined range to avoid performing actions based on involuntary blinks (e.g., if such actions are not desired by the user). It may be detected whether the period (for example, 0.5 seconds) is exceeded. The system may detect the user's eyelid movement based on the degree of opening and closing of the user's eyelids over time.

In response to matching the detected eyelid movement to one of the stored eyelid movement identifiers, the XR system generates an updated version of the XR environment 300 based on the action corresponding to the matched eyelid movement. Generate a version for display. In the example of FIG. 3, the system detects (318) eyelid movement associated with turning the pages of a book compared to a book object 304 in the upper portion of the environment 300, depicting a closed book. , performs such an action, as shown in the environment 300 in the lower part of FIG. 3, which depicts a turned page of a book object 304. Those skilled in the art will appreciate that objects within environment 300 may be manipulated in various ways, such as chair object 314 being moved adjacent to different parts of desk object 312, or modified in various ways, e.g. It will be appreciated that the cushion may be removed from object 314.

In some embodiments, a subset of identifiers 306 suitable for a selected object of interest may be used for the user's convenience in determining available actions to be performed based on a particular eyelid movement. It may be displayed to the user. In some embodiments, the XR system stores one or more user profiles that define relationships between eyelid movement identifiers and corresponding actions that can be performed on objects within the XR environment 300. You can. For example, the user profile may include, for example, actions tailored to the user's preferences, the user's favorite actions, the user's most recently performed actions, the user's most commonly performed actions, etc., which may be displayed in association with the identifier 306 for the user's convenience. It may also include actions performed by users, purchase actions by users, and the like.

In some embodiments, the XR system may detect movement of the user around the XR environment 200 and may detect changes in the user's gaze as the user moves around. In such situations, the system may select a new object of interest. Alternatively, the system detects that the user is navigating from a first position to a new position within the XR environment while the user's gaze remains on the object, and in response to such determination, Based on the user's perspective, an updated version of the object may be generated for display (eg, altering the size or angle of the object presented to the user). The updated version of the object may include an object with an altered appearance (e.g., a book with torn pages in case the user previously performed an eyelid movement associated with such an action in table 316). It may also include presenting the information to the user.

In some embodiments, the aspects discussed in Figures 1-2 may be combined with the embodiment of Figure 3 (e.g., in a single user session within the XR environment, objects within the may be regenerated in more detail and various actions may be performed on such objects).

4A-4B illustrate an example of receiving voice commands while an indicator is in proximity to an object in an XR environment, according to some embodiments of the present disclosure. An XR system (eg, system 900 of FIG. 9) may generate an XR environment 400 for display to a user via a head-mounted display 402. In some embodiments, XR environment 400 may include an interactive media guide application to facilitate selection and consumption of media content. XR environment 400 may include one or more objects 408, 410 that may correspond to identifiers related to selectable media content. The system uses a sensor (e.g., a camera) to shift the user's gaze from one part of the XR environment (e.g., in the vicinity of object 408 (FIG. 4A)) to another part of the XR environment (e.g., in the vicinity of object 408 (FIG. 4A)). (FIG. 4B)) in the vicinity of the object 410 is detected. 4A-4B are exemplary and the user's gaze is shifted from a portion of the XR environment 400 to another portion of the XR environment 400 that may not contain any objects or may contain multiple objects. I want you to understand that I am

In response to detecting the gaze shift, the XR system may generate for display an indicator 406 in the XR environment 400 that indicates the shift in gaze. For example, indicator 406 in FIG. 4A reflects that the user's gaze is on an object 408 (e.g., an identifier for the movie "The Dark Knight"), and in FIG. 4B, indicator 406 reflects that the user's gaze is on object 408 410 (for example, an identifier related to the movie "American Psycho"). In some embodiments, a single indicator may be generated for display, or alternatively, multiple indicators may be generated for display. In some embodiments, the indicator may vary in translucency based on proximity to the object of interest. In the example of FIGS. 4A-4B, the indicator 406 is shown as an arrow directed at the object of interest, but the indicator 406 may be any suitable indicia or It will be understood by those skilled in the art that markings may be provided. For example, the indicator may be a certain color or shape to emphasize the object of interest, an image or pictogram of an eyeball, an enlargement of the object in the vicinity of the indicator, an animation of the object in the vicinity of the indicator, etc.

By using a sensor (e.g., a microphone), the system determines when indicator 406 is in the vicinity of object 410 (e.g., when the indicator overlaps or is otherwise within a predetermined distance of the object of interest). A voice command may be detected in between. The XR system processes the voice command and takes an action (e.g., provided there is a match between the object contained within the voice command and the object to which the user's gaze is directed, as indicated by indicator 406). may be executed. For example, in response to receiving the voice command 404 in the example of FIG. 4A, the system may begin presenting the media asset "The Dark Knight" associated with the object 408, and in response to receiving the voice command 404 in the example of FIG. 4B. In response to receiving voice command 405, the system may begin presenting the media asset "American Psycho" associated with object 410. In some embodiments, if the user determines that the indicator does not accurately reflect the user's gaze, the system requests that the system recalibrate the user's gaze, and/or Alternatively, a suitable voice command may be received from the user indicating the portion of the display that the user believes to be gazing at.

In some embodiments, the user's gaze is detected based on retinal movement of the eye (tracked by a sensor, e.g., a camera that measures the reflection of a light source from the retina, eye tracking glasses, screen-based eye tracking). be done. The user's retinal movement may be plotted or translated into a display of the XR environment as movement of the indicator 406 on the display. In some aspects of the present disclosure, the system may determine whether the rate of retinal movement exceeds a predetermined value and, in response to such determination, display the retinal movement on a display of the XR environment. Normalization is performed when converting the movement of the indicator 406 into the movement of the indicator 406 . For example, if the rate of gaze shift exceeds a predetermined value, normalization may be used to slow down the movement of indicator 406 on the display (e.g., to make it easier for the user to track movement of indicator 406 on the display). ) may be implemented. The entire cluster of indicators may be moved to such new portion of the display.

In some embodiments, the system includes an electronic voice recognition (or voice assistance) device (e.g., television, computer, voice assistant) that responds to user voice commands, and the voice input is an audio or digital signal (or or digital input). The system may implement natural language understanding (NLU) techniques and may include natural language understanding circuitry and/or speech-to-text circuitry to transcribe voice commands into text and parse voice commands. However, keywords may be identified and extracted from voice input. The system may compare the extracted keywords and metadata associated with the object of interest and determine whether a match exists, eg, whether to execute a voice command. In some embodiments, if the received voice command does not match an object in the vicinity of the indicator, the system notifies the user of the mismatch and refrains from performing the associated action or performs the associated action regarding the new voice command. May prompt the user.

In some embodiments, the voice commands include identifying a media asset and commands to perform an action (e.g., play, fast forward, rewind, etc.) or to present a new media asset on the display (e.g., other media Contains instructions for scrolling through assets (or navigating to a new page of media assets in a carousel). In some aspects of the present disclosure, determining that the indicator is in the vicinity of the object includes determining that the user's gaze has been directed toward the object for at least a predetermined threshold period of time (e.g., 5 seconds). including.

FIG. 5 illustrates an example of receiving voice commands while an indicator is in proximity to an object in an XR environment, according to some embodiments of the present disclosure. An XR system (e.g., system 900 of FIG. 9) includes a head-mounted display 502 and uses the head-mounted display 502 to generate an XR environment 500, including objects 508, 510, for display. Good too. As shown in the upper environment 500 of FIG. 5, the system uses a sensor (e.g., a camera) to direct the user's gaze to a portion of the XR environment (e.g., including an object 508). may be detected. The system may generate a plurality of opacity-based indicators 506 for display within the XR environment 500 near the portion of the XR environment 500 that includes the object 508. In the example of FIG. 5, indicator 506 is shown as an arrow directed at the object of interest, but indicator 506 may include any suitable indicia or markings to emphasize or prominently display the object with which it is associated to the user. It will be understood by those skilled in the art that this may be provided. For example, the indicator may be a certain color or shape to emphasize the object of interest, an image or pictogram of an eyeball, an enlargement of the object in the vicinity of the indicator, an animation of the object in the vicinity of the indicator, etc.

The system identifies boundaries (e.g., edges, shape contours, ends) of object 508, such as by edge detection techniques, reading coordinates of object 508, analyzing pixel values of an area surrounding object 508, etc. You may. Based on the identified boundaries of the object, the XR system may vary the opacity of at least one of the plurality of opacity-based indicators 506. In some embodiments, the system may determine whether at least one of the plurality of opacity-based indicators overlaps or is within a predetermined distance of a boundary of the object; In response to such a determination, the individual opacity of one or more indicators 506 that overlap the boundary of object 508 may be varied. For example, the system may compare the coordinates of the object of interest and the coordinates of the indicator 506 in the XR system 900 of FIG. In some embodiments, the system determines that the user's gaze shifts from a portion of the display (e.g., containing object 508) to a portion of the display containing another object (e.g., object 510). If so, the system overlays a plurality of transparency-based indicators near the portion of the display that includes object 510. The entire cluster of indicators may be moved to such new portion of the display.

The system may vary the individual opacity based on distance from the object. As shown in the example of FIG. 5, the individual opacity of indicator 506 may increase as the distance between indicator 506 and object 508 decreases. This may be desirable to emphasize to the user the portion of the display to which the user's gaze is directed. Alternatively, as shown in the example of FIG. 6, the individual opacity of the indicators 606 may decrease as the distance between the individual indicators 606 and the object 608 decreases. This may be desirable to minimize obscuring portions of the object of interest. In some embodiments, the system may determine whether any of the indicators may not be of interest, overlap another object, or otherwise be in its vicinity; In such situations, the indicator may be semi-concentrated to avoid either obscuring parts of such objects that are not of interest or falsely indicating to the user that such objects are of interest. It may be set to be transparent.

As shown in the examples of FIGS. 5 and 6, respectively, the XR environments 500, 600 may include interactive media guides. In response to receiving a voice command to perform an action associated with an object in the vicinity of the indicator 506, 606, the system may perform the action (e.g., an actor associated with the object 508, 608). An identifier for the movie having metadata associated with "Christian Bale" is generated for presentation). Although the examples of FIGS. 5-6 show objects related to the "actor" category, various other categories may be generated and viewed for display (e.g., genre, cast, director, etc.) . In some embodiments, the environment 500, 600 may be presented with or without an XR head-mounted device. For example, the XR environment may be presented as fully immersive VR or in AR or MR overlaid on a television screen as shown in FIGS. 5-6.

In some embodiments, the features of the embodiments of FIGS. 1-4 may be used in conjunction with the features of the embodiments of FIGS. 5-6. For example, an indicator (e.g., indicator 106 in FIG. 1) may be associated with an action that can be performed (e.g., in table 316 of FIG. 3) to guide the user in performing eyelid movements or gaze. It may be displayed to the user. The indicators of FIGS. 5 and 6 may also be used in connection with other environments (eg, video games, sports, communications, social media, etc.).

FIG. 7 illustrates an example of presenting additional information related to items within an XR environment, according to some embodiments of the present disclosure. Head-mounted display 702 presents XR environment 700 to the user. XR environment 700 may include one or more objects, such as a washing machine object 702, a refrigerator object 704, a wall object 708, etc. In some embodiments, XR environment 700 includes an identifier 706 that may indicate to the user actions that may be taken within environment 700. Although environment 700 depicts a room within a building (e.g., a house or an apartment), environment 700 may include any virtual or augmented environment, such as a mountain range, a scenic location, a video game being played by a user, etc. You can.

The system detects the user's eye movements (e.g., via a sensor) and determines whether objects (e.g., washing machine object 702, refrigerator object 704, and/or wall object 708) Determine whether the user is within the user's field of view for at least a predetermined period 710 (eg, 5 seconds). In response to determining that an object (e.g., a wall object) is within the field of view for at least a predetermined period of time, the system displays one or more items 714 associated with the object 708 within the XR environment. generated for display. One or more items 714 associated with an object may comprise textual information, images, videos, or any combination thereof, including item information (e.g., stored in storage 808 of FIG. 8). It may also be read from database 712. For example, item 714 indicates the thickness of wall object 708 and such additional information is provided when a user gazes at wall object 708 via head-mounted display 702 within XR environment 700. Good too. In some embodiments, detecting eye movement includes monitoring eyelid movement. In some aspects of the present disclosure, generating one or more items 714 related to object 708 for display within an XR environment includes object 708 being within view for at least a predetermined period of time. and being within the user's gaze. In some embodiments, the system allows modifications for a predetermined period of time (eg, by the user or recommended by the system based on the user's historical data).

As another example, the environment 700 may be a mountain range, and based on detected eye movements of the user (e.g., detected via a sensor), the environment 700 may be such that an object (e.g., a mountain) In response to determining that the mountain object is within the user's field of view, information regarding the mountain object may be generated for display to the user (e.g., mountain name, mountain elevation, mountain location, etc.) .

In some aspects of the present disclosure, determining whether the object is within the field of view for a predetermined period includes continuously It may include the step of determining that. For example, if the system detects that the user's visual field has shifted during the timer's countdown to a predetermined period, the countdown may be reset. Alternatively, determining whether the object is within the field of view for a predetermined period of time determines that the field of view is discontinuously over the object for a predetermined period of time (e.g., 5 seconds) during the XR session. May include steps. For example, if the system detects that the user's visual field has shifted during a timer countdown to a predetermined period, the countdown is paused and the system detects that the user's gaze has returned to the object. It may be restarted depending on the situation.

In some embodiments, the system may determine whether the new object is within view for at least a predetermined amount of time. For example, in response to the system making such a determination, the system generates one or more items related to the new object for display within the XR environment, while One or more items (eg, object 708 and item 714) may continue to be generated for display within the XR environment. This allows the user to view additional information about multiple items for at least the duration of the XR session. In some embodiments, additional information items may be maintained in subsequent user sessions within the XR environment.

In some embodiments, the system allows at least a predetermined period of time (e.g., 10 It may be determined that a period of time (seconds) has elapsed. For example, if at least a predetermined period of time (e.g., 10 seconds) has elapsed since item 714 was provided to the user, and object 708 associated with item 714 is not within view during such period; The display of item 714 may be stopped (eg, because the user may no longer be focused on object 708).

In some embodiments, additional inputs (eg, voice commands, hand gestures, touch) may be utilized in combination with detecting the user's eye movements. For example, the system may receive a voice command indicating that the user no longer wishes to view additional information associated with the item of interest, or a voice command to display additional information. may be received, and additional information related to the item of interest (eg, item 708) may be read from the item information database 712.

FIG. 8 is a block diagram of an exemplary device in an XR system, according to some embodiments of the present disclosure. A device 800 in an XR system generates an XR environment that is transmitted via a computer network to an XR device (e.g., devices 102, 202, 302, 402, 502, 602, 702 of FIGS. 1-7, respectively). , displaying, and/or managing one or more servers. Device 800 may include control circuitry 804 and/or I/O paths 810. Control circuitry 804 may be based on any suitable processing circuitry, such as processing circuitry 806. As referred to herein, processing circuitry may include one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc. ), and may include a multi-core processor (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores) or a supercomputer.

In some embodiments, processing circuitry 806 includes multiple separate processors or processing units, such as multiple processing units of the same type (e.g., two Intel Core i7 processors) or multiple different processors (e.g., Intel Core i5 processors and Intel Core i7 processors). I/O path 810 includes processing circuitry 806 and storage 808 to transmit device information or other data and/or other content and data via a local area network (LAN) or wide area network (WAN). Control circuitry 804 may also be provided. Control circuitry 804 may be used to send and receive commands, requests, signals (digital and analog), and other suitable data using I/O path 810. I/O path 810 may connect control circuitry 804 (and specifically processing circuitry 806) to one or more communication paths. In some embodiments, storage 808 may include table 316 of FIG. 3 and/or item information database 712 of FIG. 7.

In some embodiments, control circuitry 804 executes instructions stored in memory (eg, storage 808) for performing XR content generation and/or processing functions. Instructions may be stored in either non-volatile memory 814 and/or volatile memory 812 and loaded into processing circuitry 806 at run time. A system for generating and processing XR content (e.g., the systems described with reference to FIGS. 1-7) may be implemented on a user device (e.g., end-user device 920) and/or on a server (e.g., server 900). It may be a standalone application implemented or distributed across multiple devices according to device 800. The system may be implemented as software or a set of executable instructions. Instructions for implementing any of the embodiments discussed herein of XR processing may be stored on a non-transitory computer-readable medium (e.g., a hard drive, random access memory on a DRAM integrated circuit, on a BLURAY disk). read-only memory, etc.) or on a transitory computer-readable medium (e.g., a propagated signal carrying data and/or instructions). For example, instructions according to the processes described herein may be stored in storage 808 and executed by control circuitry 804 of device 800.

Control circuitry 804 may include one or more analog tuners, one or more MPEG-2 decoders or other digital decoding circuitry, a high definition tuner, or any other suitable tuning or video circuitry or the like. It may also include video generation circuitry and tuning circuitry, such as a combination of such circuits. Encoding circuitry (eg, to convert wireless, analog, or digital signals to MPEG signals for storage) may also be included. Control circuitry 804 may also include scaler circuitry to upconvert and downconvert content to the preferred output format of end user devices 920, 925, 930 of FIG. Circuitry 804 may also include digital-to-analog converter circuitry and analog-to-digital converter circuitry for converting between digital and analog signals. Tuning and encoding circuitry may be used to receive, display, play, or record content. Tuning and encoding circuitry may also be used to receive guidance data. The circuitry described herein, including, for example, tuning, video generation, encoding, decoding, encryption, decoding, scaler, and analog/digital circuitry, may be implemented on one or more general purpose or specialized processors. may be implemented using software running on the Multiple tuners may be provided to handle simultaneous tuning functions (eg, viewing and recording functions, functions for incorporating video or other recordings of the physical environment, multiple tuner recording, etc.). If storage 808 is provided as a separate device from device 800, tuning and encoding circuitry (including multiple tuners) may be associated with storage 808. Device 800 may be a central device that communicates with each device (eg, 102, 202, 302, 402, 502, 602, 702). Alternatively, device 800 may correspond to device 104, i.e., each system (e.g., of FIGS. 1-7) may not have its own device 920, but rather a single A central device 800 may perform XR environment generation and processing on a per-system basis.

FIG. 9 shows an illustrative block diagram of an XR content system, according to some embodiments of the present disclosure. At least one XR content server 900 generates XR content such as that described herein. XR content server 900 may transmit content to multiple end user devices, including devices 920, 925, 930, via the Internet 910. End-user XR devices may include, for example, personal computers, mobile devices (e.g., smartphones), and/or XR headsets, goggles, suits, etc. configured to present and enable interaction with the XR environment. May also include wearable devices, including gloves and the like. These devices are configured to allow operators/users to view and interact with multi-user XR content (eg, via a display screen). These devices may provide visual, audio, and haptic feedback, presenting perspective and attention-directing cues, such as those described herein. The end user devices 920, 925, 930 also transmit data to the server 900 via the Internet 910, and such data is transmitted in the direction in which the devices 920, 925, 930 are facing (so that the server 900 , 925, 930), audio signals detected by each device 920, 925, 930, and user input such as selection of an XR object. Server 900 may then generate a view of the XR environment for each device 920, 925, 930.

FIG. 10 is a flowchart of a detailed illustrative process for regenerating objects for display within an XR environment, according to some embodiments of the present disclosure. Note that process 1000 or any steps thereof may be performed on or provided by any of the devices shown in FIGS. 8-9. For example, process 1300 includes computer-executable instructions that may be implemented on network device 800 and/or server 900 and/or devices 920, 925, 930, such as to distribute control of an extended reality environment among multiple devices. network device 800 (e.g., via control circuitry 804), and/or control circuitry of server 900, and/or devices 920, 925, 930, as instructed by one or more programs of It may also be performed by control circuitry. Additionally, one or more steps of process 1000 may be performed in any other process or embodiment (e.g., process 1100 of FIG. 11, process 1200 of FIG. 12, process 1300 of FIG. 13, process 1400 of FIG. 14, It may be incorporated into or combined with one or more steps of process 1500) of FIG. 15.

At 1002, input/output (I/O) circuitry (e.g., I/O circuitry of head-mounted display 102) is connected to a user for entering an XR environment (e.g., a VR, AR, or MR environment). A selection may be received. Such user selections may be received via input (eg, voice commands or touch screen).

At 1004, control circuitry (e.g., control circuitry 804 of FIG. 8 and/or control circuitry of head-mounted display 102 of FIG. 1) selects an object (e.g., An XR environment may be generated for display (eg, via head-mounted device 102 of FIG. 1), including a vehicle object 104 of FIG. 1).

At 1006, I/O circuitry (eg, I/O circuitry of head-mounted display 102) may receive input from a sensor (eg, a camera). The sensor may detect various attributes of the user's eyes (eg, eyelid movement, gaze).

At 1008, control circuitry (e.g., control circuitry 804 of FIG. 8 and/or control circuitry of head-mounted display 102 of FIG. 1) determines whether an object ( For example, the car object 104 of FIG. 1) may be identified. In some embodiments, the control circuitry may generate a data structure related to the field of view that includes an object identifier associated with a virtual object within the field of view, and such data structure includes a position of the field of view within the XR environment. It may also include coordinates representing. Control circuitry may determine the current field of view and identify objects within the user's field of view based on the data structures and/or images captured by the XR device 102.

At 1010, control circuitry may detect eyelid movement (e.g., amount of eyelid movement, blink pattern, eyelid level, etc.) of the user based on received input from a sensor (e.g., a camera). . In some embodiments, such eyelid movements may be distinguished from involuntary blinks to avoid interpreting such involuntary blinks of the user as commands. For example, the system may ignore blink patterns in which the eyes remain closed for less than a predetermined amount of time (eg, 0.5 seconds).

At 1012, control circuitry may regenerate the object (eg, car object 104 of FIG. 1) for display using the modified level of detail in response to detecting the eyelid movement. For example, the control circuitry may regenerate the object for display at a higher resolution in response to detecting a particular blink pattern. In some embodiments, the control circuitry uses the eyelid movement identifier and the possible action to determine whether the detected eyelid movement matches the eyelid movement identifier to modify the level of detail of the object. Reference may also be made to a table (eg, table 316 of FIG. 3) that stores associations between.

FIG. 11 is a flowchart of a detailed illustrative process for regenerating objects for display within an XR environment, according to some embodiments of the present disclosure. For example, process 1100 includes computer-executable instructions that may be implemented on network device 800 and/or server 900 and/or devices 920, 925, 930, such as to distribute control of an extended reality environment among multiple devices. network device 800 (e.g., via control circuitry 804), and/or control circuitry of server 900, and/or devices 920, 925, 930, as instructed by one or more programs of It may also be performed by control circuitry. Additionally, one or more steps of process 1100 may be performed in any other process or embodiment (e.g., process 1000 of FIG. 10, process 1200 of FIG. 12, process 1300 of FIG. 13, process 1400 of FIG. 14, It may be incorporated into or combined with one or more steps of process 1500) of FIG. 15.

At 1102, I/O circuitry (e.g., I/O circuitry of head-mounted display 202) receives a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). Good too. Such user selections may be received via input (eg, voice commands or touch screen).

At 1104, control circuitry (e.g., control circuitry 804 of FIG. 8 and/or control circuitry of head-mounted display 202 of FIG. 2) selects the plurality of objects ( An XR environment may be generated for display (eg, via head-mounted device 202 in FIG. 2), including, for example, car object 204 in FIG. 2 and airplane object 208 in FIG. 2).

At 1106, I/O circuitry (eg, I/O circuitry of head-mounted display 102) may receive input from a sensor (eg, a camera). The sensor may detect various attributes of the user's eyes (eg, eyelid movement, gaze).

At 1108, control circuitry (e.g., control circuitry 804 of FIG. 8 and/or control circuitry of head-mounted display 202 of FIG. 1) determines an object ( For example, a car object 204 in FIG. 2 and an airplane object 208 in FIG. 2 may be identified. In some embodiments, the control circuitry may generate a data structure related to the field of view that includes an object identifier associated with a virtual object within the field of view, and such data structure includes a position of the field of view within the XR environment. It may also include coordinates representing. Control circuitry may determine the current field of view and identify objects within the user's field of view based on the data structures and/or images captured by the XR device 202.

At 1110, control circuitry may detect eyelid movement (e.g., amount of eyelid movement, blink pattern, eyelid level, etc.) of the user based on received input from a sensor (e.g., a camera). . In some embodiments, such eyelid movements may be distinguished from involuntary blinks to avoid interpreting such involuntary blinks of the user as commands. For example, the system may ignore blink patterns in which the eyes remain closed for less than a predetermined amount of time (eg, 0.5 seconds).

At 1112, the control circuitry may determine whether the eyelid movement is associated with the first object (eg, car object 204 of FIG. 2). In the example of FIG. 2, the control circuitry determines that the detected eyelid movement is stored in table 316 of FIG. ) may be determined.

At 1114, in response to determining that the detected eyelid movement matches an eyelid movement identifier for the first object (e.g., car object 204 of FIG. 2), the control circuitry determines the modified level of detail. may be used to regenerate such objects for display. For example, the control circuitry may be initially presented at a higher resolution than initially presented, or in response to detecting a particular eyeblink pattern. Such objects may be regenerated for display at a larger size than the original.

At 1116, if the control circuitry determines that the eyelid movement is not associated with the first object (e.g., the car object 204 of FIG. 2), the control circuitry determines that the eyelid movement is associated with another object (e.g., the 2 airplane object 208). In the example of FIG. 2, the control circuitry determines that the detected eyelid movement is stored in table 316 of FIG. ) may be determined.

At 1118, in response to determining that the detected eyelid movement matches the eyelid movement identifier for the second object (e.g., airplane object 208 of FIG. 2), the control circuitry determines the modified level of detail. may be used to regenerate such objects for display. For example, the control circuitry may be initially presented at a higher resolution than initially presented, or in response to detecting a particular eyeblink pattern. The object may be regenerated for display at a larger size than the one displayed. Thus, a user may selectively shift his gaze or focus within the XR environment by actuating his eyelids.

FIG. 12 is a flowchart of a detailed illustrative process for performing actions on objects in an XR environment, according to some embodiments of the present disclosure. For example, process 1200 includes computer-executable instructions that may be implemented on network device 800 and/or server 900 and/or devices 920, 925, 930, such as to distribute control of an extended reality environment among multiple devices. network device 800 (e.g., via control circuitry 804), and/or control circuitry of server 900, and/or devices 920, 925, 930, as instructed by one or more programs of It may also be performed by control circuitry. Additionally, one or more steps of process 1200 may be performed in any other process or embodiment (e.g., process 1000 of FIG. 10, process 1100 of FIG. 11, process 1300 of FIG. 13, process 1400 of FIG. 14, It may be incorporated into or combined with one or more steps of process 1500) of FIG. 15.

At 1202, I/O circuitry (e.g., I/O circuitry of head-mounted display 302) receives a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). Good too. Such user selections may be received via input (eg, voice commands or touch screen).

At 1204, control circuitry (e.g., control circuitry 804 of FIG. 8 and/or control circuitry of head-mounted display 302 of FIG. 3) selects an object (e.g., An XR environment may be generated for display (eg, via head-mounted device 302 of FIG. 3), including the book object 304 of FIG. 3). In some embodiments, the control circuitry may generate multiple objects (eg, book object 304, lamp object 310, desk 312, chair 314 in FIG. 3).

At 1206, the control circuitry stores a table (e.g., table 316 of FIG. 3) of eyelid movement identifiers and corresponding actions that can be performed on the object in memory (e.g., storage 808 of FIG. 8 and/or The information may be stored in or accessed in memory (storage of head-mounted display 302).

At 1208, the control circuitry may detect eyelid movement of the user (e.g., amount of eyelid movement, blink pattern, eyelid level, etc.) based on received input from a sensor (e.g., a camera). . In some embodiments, such eyelid movements may be distinguished from involuntary blinks to avoid interpreting such involuntary blinks of the user as commands. For example, the system may ignore blink patterns in which the eyes remain closed for less than a predetermined amount of time (eg, 0.5 seconds).

At 1210, control circuitry determines whether the detected eyelid movement matches any of the identifiers in a table (eg, table 316 of FIG. 3). In some embodiments, the control circuitry analyzes the sensor output and associates such output with stored identifiers (e.g., predetermined number of blinks, blink pattern, amount of eyelid movement, eyelid level, etc.) You can also compare. In some embodiments, stored identifiers may include eyelid movements in combination with voice commands or other inputs.

At 1212, control circuitry determines whether the detected eyelid movement matches a stored eyelid movement identifier. In some embodiments, the control circuitry may calculate a match score, and when the calculated match score exceeds a predetermined threshold, the control circuitry may calculate a match between the detected eyelid movement and the stored eyelid movement identifier. It may be determined that a match exists.

At 1214, in response to determining that the detected eyelid movement matches the stored eyelid movement identifier, the control circuitry updates the XR environment based on the action corresponding to the matched eyelid movement. generated version for display. In the example of FIG. 3, the updated version of the XR environment 300 is shown in the lower part of FIG. (In the environment 300 illustrated in the section) includes a book object 304 with a turned page.

FIG. 13 is a flowchart of a detailed illustrative process for receiving voice commands while an indicator is in proximity to an object in an XR environment, according to some embodiments of the present disclosure. For example, process 1300 includes computer-executable instructions that may be implemented on network device 800 and/or server 900 and/or devices 920, 925, 930, such as to distribute control of an extended reality environment among multiple devices. network device 800 (e.g., via control circuitry 804), and/or control circuitry of server 900, and/or devices 920, 925, 930, as instructed by one or more programs of It may also be performed by control circuitry. Additionally, one or more steps of process 1300 may be performed in any other process or embodiment (e.g., process 1000 of FIG. 10, process 1100 of FIG. 11, process 1200 of FIG. 12, process 1400 of FIG. 14, It may be incorporated into or combined with one or more steps of process 1500) of FIG. 15.

At 1302, I/O circuitry (e.g., the I/O circuitry of head-mounted display 402 of FIGS. 4A-4B) connects a user to an XR environment (e.g., a VR, AR, or MR environment). A selection may be received. Such user selections may be received via input (eg, voice commands or touch screen).

At 1304, control circuitry (e.g., control circuitry 804 of FIG. 8 and/or control circuitry of head-mounted display 402 of FIG. 4A) selects an object (e.g., An XR environment may be generated for display (eg, via head-mounted device 402 of FIG. 4), including object 408 of FIG. 4). In some embodiments, control circuitry may generate multiple objects (eg, objects 408 and 410 in FIG. 4).

At 1306, control circuitry may detect gaze of the user. For example, a sensor (e.g., a camera) may be used to track a user's retinal movement, and such retinal movement of the user is reflected in a display of an XR environment (e.g., environment 400 of FIGS. 4A-4B). May be plotted.

At 1308, control circuitry may determine whether the user's gaze has shifted to a second portion of the XR environment that includes an object (eg, object 410 of FIGS. 4A-4B). In some embodiments, the user's gaze may be shifted away from a first portion of the display that does not include any objects. Alternatively, such a first portion of the display may include an object (eg, object 408 of FIGS. 4A-4B).

At 1310, in response to determining that the user's gaze is shifted to such a second portion of the XR environment that includes an object (e.g., object 410 of FIGS. 4A-4B), the control The circuitry may generate an indicator of the shift in gaze (eg, indicator 406 of FIG. 4) for display. For example, such an indicator allows the user to confirm that the system is accurately tracking his or her gaze.

At 1312, in response to determining that the user's gaze is not shifted to such a second portion of the XR environment, the control circuitry determines whether the user's gaze is not shifted or It may be determined that the display is shifted to a different portion of the display, and an indicator (e.g., indicator 406 of FIG. 4) may be generated for display in the portion of the display where the user's gaze is directed. .

At 1314, the I/O circuitry (e.g., the I/O circuitry of head-mounted display 402 of FIGS. 4A-4B) indicates that the indicator (e.g., indicator 406 of FIG. 4) is connected to the object of interest (e.g., The voice command may be received while in the vicinity of object 408 (or object 410 of FIG. 4B). In some embodiments, control circuitry (e.g., control circuitry 804 of FIG. 8 and/or control circuitry of head-mounted display 402 of FIG. 4A) determines whether the voice command is associated with such object of interest. You may decide whether or not. In the example of FIG. 4A, the control circuitry extracts a keyword from the voice command "Play 'The Dark Knight'" and associates the extracted keyword with the object to which the user's gaze is directed (e.g., object 408). ) may be compared to determine whether the voice command is associated with the object of interest.

At 1316, the control circuitry may perform an action corresponding to the voice command associated with the object. For example, in the example of FIG. 4B, object 410, an identifier for the media content item "American Psycho," is proximate indicator 406 to receive voice command 405 ("play "American Psycho"). In response, presentation of such media assets to the user may be initiated.

FIG. 14 is a flowchart of a detailed illustrative process for varying the opacity of an indicator in an XR environment, according to some embodiments of the present disclosure. Note that process 1400 or any steps thereof may be performed on or provided by any of the devices shown in FIGS. 8-9. For example, process 1400 includes computer-executable instructions that may be implemented on network device 800 and/or server 900 and/or devices 920, 925, 930, such as to distribute control of an extended reality environment among multiple devices. network device 800 (e.g., via control circuitry 804), and/or control circuitry of server 900, and/or devices 920, 925, 930, as instructed by one or more programs of It may also be performed by control circuitry. Additionally, one or more steps of process 1400 may be performed in any other process or embodiment (e.g., process 1000 of FIG. 10, process 1100 of FIG. 11, process 1200 of FIG. 12, process 1300 of FIG. 13, It may be incorporated into or combined with one or more steps of process 1500) of FIG. 15.

At 1402, I/O circuitry (e.g., I/O circuitry of head-mounted display 502 of FIG. 5) receives a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). You may receive it. Such user selections may be received via input (eg, voice commands or touch screen).

At 1404, control circuitry (e.g., control circuitry 804 of FIG. 8 and/or control circuitry of head-mounted display 502 of FIG. 5) selects an object (e.g., An XR environment may be generated for display (eg, via head-mounted device 502 of FIG. 5), including object 508 of FIG. 5). In some embodiments, control circuitry may generate multiple objects (eg, objects 508 and 510 in FIG. 5).

At 1406, control circuitry may detect gaze of the user. For example, a sensor (e.g., a camera) may be used to track retinal movement of a user, and such retinal movement of the user may be plotted on a display of an XR environment (e.g., environment 500 of FIG. 5). You can. Control circuitry may determine whether the user's gaze is directed at a portion of the XR environment that includes an object (eg, object 508 in FIG. 5).

At 1410, in response to determining that the user's gaze is not directed to a portion of the XR environment that includes an object (e.g., object 508 of FIG. 5), the control circuitry causes the user's gaze to It may be determined whether the object is directed to different parts of the XR environment, including different objects (eg, object 510 in FIG. 5).

At 1410, in response to determining that the user's gaze is directed to a portion of the XR environment that includes an object (e.g., object 508 of FIG. 5), the control circuitry directs such a portion of the XR environment. An opacity-based indicator (eg, indicator 506 of FIG. 6) may be generated for display within the XR environment in the vicinity of the portion. On the other hand, in response to determining that the user's gaze is directed to different parts of the XR environment, including different objects (e.g., object 510 of FIG. 5), the control circuitry may Opacity-based indicators (eg, indicator 506 of FIG. 6) may be generated in the vicinity of different portions for display within the XR environment.

At 1412, control circuitry may determine the boundaries of the object of interest (eg, object 508 in FIG. 5). For example, the control circuitry may implement edge detection techniques and/or pixel comparisons to determine the boundaries of the object.

At 1414, control circuitry varies the opacity of at least one opacity-based indicator (eg, indicator 506 of FIG. 5) based on the identified boundary of the object. In some embodiments, the opacity of the indicator that overlaps the boundary of the object of interest is varied. For example, the opacity of an indicator that overlaps the boundary of the object of interest may be reduced to avoid obscuring the object of interest (Figure 6), or when the user's gaze is directed toward the object of interest. (Figure 5).

FIG. 15 is a flowchart of a detailed illustrative process for presenting additional information related to an item within an XR environment, according to some embodiments of the present disclosure. Note that process 1500, or any steps thereof, may be performed on or provided by any of the devices shown in FIGS. 8-9. For example, process 1500 includes computer-executable instructions that may be implemented on network device 800 and/or server 900 and/or devices 920, 925, 930, such as to distribute control of an extended reality environment among multiple devices. network device 800 (e.g., via control circuitry 804), and/or control circuitry of server 900, and/or devices 920, 925, 930, as instructed by one or more programs of It may also be performed by control circuitry. Additionally, one or more steps of process 1500 may be performed in any other process or embodiment (e.g., process 1000 of FIG. 10, process 1100 of FIG. 11, process 1200 of FIG. 12, process 1300 of FIG. 13, It may be incorporated into or combined with one or more steps of process 1400) of FIG. 14.

At 1502, I/O circuitry (e.g., I/O circuitry of head-mounted display 702 of FIG. 7) receives a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). You may receive it. Such user selections may be received via input (eg, voice commands or touch screen).

At 1504, control circuitry (e.g., control circuitry 804 of FIG. 8 and/or control circuitry of head-mounted display 702 of FIG. 7) selects an object (e.g., An XR environment may be generated for display (eg, via head-mounted device 702 of FIG. 7), including object 708 of FIG. 7). In some embodiments, control circuitry may generate multiple objects (eg, objects 702, 704, 708 in FIG. 7).

At 1506, control circuitry (e.g., control circuitry 804 of FIG. 8 and/or control circuitry of head-mounted display 702 of FIG. 7) detects eye movements of the user (e.g., using sensors). You may.

At 1508, the control circuitry may determine whether the object is within the user's field of view for at least a predetermined period of time. In some embodiments, the control circuitry may generate a data structure related to the field of view that includes an object identifier associated with a virtual object within the field of view, and such data structure includes a position of the field of view within the VR environment. It may also include coordinates representing. Control circuitry may determine the current field of view and identify objects within the user's field of view based on data structures and/or images captured by head-mounted display 702. In some embodiments, the control circuitry may additionally detect whether the user's gaze is directed at an object within the field of view.

At 1510, in response to determining that the object is within view for at least a predetermined period of time (e.g., 5 seconds), the control circuitry causes the control circuitry to detect the object (e.g., object 708 of FIG. 7). One or more items (eg, item 714 of FIG. 7) related to the XR environment are generated for display within the XR environment. In some embodiments, in determining whether the object is within the field of view for at least the predetermined period of time, the control circuitry determines whether the object is within the user's field of view for continuous (or non-continuous) periods during the XR session. You may also decide whether to stay within the field of view.

The processes discussed above are intended to be illustrative rather than limiting. Those skilled in the art will appreciate that steps of the processes discussed herein may be omitted, modified, combined, and/or rearranged and any additional steps may be added without departing from the scope of the invention. It will be understood that it can be implemented. More generally, the above disclosure is meant to be illustrative rather than restrictive. The following claims are meant to set forth the boundaries as to what this invention includes. Furthermore, the features and limitations described in any one embodiment may apply to any other embodiments herein, and the flowcharts or examples related to one embodiment may be used for any other embodiment in a suitable manner. It should be noted that the embodiments may be combined, performed in a different order, or performed in parallel. Additionally, the systems and methods described herein may be implemented in real time. It is also noted that the systems and/or methods described above may be applied to or used in accordance with other systems and/or methods.
This specification discloses embodiments including, but not limited to:
1. A method for extended reality environment interaction, the method comprising:
generating for display an extended reality environment comprising objects;
detecting, by using the first sensor, that the gaze is shifted from the first portion of the extended reality environment to the second portion of the extended reality environment, the object being in the extended reality environment; a step excluded from the first part of the extended reality environment and included within the second part of the extended reality environment;
responsive to detecting the gaze shift, generating an indicator of the shift in gaze for display within the extended reality environment;
detecting a voice command while the indicator is in the vicinity of the object by using a second sensor;
in response to detecting the voice command, performing an action corresponding to the voice command;
including methods.
2. 2. The method of item 1, wherein an interactive media guide is provided on the display and the actions are instructions related to media assets accessible through the interactive media guide.
3. The gaze is detected based on the retinal movement of the eye, and the method further includes:
3. A method according to item 1 or 2, comprising converting retinal movement into movement of an indicator on a display.
4. determining whether the rate of retinal movement exceeds a predetermined value;
in response to determining that the rate of retinal movement exceeds a predetermined value, normalizing the retinal movement when converting the retinal movement into movement of the indicator on the display;
The method according to item 3, further comprising:
5. 5. The method of any of items 2-4, wherein the voice commands include identification of a media asset and a command to perform an action.
6. 6. The method of any of items 2-5, wherein the voice commands include instructions to present a new media asset on the display.
7. 7. The method of any of items 2-6, wherein the object is associated with an entity and the voice command comprises an instruction to retrieve content associated with the entity.
8. 8. The method of any of items 1-7, wherein detecting the voice command while the indicator is in the vicinity of the object comprises determining that gaze is directed at the object for at least a predetermined threshold period of time. .
9. 9. The method of any of items 1-8, wherein the indicator being in proximity to the object includes the indicator overlapping the object.
10. The method of any of items 1-9, wherein the display is presented via an extended reality head-mounted device.
11. A system for extended reality environment interaction, the system comprising:
display and
generating an extended reality environment for display, comprising an object;
By using the first sensor, it is detected that the gaze is shifted from the first part of the extended reality environment to the second part of the extended reality environment, and the object is moved to the first part of the extended reality environment. included within a second part of the extended reality environment;
responsive to detecting the gaze shift, generating an indicator of the shift in gaze for display within the extended reality environment;
detecting a voice command while the indicator is in the vicinity of the object by using a second sensor;
in response to detecting the voice command, performing an action corresponding to the voice command;
a control circuitry configured to;
A system equipped with.
12. 12. The system of item 11, wherein an interactive media guide is provided on the display and the actions are instructions related to media assets accessible through the interactive media guide.
13. The control circuitry further includes:
Detects gaze based on retinal movement of the eye;
converting retinal movement into movement of an indicator on the display,
The system according to item 11 or 12, configured to.
14. The control circuitry further includes:
determining whether the rate of retinal movement exceeds a predetermined value;
in response to determining that the rate of retinal movement exceeds a predetermined value, normalizing the retinal movement when converting the retinal movement to movement of the indicator on the display;
The system according to item 13, configured to:
15. 15. The system of any of items 12-14, wherein the voice commands include identification of a media asset and a command to perform an action.
16. 16. The system of any of items 12-15, wherein the voice commands include instructions for presenting new media assets on the display.
17. 17. The system of any of items 12-16, wherein the object is associated with an entity and the voice command comprises an instruction to retrieve content associated with the entity.
18. Items 11-17, wherein the control circuitry is further configured to determine that gaze is directed at the object for at least a predetermined threshold period of time upon detecting a voice command while the indicator is in the vicinity of the object. The system described in any of the above.
19. 19. The system of any of items 11-18, wherein the indicator being in proximity to the object includes the indicator overlapping the object.
20. The system of any of items 11-19, wherein the display is presented via an extended reality head-mounted device.
21. A method for extended reality environment interaction, the method comprising:
generating an extended reality environment for display, including the first object;
receiving input from one or more sensors;
identifying a first object within the field of view based on the received input;
detecting eyelid movement based on the received input;
in response to detecting eyelid movement, regenerating the first object for display using a modified level of detail;
including methods.
22. The extended reality environment includes a plurality of objects, the plurality of objects including a first object and a second object;
The step of identifying further includes identifying a second object within the field of view;
Regenerating the first object for display using the modified level of detail is performed in response to determining that the detected eyelid movement is associated with the first object.
The method described in item 21.
23. Item 22, further comprising regenerating the second object for display using a modified level of detail in response to determining that the detected eyelid movement is associated with the second object. The method described in.
24. Item 22, wherein the first object is in one of the foreground or background in the field of view within the extended reality environment, and the second object is in the other of the foreground or background in the field of view in the extended reality environment. or the method described in 23.
25. 25. A method according to any of items 21-24, wherein regenerating the first object for display using a modified level of detail comprises presenting the object at a higher resolution.
26. The method of any of items 21-25, further comprising performing one or more actions on the identified object based on the one or more detected eyelid movements. .
27. further comprising calculating separate virtual distances of the plurality of objects with respect to the user;
identifying the first object within the field of view includes determining that the first object is at the closest virtual distance to the user of the respective virtual distance;
The method described in any of items 22-26.
28. 28. The method of any of items 21-27, wherein detecting eyelid movement comprises determining the amount of eyelid movement.
29. 29. The method of any of items 21-28, wherein detecting eyelid movement comprises determining one or more eyelid levels.
30. detecting that the user is navigating from the first location to a new location within the extended reality environment while the first object remains within the user's field of view;
generating an updated version of the first object for display based on the user's perspective at the new location;
The method according to any of items 21-29, further comprising:
31. A system for extended reality environment interaction, the system comprising:
generating an extended reality environment for display, including the first object;
a control circuitry configured to;
receiving input from one or more sensors;
an input/output (I/O) circuitry configured to;
Equipped with
The control circuitry further includes:
identifying a first object within the field of view based on the received input;
detecting eyelid movement based on the received input;
in response to detecting the eyelid movement, regenerating the first object for display using the modified level of detail;
The system is configured as follows.
32. The extended reality environment includes a plurality of objects, the plurality of objects including a first object and a second object;
The control circuitry is further configured to identify a second object within the field of view;
The control circuitry further regenerates the first object for display using the modified level of detail in response to determining that the detected eyelid movement is associated with the first object. configured to carry out the
The system described in item 31.
33. The control circuitry further includes:
An item configured to regenerate the second object for display using a modified level of detail in response to determining that the detected eyelid movement is associated with the second object. The system according to 32.
34. Item 31, wherein the first object is in one of the foreground or background in the field of view within the extended reality environment, and the second object is in the other of the foreground or background in the field of view in the extended reality environment. or the system described in 33.
35. The control circuitry is further configured in any of items 31-34 to present the object at a higher resolution when regenerating the first object for display using the modified level of detail. The system described.
36. The control circuitry further includes:
36. The system of any of items 31-35, configured to perform one or more actions on the identified object based on the one or more detected eyelid movements.
37. The control circuitry further includes:
calculate the individual virtual distances of multiple objects to the user,
in identifying a first object within the field of view, determining that the first object is at the closest virtual distance to the user of the respective virtual distance;
37. The system of any of items 32-36, configured to.
38. 38. The system of any of items 31-37, wherein the control circuitry is further configured, upon detecting eyelid movement, to determine the amount of eyelid movement.
39. 39. The system of any of items 31-38, wherein the control circuitry is further configured to determine one or more eyelid levels upon detecting eyelid movement.
40. The control circuitry further includes:
detecting that the user is navigating from a first location to a new location within the extended reality environment while the first object remains within the field of view;
generating an updated version of the first object for display based on the user's perspective at the new location;
39. The system of any of items 31-38, configured to:
41. A method for extended reality environment interaction, the method comprising:
generating an extended reality environment for display, including the object;
storing in memory a table of eyelid movement identifiers and corresponding actions that can be performed on objects in the extended reality environment;
detecting eyelid movement by using a sensor;
matching the detected eyelid movement to one of the stored eyelid movement identifiers;
In response to matching the detected eyelid movement to one of the stored eyelid movement identifiers, displaying an updated version of the extended reality environment based on an action corresponding to the matched eyelid movement. and the steps to cause it to occur.
including methods.
42. 42. The method of item 41, further comprising selecting an object from a plurality of objects in the extended reality environment by detecting that the user's gaze is directed at the object.
43. 43. The method of item 42, further comprising generating for display a subset of operable eyelid movement identifiers for objects toward which the user's gaze is directed.
44. The method according to any of items 41-43, wherein the actions of the plurality of actions correspond to manipulating an object.
45. 45. The method of any of items 41-44, wherein the action corresponds to modifying the appearance of the object.
46. 46. The method of any of items 41-45, wherein the user is associated with a user profile that defines a relationship between eyelid movement identifiers and corresponding actions that can be performed on objects in the extended reality environment.
47. detecting that the user is navigating from a first location to a new location within the extended reality environment while the user's gaze remains on the object;
generating an updated version of the first object for display based on the user's perspective at the new location, the updated version of the object having an altered appearance;
The method of items 45-46, further comprising:
48. 48. The method of any of items 41-47, wherein the actions that can be performed on the object vary based on the type of the object.
49. The step of detecting eyelid movement is
49. A method according to any of items 41-48, comprising determining whether the eyelids remain closed for a predetermined period of time.
50. Matching the detected eyelid movement to one of the stored eyelid movement identifiers is performed in response to determining that the eyelid remains closed for a predetermined period of time, item 49 The method described in.
51. A system for extended reality environment interaction, the system comprising:
generating an extended reality environment for display, including objects;
a control circuitry configured to;
storing a table of eyelid movement identifiers and corresponding actions that can be performed on objects in the extended reality environment;
A memory configured as,
Equipped with
The control circuitry further includes:
By using sensors, detect eyelid movement,
matching the detected eyelid movement to one of the stored eyelid movement identifiers;
In response to matching the detected eyelid movement to one of the stored eyelid movement identifiers, displaying an updated version of the extended reality environment based on an action corresponding to the matched eyelid movement. cause to occur,
The system is configured as follows.
52. The control circuitry further includes:
52. The system of item 51, configured to select an object from a plurality of objects in the extended reality environment by detecting that the user's gaze is directed toward the object.
53. The control circuitry further includes:
53. The system of item 51 or 52, configured to generate for display a subset of operable eyelid movement identifiers for objects at which the user's gaze is directed.
54. The system according to any of items 51-53, wherein the action of the plurality of actions corresponds to manipulating an object.
55. 55. The system of any of items 51-54, wherein the action corresponds to modifying the appearance of the object.
56. 56. The system of any of items 51-55, wherein a user is associated with a user profile that defines a relationship between eyelid movement identifiers and corresponding actions that can be performed on objects in the extended reality environment.
57. The control circuitry further includes:
detecting that the user is navigating from a first location to a new location within the extended reality environment while the user's gaze remains on the object;
generating an updated version of the first object for display based on the user's perspective at the new location, the updated version of the first object having an altered appearance;
57. The system according to item 55 or 56, configured to.
58. 58. The system of any of items 51-57, wherein the actions that can be performed on an object vary based on the type of object.
59. The control circuitry further includes: in detecting eyelid movement;
59. The system according to any of items 51-58, configured to determine whether the eyelids remain closed for a predetermined period of time.
60. The control circuitry further performs the step of matching the detected eyelid movement to one of the stored eyelid movement identifiers in response to determining that the eyelids remain closed for a predetermined period of time. 60. The system of item 59, configured to.
61. A method for extended reality environment interaction, the method comprising:
generating for display an extended reality environment comprising objects;
detecting, by using a sensor, that gaze is directed to a first portion of the extended reality environment, the object being contained within the first portion of the extended reality environment;
generating a plurality of opacity-based indicators for display within the extended reality environment in the vicinity of the first portion of the extended reality environment;
identifying boundaries of the object;
varying the opacity of at least one of the plurality of opacity-based indicators based on the identified boundaries of the object;
including methods.
62. further comprising determining whether at least one of the plurality of opacity-based indicators overlaps a boundary of the object;
Varying the opacity of at least one of the plurality of opacity-based indicators based on a boundary of the object includes varying the individual opacity of at least one of the plurality of opacity-based indicators that overlap the boundary. 62. The method of item 61, comprising the step of varying.
63. 63. The method of item 61 or 62, wherein the plurality of opacity-based indicators are arrows directed towards the object.
64. detecting whether gaze is shifted to a second portion of the extended reality environment by using a sensor;
in response to determining that gaze is shifted to the second portion, overlaying a plurality of opacity-based indicators proximate the second portion of the display;
64. The method of any of items 61-63, further comprising:
65. 65. A method according to any of items 61-64, wherein the individual opacity is varied based on distance from the object.
66. 66. The method of item 65, wherein the individual opacity of the indicators increases as the distance between the individual indicators and the object decreases.
67. 67. A method according to item 65 or 66, wherein the individual opacity of the indicators increases as the distance between the individual indicators and the object increases.
68. Item 61, wherein an interactive media guide is provided on the display and an action related to a media asset accessible via the interactive media guide is received based, at least in part, on the detected gaze. -67. The method according to any one of 67.
69. 69. The method of any of items 61-68, wherein the display is presented via an extended reality head-mounted device.
70. 69. The method of any of items 61-69, wherein the display is presented without the use of an extended reality head-mounted device.
71. A system for extended reality environment interaction, the system comprising:
display and
generating an extended reality environment for display, comprising an object;
detecting, by using a sensor, that gaze is directed to a first portion of the extended reality environment, and the object is contained within the first portion of the extended reality environment;
generating a plurality of opacity-based indicators for display within the extended reality environment in the vicinity of the first portion of the extended reality environment;
identify the boundaries of the object,
varying the opacity of at least one of the plurality of opacity-based indicators based on the identified boundary of the object;
a control circuitry configured to;
A system equipped with.
72. The control circuitry further includes:
determining whether at least one of the plurality of opacity-based indicators overlaps a boundary of the object;
varying the opacity of at least one of the plurality of opacity-based indicators based on a boundary of the object, varying the individual opacity of at least one of the plurality of opacity-based indicators that overlap the boundary; let,
72. The system according to item 71, configured to:
73. 73. The system of item 71 or 72, wherein the plurality of opacity-based indicators are arrows directed towards the object.
74. The control circuitry further includes:
detecting whether the gaze is shifted to a second portion of the extended reality environment by using a sensor;
in response to determining that gaze is shifted to the second portion, overlaying a plurality of opacity-based indicators in the vicinity of the second portion of the display;
The system according to any of items 71-73, configured to.
75. 75. The system of any of items 71-74, wherein the individual opacity is varied based on distance from the object.
76. 76. The system of item 75, wherein the individual opacity of the indicators increases as the distance between the individual indicators and the object decreases.
77. 77. The system of item 75 or 76, wherein the individual opacity of the indicators increases as the distance between the individual indicators and the object increases.
78. Item 71, wherein an interactive media guide is provided on the display and an action related to a media asset accessible via the interactive media guide is received based at least in part on the detected gaze. - The system according to any one of 77.
79. 79. The system of any of items 71-78, wherein the display is presented via an extended reality head-mounted device.
80. 79. The system of any of items 71-79, wherein the display is presented without the use of an extended reality head-mounted device.
81. A method for extended reality environment interaction, the method comprising:
generating an extended reality environment for display, including the object;
detecting eye movements;
determining whether the object is within the field of view for at least a predetermined period of time based on the detection;
Responsive to determining that the object is within the field of view for at least a predetermined period of time, generating one or more items related to the object for display within the extended reality environment;
including methods.
82. 82. The method of item 81, wherein the one or more items associated with the object comprise textual information, images, video, or any combination thereof.
83. determining that at least a second predetermined period of time has elapsed since the beginning of the display of the one or more items without the object being within the field of view for the first predetermined period of time;
the one or more items in response to determining that a second predetermined period of time has elapsed from the beginning of display of the one or more items without the object being within the field of view; a step of stopping display of the
83. The method according to item 81 or 82, further comprising:
84. 84. The method of any of items 81-83, wherein detecting eye movement comprises monitoring eyelid movement.
85. The step of generating one or more items related to the object for display within the extended reality environment is responsive to determining that the object is within the field of view and within the gaze for at least a predetermined period of time. The method according to any of items 81-84, which is carried out by
86. Determining whether the object is within the field of view for a predetermined period of time based on the detection includes determining that the field of view is continuously over the object for a predetermined period of time during the extended reality session. The method described in any of items 81-85.
87. Determining whether the object is within the field of view for a predetermined period of time based on the detection includes determining that the field of view is discontinuously over the object for a predetermined period of time during the extended reality session. , the method according to any of items 81-86.
88. determining that the new object is in view for at least a predetermined amount of time;
in response to determining that the new object is in view for at least a predetermined period of time, generating one or more items related to the new object for display within the extended reality environment; continuing to generate one or more items related to for display within the extended reality environment;
88. The method of any of items 81-87, further comprising:
89. 89. A method according to any of items 81-88, further comprising enabling modification of the amount of time for the predetermined time period.
90. 89. The method of any of items 81-89, wherein the extended reality environment is presented via an extended reality head-mounted device.
91. A system for extended reality environment interaction, the system comprising:
display and
generating an extended reality environment for display, including objects;
detect eye movements,
based on the detection, determining whether the object is within the field of view for at least a predetermined period of time;
in response to determining that the object is within the field of view for at least a predetermined period of time, generating one or more items related to the object for display within the extended reality environment;
a control circuitry configured to;
A system equipped with.
92. 92. The system of item 91, wherein the one or more items associated with the object comprise textual information, images, videos, or any combination thereof.
93. The control circuitry further includes:
determining that at least a second predetermined period of time has elapsed since the beginning of the display of the one or more items without the object being within the field of view for the first predetermined period of time;
the one or more items in response to determining that a second predetermined period of time has elapsed from the beginning of display of the one or more items without the object being within the field of view; stop displaying,
93. The system according to item 91 or 92, configured to.
94. 94. The system of any of items 91-93, wherein the control circuitry is further configured to monitor eyelid movement when detecting eye movement.
95. The control circuitry is further configured to display one or more items associated with the object within the extended reality environment in response to determining that the object is within the field of view and within the gaze for at least a predetermined period of time. 95. The system of any of items 91-94, configured to generate for.
96. The control circuitry is further configured, in determining whether the object is within the field of view for a predetermined period of time, to determine that the field of view is continuously over the object for a predetermined period of time during the extended reality session. The system according to any of items 91-95.
97. The control circuitry is further configured, in determining whether the object is within the field of view for a predetermined period of time, to determine that the field of view is discontinuously over the object for a predetermined period of time during the extended reality session. The system according to any one of items 91-96, wherein
98. The control circuitry further includes:
determining that the new object is in view for at least a predetermined amount of time;
in response to determining that the new object is in view for at least a predetermined period of time, generating one or more items related to the new object for display within the extended reality environment; causing one or more items associated with to occur for display within the extended reality environment;
The system according to any of items 91-97, configured to.
99. The control circuitry further includes:
99. A system according to any of items 91-98, configured to allow modification of an amount of time for a given time period.
100. 99. The system of any of items 91-99, wherein the extended reality environment is presented via an extended reality head-mounted device.

Claims (15)

A method for extended reality environment interaction, the method comprising:
generating an extended reality environment with objects for display;
detecting, by using a first sensor, that gaze is shifted from a first portion of the extended reality environment to a second portion of the extended reality environment, the object comprising: excluded from a first portion of the extended reality environment and included within a second portion of the extended reality environment;
responsive to detecting the gaze shift, generating an indicator of the shift in gaze for display within the extended reality environment;
detecting a voice command while the indicator is in the vicinity of the object by using a second sensor;
and performing an action corresponding to the voice command in response to detecting the voice command. 2. The method of claim 1, wherein an interactive media guide is provided on the display and the actions are instructions related to media assets accessible via the interactive media guide. The gaze is detected based on retinal movement of the eye, and the method further comprises:
3. A method according to claim 1 or 2, comprising converting the retinal movement into a movement of the indicator on the display. determining whether the rate of retinal movement exceeds a predetermined value;
normalizing the retinal movement when converting the retinal movement into movement of the indicator on the display in response to determining that the rate of the retinal movement exceeds the predetermined value; 4. The method of claim 3, further comprising. A method according to any of claims 2-4, wherein the voice command includes an identification of the media asset and a command to perform the action. 6. A method according to any of claims 2-5, wherein the voice commands include instructions for presenting new media assets on the display. 7. A method according to any of claims 2-6, wherein the object is associated with an entity and the voice command comprises an instruction to retrieve content associated with the entity. 8. The method of claim 1-7, wherein detecting the voice command while the indicator is in the vicinity of the object comprises determining that the gaze is directed at the object for at least a predetermined threshold period of time. Any method described. 9. A method according to any of claims 1-8, wherein the indicator being in the vicinity of the object includes the indicator overlapping the object. A method according to any preceding claim, wherein the display is presented via an extended reality head-mounted device. A system for extended reality environment interaction, said system comprising control circuitry configured to carry out the method of any of claims 1-10. A system for extended reality environment interaction, the system comprising:
display and
A control circuitry network,
generating an extended reality environment with objects for display;
detecting, by using a first sensor, that gaze is shifted from a first portion of the extended reality environment to a second portion of the extended reality environment, the object comprising: excluded from a first portion of the extended reality environment and included within a second portion of the extended reality environment;
responsive to detecting the gaze shift, generating an indicator of the shift in gaze for display within the extended reality environment;
detecting a voice command while the indicator is in the vicinity of the object by using a second sensor;
and control circuitry configured to: perform an action corresponding to the voice command in response to detecting the voice command. 13. The system of claim 12, wherein an interactive media guide is provided on the display and the actions are instructions related to media assets accessible via the interactive media guide. The control circuitry further includes:
detecting the gaze based on retinal movement of the eye;
14. The system of claim 12 or 13, configured to convert the retinal movement into a movement of the indicator on the display. a non-transitory computer-readable medium, the non-transitory computer-readable medium having instructions encoded thereon, the instructions, when executed by control circuitry, causing the control circuitry to , a non-transitory computer-readable medium for implementing the method of any of claims 1-10.