EP4619845A1

EP4619845A1 - Techniques for displaying extended reality content based on operator related parameters

Info

Publication number: EP4619845A1
Application number: EP23828281.8A
Authority: EP
Inventors: Sundar Murugappan; Pratap Ganapathy; Mark GARIBALDI; Michael Jones; Sida LI; Richard Mahoney; Govinda PAYYAVULA; Kelly PRINCE
Original assignee: Intuitive Surgical Operations Inc
Current assignee: Intuitive Surgical Operations Inc
Priority date: 2022-11-15
Filing date: 2023-11-14
Publication date: 2025-09-24
Also published as: WO2024107455A1; CN120476366A

Abstract

Techniques for displaying extended reality (XR) content include the following. An extended reality (XR) system comprises an XR device and a processor system. The processor system is configured to: determine an operator position of an operator portion of an operator based on first sensor data, determine a first object position of an object portion of an object based on second sensor data, and determine a first feature position for a virtual geometric feature based on the first object position. The processor system is also configured to determine a first content position based on the operator position while using the virtual geometric feature located at the first feature position to constrain the first content position, and cause the XR device to display an XR content based on the first content position.

Description

TECHNIQUES FOR DISPLAYING EXTENDED REALITY CONTENT BASED ON OPERATOR RELATED PARAMETERS

RELATED APPLICATIONS

[0001] This application claims the benefit to U.S. Provisional Application No. 63/425,618, filed November 15, 2022, and entitled “Techniques For Displaying Extended Reality Content Based On Operator Related Parameters,” which is incorporated by reference herein.

TECHNICAL FIELD

[0002] The present disclosure relates generally to electronic devices and more particularly relates to techniques for displaying extended reality content based on operator related parameters.

BACKGROUND

[0003] Computer-assisted electronic devices are being used more and more often. This is especially true in industrial, entertainment, educational, and other settings. As medical examples, the medical facilities of today have large arrays of computer-assisted devices being found in operating rooms, interventional suites, intensive care wards, emergency rooms, and/or the like. Many of these computer-assisted devices are moveable, and may be capable of autonomous or semi-autonomous motion. It is also known for personnel to control the motion and/or operation of moveable computer-assisted devices using one or more input devices located at a user control system. As a specific example of electronic systems comprising computer-assisted devices, minimally invasive, robotic telesurgical devices permit surgeons to operate on patients from bedside or remote locations. Telesurgery refers generally to surgery performed using surgical devices where the surgeon uses some form of remote control, such as a servomechanism, to manipulate surgical instrument movements rather than directly holding and moving the instruments by hand.

[0004] Extended reality (XR) systems are oftentimes used in conjunction with computer- assisted devices to assist in the performance of tasks at worksites. Examples of XR devices include augmented reality (AR) devices and virtual reality (VR) devices. As used herein, AR refers to a view of the physical environment with an overlay of one or more computergenerated graphical elements, including mixed reality (MR) environments in which physical objects and computer-generated elements can interact. As used herein, VR refers to a virtual environment that includes computer-generated elements. For example, an XR device can present data about an operating environment of a computer-assisted device, graphical elements for entertainment, or visual guidance during operation of the computer-assisted device, among other things.

[0005] Some conventional techniques for displaying XR content cause the XR content to be displayed in less optimal locations in some instances, thereby decreasing visibility of the XR content, increasing the time needed to view or interact with the XR content, decreasing the efficiency of operations performed with an XR device, reducing operator enjoyment or causing operator discomfort, requiring the operator to reposition him or herself, and/or the like.

[0006] Accordingly, improved techniques for displaying XR content are desirable.

SUMMARY

[0007] Consistent with some embodiments, an extended reality (XR) system includes an XR device and a processor system. The processor system is configured to: determine an operator position of an operator portion of an operator based on first sensor data, determine a first object position of an object portion of an object based on second sensor data, determine a first feature position for a virtual geometric feature based on the first object position, determine a first content position based on the operator position while using the virtual geometric feature located at the first feature position to constrain the first content position, and cause the XR device to display an XR content based on the first content position.

[0008] Consistent with some embodiments, a method of causing an extended reality (XR) device to display XR content includes: determining an operator position of an operator portion of an operator based on first sensor data, determining a first object position of an object portion of an object based on second sensor data, determining a first feature position for a virtual geometric feature based on the first object position, determining a first content position based on the operator position while using the virtual geometric feature located at the first feature position to constrain the first content position, and causing the XR device to display the XR content based on the first content position.

[0009] Other embodiments include, without limitation, one or more non-transitory machine-readable media including a plurality of machine-readable instructions, which when executed by one or more processors, are adapted to cause the one or more processors to perform any of the methods disclosed herein.

[0010] The foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a simplified diagram including an example of a computer-assisted device and an extended reality (XR) device, according to various embodiments.

[0012] Figure 2 is a perspective view illustrating an XR device, according to various embodiments.

[0013] Figure 3 illustrates the XR module of Figure 1 in greater detail, according to various embodiments.

[0014] Figure 4 illustrates a simplified diagram of a method for displaying XR content based on one or more operator-related parameters, according to various embodiments.

[0015] Figure 5 illustrates an example process of the method of Figure 4 in greater detail, when the portion of the operator is the head of the operator, according to various embodiments.

[0016] Figure 6 illustrates an example of displaying XR content, according to various embodiments.

[0017] Figure 7 illustrates another example of displaying XR content, according to various embodiments.

[0018] Figure 8 illustrates another example of displaying XR content, according to various embodiments.

DETAILED DESCRIPTION

[0019] This description and the accompanying drawings that illustrate inventive aspects, embodiments, embodiments, or modules should not be taken as limiting — the claims define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure the invention. Like numbers in two or more figures represent the same or similar elements. [0020] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.

[0021] Further, the terminology in this description is not intended to limit the invention. For example, spatially relative terms-such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like-may be used to describe one element’s or feature’s relationship to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational placements) of the elements or their operation in addition to the position and orientation shown in the figures. For example, if the content of one of the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features. Likewise, descriptions of movement along and around various axes include various special element positions and orientations. In addition, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise. And, the terms “comprises”, “comprising”, “includes”, and the like specify the presence of stated features, steps, operations, elements, and/or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0022] Elements described in detail with reference to one embodiment, embodiment, or module may, whenever practical, be included in other embodiments, embodiments, or modules in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, embodiment, or application may be incorporated into other embodiments, embodiments, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or embodiment nonfunctional, or unless two or more of the elements provide conflicting functions.

[0023] In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0024] This disclosure describes various elements (such as systems and devices, and portions of systems and devices) with examples in three-dimensional space. In such examples, the term "position" refers to the location of an element or a portion of an element in space (e.g., three degrees of translational freedom in three-dimensional (3D) space, which can be described as along Cartesian x-, y-, and z-coordinates). Also in such examples, the term "orientation" refers to the rotational placement of an element or a portion of an element in space (e.g., three degrees of rotational freedom in 3D space, such as rotations about Cartesian z-, y-, and z- axes, rotation sets such as roll, pitch, and yaw, and the like). Other examples may encompass other dimensional spaces, such as two-dimensional spaces. As used herein, the term "pose" refers to the position, the orientation, or the position and the orientation combined, of an element or a portion of an element. As used herein, the term "proximal" for elements in a kinematic series refers to a direction toward the base of the kinematic series, and the term "distal" refers to a direction away from the base along the kinematic series.

[0025] Aspects of this disclosure are described in reference to electronic systems and computer-assisted devices, which may include systems and devices that are teleoperated, remote-controlled, autonomous, semiautonomous, manually manipulated, and/or the like. Example computer-assisted systems include those that comprise robots or robotic devices. Further, aspects of this disclosure are often described in terms of an embodiment using a medical system, such as the da Vinci® Surgical System commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that inventive aspects disclosed herein may be embodied and implemented in various ways, including in medical and non-medical embodiments, and including with robotic and, as applicable, non-robotic embodiments. Embodiments described for da Vinci® Surgical Systems are merely exemplary, and are not to be considered as limiting the scope of the inventive aspects disclosed herein. For example, techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperational systems. As further examples, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and/or the like. Additional example applications include use for procedures on tissue removed from human or animal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.

System Overview

[0026] Figure 1 is a simplified diagram of an electronic system 100, according to various embodiments. As shown, electronic system 100 includes, without limitation, a computer- assisted device 110 and an extended reality (XR) device 130. Computer-assisted device 110 includes a repositionable structure, the repositionable structure comprising a manipulator arm 120 configured to support an instrument 122. Although Figure 1 shows the repositionable structure as comprising one manipulator arm 120 configured to support one instrument 122, this is for illustrative purposes. The repositionable structure of a computer-assisted device can comprise only a manipulator arm, a manipulator arm and other mechanical structures physically coupled to the manipulator arm, one or multiple (e.g., zero, one, two, three, four, or any suitable number) of manipulator arms, one or more repositionable hardware components other than manipulator arms, a combination or portion of any of the foregoing, and/or the like. Further, each manipulator arm or other repositionable component can be configured to support one instrument, or a plurality of instruments.

[0027] In some examples, instrument 122 is an imaging instrument, a manipulation instrument such as graspers or scissors, a fastening instrument such as a stapler, an irrigation instrument, a suction instrument, an energy application instrument, an instrument with multiple functions, or any other appropriate instrument. In some examples, instrument 122 is an imaging instrument such as a monoscopic or stereoscopic camera, a still or video camera, an endoscope, a hyperspectral device, an infrared or ultrasonic device, an ultrasonic device, a fluoroscopic device, and/or the like. In some examples, instrument 122 is a medical instrument, such as a medical endoscope, forceps, clip appliers, a gripper, a retractor, a cautery instrument, a suction instrument, a suturing device, a stapling device, a cutting device, and/or the like. In some examples, instrument 122 includes an end effector capable of performing one or multiple tasks, such as grasping a material (e.g., tissue of a patient in a medical example) located in a workspace and delivering energy to material (e.g., delivering electrocautery energy to a patient in a medical example). In some examples, the energy includes ultrasonic, radio frequency, electrical, magnetic, thermal, light, and/or other types of energy. In some examples, the manipulator arm 120 includes one or more joints and links and is configured to support the instrument 122. In some examples, instrument 122 is, during use, inserted into a workspace (e.g., anatomy of a patient or cadaver, a veterinary subject, an anatomical model, and/or the like in some medical examples) through a cannula, access port, and/or the like to perform a procedure.

[0028] In some examples, computer-assisted device 110 is a teleoperated device. In some medical examples, the teleoperated device is a teleoperated medical device, such as a telesurgical device, that can be found in an operating room and/or an interventional suite. In some examples, computer-assisted device 110 is a follower device that is teleoperated by being controlled by one or more leader devices (not shown), such as one or more input devices designed to be contacted and manipulated by an operator (not shown). The one or more input devices may be mechanically grounded (kinematically grounded by mechanical structures) or mechanically ungrounded (not kinematically grounded by mechanical structures). Systems that include a leader device and a follower device are referred to as leader-follower systems, and also sometimes referred to in literature as master-slave systems where the leader is termed the “master” and the follower is termed the “slave.” When computer-assisted device 110 is a teleoperated follower device comprising a repositionable structure (e.g., a manipulator arm 120), and the follower device is controlled to move and articulate in response to manipulation of leader device(s) by an operator, the computer-assisted device 110 “follows” the leader input device(s) through teleoperation. The operator is then able to perform tasks at a worksite using the manipulator arm 120 and/or instrument 122 (if supported by the manipulator arm 120). In various embodiments, the repositionable structure (e.g., a manipulator arm) may or may not be configured to support an instrument (e.g., instrument 122).

[0029] In some embodiments, XR device 130 comprises an augmented reality (AR) device or a virtual reality (VR) device. XR device 130 is described in greater detail below in conjunction with Figure 2. In some examples, XR device 130 is used in conjunction with computer-assisted device 110. For example, XR device 130 could be used to present instructional content on how to operate computer-assisted device 110. As another example, XR device 130 could be used to present content that provides guidance during operation of computer-assisted device 110. Illustratively in Figure 1, XR device 130 has displayed AR content 136 next to repositionable structure with a manipulator arm 120 of computer-assisted device 110. AR content 136 can be provided in any manner appropriate for visual AR content, such as a visual overlay. AR content 136 can be rendered for display by XR device 130, by computer-assisted device 110, or by any other device or devices. In the example shown in Figure 1, in order for XR content (e.g., AR content 136) that is presented by XR device 130 to be displayed at appropriate positions and orientations relative to computer-assisted device 110, a registration transform 134 between XR device 130 and computer-assisted device 110 is established. Registration transform 134 provides a geometric relationship between a frame of reference related to XR device 130 (such as reference frame 132 fixed relative to the XR device 130 or another frame of reference) and a frame of reference related to the computer- assisted device 110 (such as reference frame 124 fixed relative to computer-assisted device 110 or another frame of reference). XR content to be displayed relative to a portion of computer-assisted device 110 can be defined at a position in reference frame 124, converted to a position in reference frame 132 using registration transform 134, and displayed by XR device 130 at the position defined in reference frame 132 (or, in some instances, in a position adjusted from the position defined in reference frame 132, as discussed further below).

[0030] In some embodiments, a virtual geometric feature associated with a portion of computer-assisted device 110 is used to determine a position (and optionally orientation) in reference frame 124 at which to display AR content 136. This position in reference frame 124 is then mapped to a corresponding position (and optionally orientation) in reference frame 132 using at least registration transform 134. The XR device 130 can then be caused to display AR content 136 at the corresponding position in reference frame 132. This is discussed in greater detail below in conjunction with Figures 3-8. In some instances, the AR content 136 is displayed in a position adjusted from the position defined in reference frame 132, as discussed further below.

[0031] As shown in Figure 1, reference frame 124 is a base reference frame of computer- assisted device 110. In some examples, reference frame 124 has an origin located at a central point on a base of computer-assisted device 110 and is aligned with one or more major axes of computer-assisted device 110. In some examples, reference frame 124 has some other origin or orientation, such as one fixed relative to a link of manipulator arm 120, a base of the repositionable structure having the manipulator arm 120, or another portion of the repositionable structure having the manipulator arm 120. In some examples, the base of computer-assisted device 110 is on a planar surface on which computer-assisted device 110 is wheeled, slid, and/or otherwise repositioned. In some examples, a z-axis of reference frame 124 corresponds to a vertical up direction for the computer-assisted device 110 or operator 108. In some examples, reference frame 132 has an origin located at a point within XR device 130 and is oriented in any suitable manner. In some examples, a z-axis of reference frame 132 corresponds to a direction of view of XR device 130, a y-axis of reference frame 132 corresponds to the view up direction indicating an “up” direction when orienting XR content for display by XR device 130, and an x-axis of reference frame 132 is orthogonal to both the z- and y-axes of reference frame 132 and has defined positive and negative directions (e.g., left and right).

[0032] In some examples, the geometric relationship between reference frame 124 and reference frame 132 is characterized using a 6 degrees of freedom (6 DOF) registration transform 134, or a series of transforms that can be combined into a single registration transform 134 . For example, the registration transform 134 can include parameters for describing 3 rotational degrees of freedom (DOF) of a rotation portion and a 3 translational DOFs of a translation portion. In such cases, the rotation portion can be represented with a 3x3 matrix describing the 3 -dimensional rotational difference between reference frame 124 and reference frame 132. In some examples, the rotation portion describes rotation about the three axes of reference frame 124. In some examples, the three axes correspond to x, y, and z axes; or roll, pitch, and yaw axes, or the like. In some examples, the rotation portion is represented using angle-axis, quaternion, and/or similar equivalent notations. In some examples, the translation portion can be represented by a 3x1 vector describing the 3 -dimensional displacement between reference frame 124 and reference frame 132.

[0033] As shown, a control system 140 is provided external to computer-assisted device 110 and XR device 130, and control system 140 communicates with both computer-assisted device 110 and XR device 130. In other embodiments, control system 140 can be included in computer-assisted device 110 or in XR device 130. In some embodiments, control system 140 generates XR content that is presented by XR device 130. In some embodiments, control system 140 also determines or provides control signals to computer-assisted device 110 to control movement of manipulator arm 120 and/or instrument 122 based on the received information and operator input. In some embodiments, control system 140 supports one or more wired communication protocols, (e.g., Ethernet, USB, and/or the like) and/or one or more wireless communication protocols (e.g., Bluetooth, IrDA, HomeRF, IEEE 1002.11, DECT, Wireless Telemetry, and/or the like).

[0034] Control system 140 is implemented on one or more computing systems. As shown, control system 140 includes a processor system 150 and a memory 160 storing a XR module 170. In some embodiments, control system 140 includes one or more processors that execute the XR module 170, non-persistent storage (e.g., volatile memory, such as random access memory (RAM), cache memory), persistent storage (e.g., a floppy disk, a flexible disk, a hard disk, any other magnetic medium, a compact disk (CD) drive, a digital versatile disk (DVD) drive, any other optical medium, a flash memory, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, a physical memory, and/or any other medium from which a processor or computer is adapted to read), a communication interface (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and numerous other elements and functionalities. In some embodiments, the non-persistent storage and/or persistent storage of control system 140 includes non-transitory, tangible, machine readable media that stores, in whole or in part, temporarily or permanently, computer readable program code that, when executed by one or more processors (e.g., processor system 150), cause the one or more processors to perform embodiments of the disclosure, including the processes of method 400 and/or the processes of Figures 4 and/or 5, described below. Although described herein primarily with respect to XR module 170 that runs on control system 140, in some embodiments the functionality of XR module 170 can be implemented in any technically feasible software and/or hardware, including partially or entirely within XR device 130, within another computing system, within a cloud computing system, etc.

[0035] An extended reality (XR) system that causes XR device 130 to display XR content can comprise any appropriate portion(s) of an electronic system, such as electronic system 100. For example, in some embodiments, an XR system includes the XR module 170 only, the control system 140 only, the XR module 170 with the XR device 130, the control system 140 with the XR device 130, all of electronic system 100, etc.

[0036] Each of the one or more processors of control system 140 is an integrated circuit for processing instructions. For example, the one or more processors can be one or more cores or micro-cores of a processor, a central processing unit (CPU), a microprocessor, a field- programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), a tensor processing unit (TPU), and/or the like. In some examples, control system 140 also includes one or more input devices (not shown), such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device.

[0037] A communication interface of control system 140 can include an integrated circuit for connecting the computing system to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) and/or to another device, such as another computing system.

[0038] In some examples, control system 140 further includes one or more output devices (not shown), such as a display device (e.g., a liquid crystal display (LCD), a plasma display, touchscreen, organic LED display (OLED), projector, or other display device), a printer, a speaker, external storage, or any other output device. One or more of the output devices can be the same or different from the input device(s). Many different types of computing systems exist, and the aforementioned input and output device(s) can take other forms.

[0039] In some embodiments, control system 140 is connected to or be a part of a network. The network can include multiple nodes. Control system 140 is implemented on one node or on a group of nodes. By way of example, control system 140 can be implemented on a node of a distributed system that is connected to other nodes. By way of another example, control system 140 can be implemented on a distributed computing system having multiple nodes, where different functions and/or components of control system 140 are located on a different node within the distributed computing system. Further, one or more elements of the aforementioned control system 140 can be located at a remote location and connected to the other elements over a network.

[0040] Some embodiments include one or more components of a teleoperated medical system such as a da Vinci® Surgical System, commercialized by Intuitive Surgical, Inc. of Sunnyvale, California, U.S.A. Embodiments on da Vinci® Surgical Systems are merely examples and are not to be considered as limiting the scope of the features disclosed herein. For example, different types of teleoperated systems having computer-assisted devices comprising follower devices configured to be placed at worksites can use the features described herein. Further, non-teleoperated systems can also make use of features described herein.

[0041] Figure 2 is a perspective view illustrating head-mounted XR device 130 in greater detail, according to various embodiments. As shown, XR device 130 includes a body 205 and a head mount 210. Body 205 includes one or more electronic display elements of an electronic display 230. Body 205 also includes a sensor system 240 that acquires sensor data associated with the physical environment external to XR device 130, which can also be external to any objects (such as computer-assisted devices) in the physical environment. Sensor system 240 can include any technically feasible sensor or sensors, such monoscopic and stereoscopic optical systems, ultrasonic systems, depth cameras such as cameras using time-of-flight sensors, LIDAR (Light Detection and Ranging) sensors, stereo RGB (red, green, blue) sensors, RGB-D depth-sensors, etc. Although Figure 2 shows a head-mounted XR device, other XR devices may be used in other embodiments. Examples of other types of XR devices include appropriately configured tablets, smart-phones, projectors, etc. Although sensor system 240 is shown in Figure 2 as included in XR device 130, a sensor system used to provide sensor data associated with the physical environment external to XR device 130 (which can also include data external to one or more objects in the physical environment) can be provided in any appropriate location. For example, part or all of such a sensor system can be alternatively or additionally located elsewhere in the physical environment, including mounted on walls, ceilings, or stands, or coupled to a computer-assisted device.

[0042] In addition, body 205 includes electronic display 230 and an optics block 235 that together provide image light to a target location of body 205 where an eye 202 of an operator may be positioned. In some examples, body 205 also includes one or more other sensors, such as one or more imaging devices (e.g„ one or more imaging sensors for tracking eye 202), accelerometers, and/or angular velocity sensors (which can be part of inertial measurement units (IMUs)), position sensors, and/or other sensors.

[0043] Electronic display 230 is configured to display rendered images that are viewable by the operator. In various embodiments, electronic display 230 includes a single electronic display or multiple electronic displays (e.g„ a display for each eye of an operator). Examples of the electronic display 230 include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a QOLED, a QLED, some other display, or some combination thereof. Optics block 235 includes optical elements that can be used to adjust an orientation of image light emitted from electronic display 230 such that electronic display 230 appears at particular virtual image distances from the operator.

[0044] In some embodiments, XR device 130 operates as an AR device that presents computer-generated media to an operator using electronic display 230 that augments views of a physical, real-world environment visible to the operator through electronic display 230. Examples of computer-generated media presented by XR device 130 include one or more images, video, audio, or some combination thereof. Alternatively or additionally, in some embodiments, XR device 130 operates as a VR device, or some combination of an AR device and a VR device, such as a device that permits switching between AR and VR environments. In the case of a VR device, sensor system 240 can capture images of the physical environment and display the captured images along with computer-generated elements, such as AR content 136, which is also sometimes referred to as “video see through.” Examples of commercially available AR devices include Microsoft HoloLens®, Google Glass®, and Meta 2®. Examples of commercially available MR devices include Microsoft HoloLens 2®, Samsung Odyssey+®, HP Reverb®, and Oculus Quest 2®. Examples of commercially available VR devices include Oculus Rift®, Samsung Gear VR®, HTC Vive®, and Google Daydream View®.

[0045] It should be noted that Figure 2 merely shows an example configuration of an XR device. In some embodiments, the techniques for displaying XR content that are disclosed herein are usable with other configurations and/or types of XR devices. The other configurations and/or types of XR devices include head-mounted XR devices, hand-held XR devices, and XR devices that are placed in the environment, among other things. Examples of alternative configurations and/or types of XR devices include optical head-mounted displays (HMDs), mobile devices mobile phones, tablet computers, etc.), fully immersive projection systems, etc.

Displaying Extended Reality Content Based on One or More Operator Related Parameters

[0046] Some conventional XR devices display XR content at a position that is fixed relative to the position of a portion of an object. In such cases, the XR content is rendered at the same position relative to the portion of the object even when an operator viewing the XR content moves. In some cases, the displayed position of XR content moves along with a portion of an object such that the displayed position stays fixed relative to the portion of the object.

[0047] In some embodiments, an XR device displays XR content coincident to a virtual geometric feature based on the position of a portion of an operator and the position of a portion of an object, such as a portion of a computer-assisted device.

[0048] Figure 3 illustrates XR module 170 of Figure 1 in greater detail, according to various embodiments. As shown, XR module 170 includes, without limitation, an operator position evaluation module 306, an object position evaluation module 308, and an overlay module 310.

[0049] During operation, operator position evaluation module 306 receives sensor data 302 that is acquired by a sensor system and determines the position of a portion of an operator (“operator portion”). The portion of the operator can be the head, chest, or any other suitable portion of the operator. The sensor data used to determine the position of the portion of the operator can be acquired by eye tracking sensors, gyroscopes, cameras, and/or any other suitable sensor device(s) that are mounted on the XR device or elsewhere in the environment. In some embodiments, operator position evaluation module 306 also determines an orientation metric associated with the portion of the operator, such as a direction of view of the head of the operator.

[0050] During operation, object position evaluation module 308 receives sensor data 304 that is acquired by a sensor system (e.g., sensor system 240) and determines the position of a portion of an object (e.g., an object portion). In some embodiments, the object position evaluation module 308 also determines an orientation metric associated with the portion of the object. The sensor system that acquires sensor data 304 can be the same as, or different from, the sensor system that acquires sensor data 302. When the object includes a repositionable structure, the sensor data 304 can include kinematic data associated with one or more joints and/or links of the repositionable structure. For example, kinematic data can be acquired by joint sensors that transmit positions and orientations of joints of the repositionable structure to record movements thereof, shape sensors that monitor the shape of an optical fiber to determine the pose of the repositionable structure located at one end of the optical fiber relative to a frame of reference located at the other end of the fiber, and/or in any other technically feasible manner. Although discussed herein primarily with respect to examples in which the portion of the object is a portion of a computer-assisted device that includes a repositionable structure, techniques disclosed herein are also applicable to cases in which the object is not a computer-assisted device, or is a computer-assisted device that does not include a repositionable structure.

[0051] During operation, overlay module 310 generates XR content and causes the XR content to be displayed via an XR device (e.g., XR device 130). The XR content includes AR and/or VR content in some embodiments. Any suitable XR content, such as the AR content 136 described above in conjunction with Figure 1, can be generated. In some embodiments, the XR content includes text, graphical images, animations or videos, and/or virtual controls. The XR content is used to entertain, aid or instruct, present virtual controls, and/or otherwise provide input for or output to an operator. For example, the XR content could include instructional content on how to operate a computer-assisted device (e.g., computer-assisted device 110). As another example, the XR content could include content that provides data related to a procedure being performed by a computer-assisted device (e.g., computer-assisted device 110), such as previously captured images, models, real-time captured images, data about the functioning of the computer-assisted device, communications from others, tutorials or videos, guidance during operation of the computer-assisted device, etc.

[0052] In some embodiments, overlay module 310 determines a position with which to display XR content (e.g., a content position), and optionally an orientation with which the XR content is displayed (e.g., a content orientation), based on (1) the position of a portion of an operator (also referred to herein as an “operator position”), and optionally the orientation of the operator (also referred to herein as an “operator orientation”), determined by operator position evaluation module 306; and (2) the position of the portion of the object (also referred to herein as an “object position”), and optionally the orientation of the portion of the object (also referred to herein as an “object orientation”), determined by object position evaluation module 308. In such cases, overlay module 310 first determines, based on the position and optionally the orientation of the portion of the object, a position of a virtual geometric feature (also referred to herein as a “feature position”), and optionally an orientation of the virtual geometric feature (also referred to herein as a “feature orientation”), that has a known relationship with respect to a portion of the object. Then, overlay module 310 determines the position (and optionally orientation) of the XR content based on the position (and optionally orientation) of the virtual geometric feature and the position (and optionally orientation) of the portion of the operator, as discussed in greater detail below in conjunction with Figures 4-8.

[0053] In addition, overlay module 310 applies a registration transform to map the determined position (and optionally orientation) of the XR content in a reference frame of the object to a corresponding position (and optionally orientation) in a reference frame of XR device 130. Then, overlay module 310 generates (e.g., renders) XR content for display at the corresponding position (and optionally orientation) in the reference frame of XR device 130. Thereafter, overlay module 310 transmits, to XR device 130, a display signal 312 that causes XR device 130 to display the XR content at the corresponding position (and optionally orientation) in the reference frame of XR device 130.

[0054] In the AR case, overlay module 310 generates content for display to an operator to enhance a view of the physical environment, which is sometimes also referred to as “optical see through.” In the VR case, the content generated by overlay module 310 is combined with image data depicting the physical environment to generate a composite image for display to the operator, which is sometimes also referred to as “video see through.” The image data is captured by one or more imaging devices in sensor system 240, or elsewhere. In addition, display signal 312 is generated based on the composite image.

[0055] Figure 4 illustrates a simplified diagram of a method for displaying XR content based on one or more operator-related parameters, according to various embodiments. One or more of the processes 402-410 of method 400 can be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine readable media that when executed by one or more processor systems (e.g., the processor system 150 in control system 140) cause the one or more processors to perform one or more of the processes 402-410. In some embodiments, method 400 can be performed by one or more modules, such as XR module 170. In some embodiments, method 400 can include additional processes, which are not shown.

[0056] As shown, method 400 begins at process 402, where the position of a portion of an operator is determined based on sensor data acquired by a first sensor system. Examples of the first sensor system are described above in conjunction with Figure 3. In some examples, the portion of the operator is a head of the operator. In some examples, the sensor data includes data acquired by cameras, gyroscopes, eye tracking sensors, and/or any other suitable sensor device(s) that are mounted on an XR device (e.g., XR device 130) or elsewhere in the environment. In some embodiments, an orientation of the portion of the operator is also determined from the sensor data. In some examples, the orientation of the portion of the operator can be the direction of view of the operator, or derived from the direction of view of the operator, etc. As some examples usable for such cases, the direction of view can be an eye gaze direction, or a proxy or estimation of the gaze direction based on some other parameter(s). For example, a proxy for the direction of view can be a direction of a head of the operator, such as a direction that the nose or mouth or face is facing, or estimated from such direction of the head. In some embodiments, the direction of view can be represented by a gaze direction vector associated with the direction of view. In a specific example, the gaze direction vector originates from the head of the operator and extends in a direction that the operator is looking, such as from the center of a line segment connecting the pupils of the operator and in the direction the operator has directed the pupils.

[0057] At process 404, the position of a portion of an object is determined based on sensor data acquired by a second sensor system. Process 404 can be performed concurrently with, before, and/or after process 402 in some embodiments. Examples of the second sensor system are described above in conjunction with Figure 3. In some embodiments, the object is a computer-assisted device (e.g., computer-assisted device 110) that includes a repositionable structure. In such cases, the portion of the object can be a portion of the repositionable structure (such as a part of a link of the repositionable structure), or another part of the computer-assisted device. When the portion of the computer-assisted device is a portion of the repositionable structure, the sensor data includes kinematic data associated with one or more joints and/or links of the repositionable structure, and the position of the portion of the repositionable structure is determined from the kinematic data using forward kinematics. In other embodiments, techniques other than forward kinematics can be used to determine the position of the portion of the repositionable structure based on data acquired by other types of sensors, such as a gyroscope, shape sensors, etc. In some examples, when the portion of the object is not part of a repositionable structure, the sensor data includes image data captured by camera(s), data acquired by time-of-flight sensor(s), gyroscopic data, and/or the like. The position of the portion of the object can be determined in any technically feasible manner, including using well-known techniques, depending on the type of sensor data. In some embodiments, an orientation of the portion of the object is also determined from the sensor data.

[0058] At process 406, the position for a virtual geometric feature is determined based on the position of the portion of the object. In some embodiments, the virtual geometric feature can have any suitable (1) shape and/or size, and (2) relationship with respect to the portion of the object. Examples of virtual geometric features include a portion of a line (“line portion,” such as a line segment) or a line (e.g., an infinite line), a portion of a spline (“spline portion”), a spline (e.g., an infinite spline), a portion of a surface (“surface portion”) or an entire surface of any shape and size, a complex feature combining any number of any of the foregoing, and/or the like. In some embodiments, a virtual geometric feature is selected based on the type of XR content to be displayed and/or physical characteristics of the portion of the object. For example, different virtual geometric features could be used for menus, labels, virtual controls, and other types of XR content. As another example, different virtual geometric features could be used for different types of portions of an object, such as for the helm, link of a manipulator, etc. of a computer assisted device. As yet another example, different virtual geometric features could be used for large, as opposed to small, portions of an object. [0059] In some embodiments, the virtual geometric feature has a known relationship with respect to the portion of the object. The known relationship can be a fixed relationship or a dynamic relationship. As an example of a fixed relationship, the position of a virtual geometric feature could be offset by a predefined amount (e.g., by a predefined distance or other offset parameter) relative to the position of a portion of an object while being parallel to an axis of the portion of the object. As another example of a fixed relationship, the virtual geometric feature could be a spline that circumscribes a portion of an object and is extruded into a surface that circumscribes the portion of the object. As an example of a dynamic relationship, the virtual geometric feature could change from being a certain offset away from a portion of the object to being further away or closer to the portion of the object. As another example of a dynamic relationship, the shape of a virtual geometric feature could change, such as from a line to a spline or vice versa, based on the position of a portion of the operator. As yet another example of a dynamic relationship, the depth with which XR content is displayed on a virtual geometric feature could depend on a distance of the portion of the operator from the portion of the object. In addition, the virtual geometric feature can have any suitable orientation. In some examples, the virtual geometric feature has a fixed or dynamic orientation relative to an orientation of the portion of the object or an axis associated with the portion of the object. For example, when the virtual geometric feature is a line, the line could be vertical, horizontal, slanted, etc.

[0060] At process 408, XR content is caused to be displayed based on the position of the portion of the operator and the position for the virtual geometric feature. As described, in some embodiments, the XR content includes text, graphical images, animations or videos, and/or virtual controls that are used to entertain, aid or instruct, present virtual controls, and/or otherwise provide input for or output to an operator. In some embodiments, the XR content is constrained by the virtual geometric feature located at the position for the virtual geometric feature. In some embodiments, the XR content is constrained to be coincident with the virtual geometric feature. In some embodiments, causing the XR content to be displayed comprises: (1) applying a registration transform to map a position of the XR content that is coincident with the virtual geometric feature, in a reference frame of the object, to a corresponding position in a reference frame of an XR device (e.g., XR device 130); and (2) transmitting one or more control signals to the XR device that cause the XR device to display the XR content at the corresponding position in the reference frame of the XR device.

[0061] In some embodiments, the XR content can be constrained by the virtual geometric feature in any technically feasible manner. In some examples, a gaze direction vector associtaed with the operator changes which DOFs of the operator are used to constrain the XR content to the virtual geometric feature. For example, when the gaze direction vector is parallel to the ground or within 45 degrees of plane parallel to the ground, then upward- downward motions of the head of the operator can be mapped to motions of the XR content along a vertical line that is the virtual geometric feature. As another example, when the gaze direction vector is pointed downward or within 45 degrees of downward, then sideways motions of the head of the operator can be be mapped to motions of the XR content along a horizontal line that is the virtual geometric feature, and the gaze direction vector is used to determine the position of the XR content.

[0062] In some examples, XR content is displayed based on a virtual geometric feature without being coincident to the virtual geometric feature. For example, when, in a field of view of the operator, XR content to be displayed overlaps, underlays, or intersects (e.g„ would occlude or be occluded by) a portion of an object, a size, position, and/or an orientation of the XR content can be adjusted to reduce an amount of the XR content that overlaps, underlays, or intersects the portion of the object. The reduction provided by such adjustment (e.g., adjusted position, adjusted orientation, and/or adjusted size) is in comparison to the unadjusted position, orientation, and/or size of the XR content. The reduction can be partial where some overlapping/underlaying of the XR content or intersection of the XR content with the portion of the object would still exist with the adjustment. Alternatively, the reduction can be complete where the overlapping/underlaying of the XR content or the intersection of the XR content with the portion of the object is eliminated or reduced to an extent not perceptible by the operator. For example, the XR content can be moved in front of the portion of the object or to an offset position that reduces an amount of the XR content that overlaps, underlays, or intersects the portion of the object. As a specific example, spatial mapping can be used to identify objects (which can be represented as virtual meshes) of interest in a scene, and in conjunction with a registration, place a 3D model of the portion of the object overlaid with the meshes. Then, XR content can be placed in front of, or offset from, the 3D model taking occlusion meshes into account. When the XR content is placed in front of the portion of the object or at the offset position, the adjusted XR content can still be coincident to the virtual geometric feature, or no longer coincident to the virtual geometric feature. As another example, when XR content to be displayed overlaps/underlays or intersects a portion of an object, a depth and/or size of the XR content can be adjusted to reduce the amount to which the XR content overlaps/underlays or intersects the portion of the object. As noted, the reduction can be partial or complete. In some embodiments, XR content is permitted to be displayed in front of, behind, and/or in a manner that intersects one or more portions of an object.

[0063] In some embodiments, the orientation of the XR content relative to the virtual geometric feature is controlled based on an orientation of the portion of the operator relative to the portion of the object. For example, the orientation of the XR content (e.g., text boxes, menus, images, videos, and/or the like) can be controlled so that the XR content always faces the operator in a field of view of the operator, which is also referred to herein as “billboarding.” In such cases, the XR content is displayed perpendicular to a gaze direction vector, the vector associated with a direction of view of the operator.

[0064] In some embodiments, additional rules are applied to determine how to display the XR content. In some examples, XR content is displayed relative to other XR content and/or to objects in the physical environment according to predefined rules. For example, a rule could prioritize animations to be displayed in at the center of a field of view, while other elements (e.g., text) are permitted to be displayed away from the center of the field of view and/or out of the field of view. As another example, some XR content could be displayed to fit a field of view of the operator, i,e„ to be displayed entirely within the field of view of the operator, so that the operator has a full view of the content. In such cases, the z-position (depth) and scaling (up or down) of the XR content can also be optimized so that the content is as big as possible.

[0065] In some embodiments, when the portion of the operator is a head of the operator, the position of the XR content relative to the virtual geometric feature is controlled based on the position of the head of the operator, and optionally the gaze direction vector (e.g., a vector associated with a direction of view of the operator) or another orientation metric associated with the head of the operator. Figure 5 illustrates an example process 408 of the method 400 of Figure 4 in greater detail, according to various embodiments. In the example of Figure 5, the portion of the operator is the head of the operator, the virtual geometric feature is used to constrain the XR content by requiring coincidence, and other considerations are described further below. It should be understood that Figure 5 presents just one example of process 408, and other embodiments may use other portions of the operator, other virtual geometric features, other types of constraints, etc.

[0066] As shown, at process 502, when a gaze direction vector intersects a surface associated with the virtual geometric feature, then at process 504, XR content is displayed coincident to the virtual geometric feature based on the intersection of the gaze direction vector and the surface associated with the virtual geometric feature. In such cases, the gaze direction vector begins from a position associated with the head of the operator (e.g., the center of a line between eyes of the operator) and intersects the surface associated with the virtual geometric feature. For example, when the virtual geometric feature is a vertical line, a height (or x-y position) of the XR content along the line is determined by intersecting a gaze direction vector with a planar surface that includes the line and is perpendicular to the gaze direction vector. As another example, when the virtual geometric feature is a spline that circumscribes the object portion, a position of the XR content along the spline is determined by intersecting a gaze direction vector with a non-planar surface that circumscribes the object portion and is generated by either extruding the spline in a direction corresponding to an orientation axis of the object portion, or by extruding or sweeping between the spline and another spline (e.g., a sweep surface between two rail curves).

[0067] Alternatively, when the gaze direction vector does not intersect the surface associated with the virtual geometric feature, then at process 506, a closest point on the virtual geometric feature to the gaze direction vector is determined. In some embodiments, the closest point is determined by (1) projecting the virtual geometric feature onto a plane that is perpendicular to the gaze direction vector, and (2) determining a point of the projected virtual geometric feature that is closest to the gaze direction vector. In some embodiments, the gaze direction vector is associated with a direction of view of the operator.

[0068] At process 508, XR content is caused to be displayed coincident to the virtual geometric feature based on the closest point. In some embodiments, the XR content is displayed at or near the closest point.

[0069] Returning to Figure 4, the displayed position and/or orientation of the XR content can also account for operator preference in some embodiments. In some examples, the position and/or pitch of the XR content is adjusted based on operator preference that is specified in any technically feasible manner (e.g., via a user interface). For example, when the position of XR content along a vertical line segment (i.e„ the virtual geometric feature) is determined based on the eye gaze height of an operator, the position of the XR content could be further adjusted to be higher or lower than the eye gaze height based on operator preference. As another example, an offset of XR content from an axis associated with a portion of an object can be adjusted based on operator preference. [0070] In some embodiments, the displayed position and/or orientation of the XR content is also based on a control point associated with the XR content. The control point can be at any suitable position. For example, the control point could be positioned at a corner (e.g„ the upper-left comer) of the XR content, along an edge a center point of a left edge) of the XR content, a center of the XR content, or at any other position relative to the XR content (e.g., within the XR content or located near the XR content). In some examples, the control point is positioned based on the intersection point determined at process 504 or the closest point determined at process 506, described above in conjunction with Figure 5.

[0071] Subsequent to process 408, method 400 returns to process 402, where the position of the portion of the operator is determined based on additional sensor data acquired by the first sensor system. For example, when the portion of the operator and/or the portion of the object moves, the processes of method 400 are repeated to update the displayed position (and optionally orientation) of the XR content.

[0072] Although method 400 is described primarily with respect to a single operator, in some embodiments, processes similar to those of method 400 are performed to display XR content for multiple operators. In such cases, the XR content is displayed based on the positions (and optionally view directions or other orientation metrics) of portions of the multiple operators and constrained by the virtual geometric feature. In some examples, the head positions and directions of view of the multiple operators are averaged, and method 400 is performed for the averaged head positions and directions of view. In some examples, XR content is displayed at a different position (and optionally orientation) determined according to method 400 for each of the multiple operators.

[0073] Figure 6 illustrates an example of displaying XR content, according to various embodiments. As shown, a line segment 602 is the virtual geometric feature in this example. Line segment 602 is offset from a portion of an object, shown as an axis 603 of a portion of a manipulator arm of computer-assisted device 110, by a fixed distance. To display XR content 604, a point on line segment 602 that is at or closest to the height of a portion of operator 108, shown as the head of operator 108, is determined. The point is determined by intersecting a gaze direction vector 606 with a plane 605, on which line segment 602 lies, that is perpendicular to gaze direction vector 606. Line segment 602 extends a finite distance in plane 605 and is, therefore, associated with a range of height (y-axis) values. When the intersection is at a height that is within the range of height values associated with line segment 602, then XR content 604 is displayed coincident to line segment 602 at the intersection height. When the intersection is at a height that is outside the range of height values associated with line segment 602, then XR content 604 is displayed coincident to line segment 602 at one of the end points of line segment 602 that is closest to the intersection height. In addition, XR content 604 is displayed in an orientation that faces towards operator 108 in a field of view of operator 108. The displayed position and/or orientation of XR content 604 is also adjusted based on operator preference, as described in conjunction with Figure 4. For example, the displayed position of XR content 604 could be adjusted to be higher or lower than the intersection height described above, the displayed offset of XR content 604 from axis 603 could be adjusted, and/or the displayed pitch of XR content 604 could be adjusted based on operator preference. Although a substantially vertical line segment 602 is shown for illustrative purposes, similar processes can be performed to display XR content coincident to a horizontal or slanted line segment based on a gaze direction vector and move the XR content along the horizontal or slanted line segment left to right or vice versa in the x-axis and/or z-axis directions for a horizontal line segment) according to the head motion of an operator.

[0074] Figure 7 illustrates another example of displaying XR content, according to various embodiments. As shown, a spline 702 that circumscribes a portion of computer-assisted device 110, shown as a columnar beam of computer-assisted device, is the virtual geometric feature in this example. To display XR content 706, a point on spline 702 that is at or closest to a portion of operator 108, shown as the head of operator 108, is determined. The point is determined by intersecting a gaze direction vector 708 with a surface 704 that is generated by extruding spline 702 along an axis associated with the columnar beam of computer-assisted device 110. XR content 706 is displayed coincident to a point on spline 702 that is closest to the intersection between gaze direction vector 708 and surface 704. In addition, XR content 706 is displayed in an orientation that faces towards operator 108. The displayed position and/or orientation of XR content 706 is also adjusted based on operator preference, such as by adjusting the displayed height, offset from the axis associated with the columnar beam, and/or pitch of XR content 706, as described above in conjunction with Figure 4.

[0075] Figure 8 illustrates another example of displaying XR content, according to various embodiments. As shown, spline 702 is again the virtual geometric feature in this example. Illustratively, gaze direction vector 802 does not intersect surface 704 that is generated by extruding spline 702 along an axis associated with the columnar beam of computer-assisted device 110. Accordingly, XR content 806 is displayed coincident to spline 702 based on a closest point on spline 702 to gaze direction vector 802. The closest point is shown as point 804. Similar to the description above in conjunction with Figure 7, XR content 806 is displayed in an orientation that faces towards operator 108 in a field of view of operator 108, and the displayed position and/or orientation of XR content 806 is adjusted based on operator preference, such as by adjusting the displayed height, offset from the axis associated with the columnar beam, and/or pitch of XR content 806.

[0076] The disclosed techniques display XR content based on the position (and optionally orientation) of a portion of an operator and the position (and optionally orientation) of a portion of an object. The XR content is displayed in positions (and optionally orientations) that are more easily viewable, are more easily interacted with, are more ergonomic, increase visibility or accessibility of the XR content, help increase the efficiency of operations performed with the XR device, reduce operator discomfort, reduce the need for operator repositioning, and/or the like. The displayed XR content can include content to entertain, aid or instruct, present virtual controls, and/or otherwise provide input for or output to the operator.

[0077] Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. Thus, the scope of the invention should be limited only by the following claims, and it is appropriate that the claims be construed broadly and, in a manner, consistent with the scope of the embodiments disclosed herein.

Claims

WHAT IS CLAIMED IS:

1. An extended reality (XR) system comprising: an XR device; and a processor system configured to: determine an operator position of an operator portion of an operator based on first sensor data, determine a first object position of an object portion of an object based on second sensor data, determine a first feature position for a virtual geometric feature based on the first object position, determine a first content position based on the operator position while using the virtual geometric feature located at the first feature position to constrain the first content position, and cause the XR device to display an XR content based on the first content position.

2. The XR system of claim 1, wherein the processor system is further configured to, subsequent to a movement of the object portion from the first object position to a second object position: determine the second object position based on third sensor data; determine a second feature position for the virtual geometric feature based on the second object position; determine a second content position based on the operator position of the operator portion while using the virtual geometric feature located at the second feature position to constrain the second content position; and cause the XR device to display the XR content based on the second content position.

3. The XR system of claim 1, wherein the virtual geometric feature comprises at least one feature selected from the group consisting of: a line portion, a line, a spline portion, a spline, a surface portion, and a surface.

4. The XR system of claim 1, wherein to determine the first content position, the processor system is configured to: determine an intersection between a gaze direction vector associated with a direction of view of the operator and a surface associated with the virtual geometric feature; and determine the first content position based on the intersection and the virtual geometric feature.

5. The XR system of claim 4, wherein the processor system is further configured to: generate the surface based on the virtual geometric feature.

6. The XR system of claim 5, wherein the virtual geometric feature comprises a spline, and wherein to generate the surface, the processor system is configured to: generate a non-planar surface by extruding the spline.

7. The XR system of claim 1, wherein to determine the first content position, the processor system is configured to: determine whether a gaze direction vector associated with a direction of view of the operator intersects a surface associated with the virtual geometric feature; and in response to determining that the gaze direction vector does not intersect the surface, determine the first content position based on a closest point of the virtual geometric feature closest to the gaze direction vector.

8. The XR system of claim 7, wherein the processor system is further configured to determine the closest point by: projecting the virtual geometric feature onto a plane perpendicular to the gaze direction vector; and determining a point of the projected virtual geometric feature that is closest to the gaze direction vector .

9. The XR system of claim 1, wherein when the virtual geometric feature is located at the first feature position and the object portion is at the first object position, the virtual geometric feature is offset by a predefined amount from an axis associated with the object portion.

10. The XR system of claim 1, wherein when the virtual geometric feature is located at the first feature position and the object portion is at the first object position, the virtual geometric feature circumscribes the object portion.

11. The XR system of any one of claims 1 to 10, wherein the processor system is further configured to: determine the virtual geometric feature based on at least one characteristic selected from the group consisting of a type of the object portion and a size of the object portion.

12. The XR system of any one of claims 1 to 10, wherein the processor system is further configured to determine the virtual geometric feature based on a type of the XR content.

13. The XR system of any one of claims 1 to 10, wherein the first content position is further determined based on a height of a portion of the operator.

14. The XR system of any one of claims 1 to 10, wherein the first content position is determined further based on at least one point selected from the group consisting of a center of the XR content, a corner of the XR content, a point along an edge of the XR content, and a point within the XR content.

15. The XR system of any one of claims 1 to 10, wherein the processor system is further configured to, in response to determining that displaying the XR content at the first content position would locate the XR content at a position that, in a field of view of the operator, overlaps, underlays, or intersects another object portion of the object: determine, based on the first content position, at least one adjustment selected from the group consisting of: an adjusted first content position and an adjusted content size, wherein displaying the XR content according to the at least one adjustment reduces an amount of the XR content that overlaps, underlays, or intersects the another object portion; and cause the XR device to display the XR content based on the first content position by: displaying the XR content according to the at least one adjustment.

16. The XR system of claim 15, wherein to determine the adjusted first content position, the processor system is configured to: determine the adjusted first content position to be, in the field of view of the operator, in front of or offset from the another object portion; or determine the adjusted first content position to reduce an intersection between the XR content and the another object portion.

17. The XR system of any one of claims 1 to 10, wherein the processor system is further configured to cause the XR content to be displayed according to one or more predefined rules, the one or more predefined rules including at least one rule selected from the group consisting of a rule specifying that a first type of XR content is to be displayed closer to a center of a field of view of the operator relative to a second type of XR content; and a rule specifying that the XR content is to be displayed entirely within the field of view.

18. The XR system of any one of claims 1 to 10, wherein the processor system is further configured to: determine an operator orientation of the operator portion based on the first sensor data; determine an object orientation of the object portion based on the second sensor data; determine a feature orientation for the virtual geometric feature based on the object orientation; determine a content orientation of the XR content based on the operator orientation while using the virtual geometric feature oriented at the feature orientation to constrain the content orientation; and cause the XR device to display the XR content further based on the content orientation.

19. The XR system of any one of claims 1 to 10, wherein the processor system is further configured to: determine an operator orientation of the operator portion based on the first sensor data; determine a first content orientation for the XR content based on the operator orientation, wherein displaying the XR content at the first content position with the first content orientation would cause the XR content to, in a field of view of the operator, face the operator; and cause the XR device to display the XR content further based on the first content orientation.

20. The XR system of any one of claims 1 to 10, wherein the virtual geometric feature comprises a line segment parallel to an axis associated with the object portion, wherein the operator position comprises a parameter based on a height of a head of the operator, and wherein the first content position comprises a parameter defining a height at which to display the XR content.

21. The XR system of any one of claims 1 to 10, wherein the virtual geometric feature comprises a horizontal line segment, wherein the operator position comprises a parameter based on a position of a head of the operator, and wherein the first content position comprises a parameter defining a position along the horizontal line segment at which to display the XR content.

22. The XR system of any one of claims 1 to 10, wherein: the object comprises a repositionable structure, the repositionable structure comprising a plurality of links coupled by a plurality of joints; the object portion comprises a portion of a link of the plurality of links; and to determine the first object position, the processor system is configured to compute the first object position based on kinematic data.

23. A method of causing an extended reality (XR) device to display XR content, the method comprising: determining, by a processor system, an operator position of an operator portion of an operator based on first sensor data; determining, by the processor system, a first object position of an object portion of an object based on second sensor data; determining, by the processor system, a first feature position for a virtual geometric feature based on the first object position; determining, by the processor system, a first content position based on the operator position while using the virtual geometric feature located at the first feature position to constrain the first content position; and causing, by the processor system, the XR device to display the XR content based on the first content position.

24. The method of claim 23, further comprising, subsequent to a movement of the object portion from the first object position to a second object position: determining, by the processor system, the second object position based on third sensor data; determining, by the processor system, a second feature position for the virtual geometric feature based on the second object position; determining, by the processor system, a second content position based on the operator position of the operator portion while using the virtual geometric feature located at the second feature position to constrain the second content position; and causing, by the processor system, the XR device to display the XR content based on the second content position.

25. The method of claim 23, wherein the virtual geometric feature comprises at least one feature selected from the group consisting of: a line portion, a line, a spline portion, a spline, a surface portion, and a surface.

26. The method of claim 23, wherein determining the first content position comprises: determining an intersection between a gaze direction vector associated with a direction of view of the operator and a surface associated with the virtual geometric feature; and determining the first content position based on the intersection and the virtual geometric feature.

27. The method of claim 26, further comprising: generating, by the processor system, the surface based on the virtual geometric feature.

28. The method of claim 23, wherein determining the first content position comprises: determining, by the processor system, whether a gaze direction vector associated with a direction of view of the operator intersects a surface associated with the virtual geometric feature; and in response to determining that the gaze direction vector does not intersect the surface, determining, by the processor system, the first content position based on a closest point of the virtual geometric feature closest to the gaze direction vector.

29. The method of any of claims 23 to 28, further comprising: determining, by the processor system, the virtual geometric feature based on at least one characteristic selected from the group consisting of: a type of the object portion and a size of the object portion.

30. The method of any of claims 23 to 28, further comprising: determining, by the processor system, the virtual geometric feature based on a type of the XR content.

31. The method of any of claims 23 to 28, wherein the first content position is further determined based on a height of a portion of the operator.

32. The method of any of claims 23 to 28, further comprising, in response to determining that displaying the XR content at the first content position would locate the XR content at a position that, in a field of view of the operator, overlaps, underlays, or intersects another object portion of the object: determining, by the processor system and based on the first content position, at least one adjustment selected from the group consisting of: an adjusted first content position and an adjusted content size, wherein displaying the XR content according to the at least one adjustment reduces an amount of the XR content that overlaps, underlays, or intersects the another object portion; and wherein causing the XR device to display the XR content based on the first content position comprises: displaying the XR content according to the at least one adjustment.

33. The method of any of claims 23 to 28, further comprising: determining, by the processor system, an operator orientation of the operator portion based on the first sensor data; determining, by the processor system, an object orientation of the object portion based on the second sensor data; determining, by the processor system, a feature orientation for the virtual geometric feature based on the object orientation; determining, by the processor system, a content orientation of the XR content based on the operator orientation while using the virtual geometric feature oriented at the feature orientation to constrain the content orientation; and causing, by the processor system, the XR device to display the XR content further based on the content orientation.

34. The method of claim 23, further comprising: determining, by the processor system, an operator orientation of the operator portion based on the first sensor data determining, by the processor system, a first content orientation for the XR content based on the operator orientation, wherein displaying the XR content at the first content position with the first content orientation would cause the XR content to, in a field of view of the operator, face the operator; and causing, by the processor system, the XR device to display the XR content further based on the first content orientation.

35. The method of any of claims 23 to 28 wherein the virtual geometric feature comprises a line segment parallel to an axis associated with the object portion, wherein the operator position comprises a parameter based on a height of a head of the operator, and wherein the first content position comprises a parameter defining a height at which to display the XR content.

36. The method of any of claims 23 to 28, wherein the virtual geometric feature comprises a horizontal line segment, wherein the operator position comprises a parameter based on a position of a head of the operator, and wherein the first content position comprises a parameter defining a position along the horizontal line segment at which to display the XR content.

37. One or more non-transitory machine-readable media comprising a plurality of machine-readable instructions which when executed by one or more processors are adapted to cause the one or more processors to perform the method of any one of claims 23-36.