JP2023517954A - Systems and methods for multi-user virtual and augmented reality - Google Patents

Abstract

Apparatus for providing virtual content in an environment in which a first and second user can interact with each other, the apparatus comprising a first display screen and/or a second display screen worn by the first user. a communication interface configured to communicate with a second display screen worn by a user of a; obtaining a first location of the first user; , determine a first set of anchor points, obtain a second location of the second user, determine a second set of anchor points based on the second location of the second user, and determining one or more common anchor points within both the first set and the second set; based on at least one of the one or more common anchor points, the first user and/or a processing unit configured to provide virtual content for experience by a second user.

Description

TECHNICAL FIELD The present disclosure relates to computing, learning network configurations, and connected mobile computing systems, methods, and configurations, and more particularly at least one wearable that can be utilized for virtual and/or augmented reality operations. Mobile computing systems, methods, and configurations featuring components.

Modern computing and display technologies are driving the development of “mixed reality” (MR) systems for so-called “virtual reality” (VR) or “augmented reality” (AR) experiences, which are digitally reproduced. The image, or portion thereof, is presented to the user in a manner that appears or can be perceived as real. VR scenarios typically involve the presentation of digital or virtual image information without transparency to other real-world visual inputs. AR scenarios typically involve the presentation of digital or virtual image information as an extension to the real-world visualization of the user's surroundings (ie, transparency to real-world visual input). AR scenarios thus involve the presentation of digital or virtual image information with transparency to real-world visual input.

MR systems can generate and display color data, which increases the realism of MR scenarios. Many of these MR systems operate by sequentially projecting sub-images in different (e.g., primary) colors or "fields" (e.g., red, green, and blue) corresponding to a color image in rapid succession. Display color data. Projecting the color sub-images at a sufficiently high rate (eg, 60 Hz, 120 Hz, etc.) can result in a smooth color MR scenario in the user's memory.

Various optical systems produce images, including color images, at various depths to display MR (VR and AR) scenarios.

MR systems employ at least wearable display devices (e.g., head-mounted displays, helmet-mounted displays, or smart glasses) that are loosely coupled to the user's head and thus move as the user's head moves. can. When a user's head motion is detected by the display device, the displayed data is updated (e.g., "warping").

As an example, if a user wearing a head-mounted display device views a virtual representation of a virtual object on the display and walks around the area in which the virtual object appears, the virtual object is It can give the user the perception that they are walking around objects that are rendered and occupy real space. When a head-mounted display device is used to present multiple virtual objects, head pose measurements match the user's dynamically changing head poses to provide increased immersion. can be used to render the scene as

AR-enabled head-mounted display devices provide simultaneous viewing of both real and virtual objects. An "optical see-through" display allows a user to see through transparent (eg, semi-transparent or fully transparent) elements in the display system and directly see light from real objects in the environment. The transparent element, often referred to as a "combiner," superimposes the light from the display across the user's view of the real world, and the light from the display spreads the image of the virtual content across the see-through view of real objects in the environment. projected. A camera may be mounted on the head-mounted display device to capture images or video of the scene being viewed by the user.

Current optical systems, such as those in MR systems, optically render virtual content. Content is "virtual" in that it does not correspond to actual physical objects located at discrete locations in space. Instead, virtual content exists only within the head-mounted display device user's brain (eg, the visual center) when stimulated by light beams directed at the user's eyes.

In some cases, the head-mounted image display device may display virtual objects relative to the real environment and/or allow users to place and/or manipulate virtual objects relative to the real environment. . In such cases, the image display device may be configured to locate the user relative to the real environment so that the virtual object can be correctly displaced relative to the real environment.

A mixed reality or augmented reality eyepiece display should be lightweight, low cost, have a small form factor, have a wide virtual image field of view, and be as transparent as possible. In addition, to be practical for a variety of use cases without exceeding acceptable tolerances for convergence-divergence motion-accommodative mismatch, virtual image information may be presented in multiple focal planes (e.g., two planes). or more).

It would also be desirable to have new techniques for presenting virtual objects to a user's view such that the virtual objects can be positioned accurately relative to the physical environment when viewed by the user. deaf. In some cases, when a virtual object is placed virtually with respect to a physical environment located far from the user, the virtual object may be offset, or "drift" away from its intended location. can. This can occur because the local coordinate frame for the user is correctly aligned with features in the physical environment, but may not match exactly with other features in the physical environment that are more distant from the user. .

Methods and apparatus for presenting virtual content, such as virtual objects, for display by one or more screens of one or more image display devices (worn by one or more users) are described herein. In some embodiments, virtual content may be displayed such that it appears to be within a physical environment when viewed through a screen by a user. Virtual content may be provided based on one or more anchor points that are aligned with the physical environment. In some embodiments, virtual content may be provided as a movable object, and the position of the movable object may be based on one or more anchor points in close proximity to the action of the movable object. This allows the object to be virtually placed accurately with respect to the user (when viewed by the user through the screen they are wearing) even if the object is far from the user. In gaming applications, such features may allow multiple users to interact with the same object, even if the users are relatively far apart. For example, in a gaming application, virtual objects may be virtually passed back and forth between users. Placement (orientation) of virtual objects based on anchor point proximity as described herein prevents offset and drift issues, thus allowing virtual objects to be accurately positioned.

A device for providing virtual content in an environment in which a first user and a second user can interact with each other includes a first display screen worn by a first user and/or or a communication interface configured to communicate with a second display screen worn by a second user; and a processing unit for obtaining a first position of the first user; determining a first set of one or more anchor points based on the first location of the user; obtaining a second location of the second user; based on, determine a second set of one or more anchor points, determine one or more common anchor points that are within both the first set and the second set, determine one and a processing unit configured to provide virtual content for experience by the first user and/or the second user based on at least one of the or more common anchor points.

Optionally, the one or more common anchor points comprise a plurality of common anchor points, and the processing unit is configured to select a subset of common anchor points from the plurality of common anchor points.

Optionally, the processing unit is configured to select a subset of common anchor points to reduce localization errors of the first user and the second user relative to each other.

Optionally, the one or more common anchor points comprise a single common anchor point.

Optionally, the processing unit is configured to position and/or orient the virtual content based on at least one of the one or more common anchor points.

Optionally, each of the one or more anchor points in the first set is a point in a Persistent Coordinate Frame (PCF).

Optionally, the processing unit is configured to provide the virtual content for display as a movable virtual object within the first display screen and/or the second display screen.

Optionally, the processing unit is configured to display the virtual object within the first display screen for display such that the virtual object appears to move within a space between the first user and the second user. configured to provide

Optionally, the one or more common anchor points comprise a first common anchor point and a second common anchor point, and the processing unit determines that the movable virtual object is the first common anchor point relative to the first display screen. configured to present the movable virtual object for display within the first display screen to have an object position and a second object position relative to the first display screen; The object position is based on the first common anchor point and the second object position of the movable virtual object is based on the second common anchor point.

Optionally, the processing unit is configured to select a first common anchor point for placing the virtual object at the first object position based on where the action of the virtual object occurs.

Optionally, the one or more common anchor points comprise a single common anchor point, and the processing unit determines that the movable virtual object is at the first object position relative to the first display screen and at the first object position relative to the first display screen. a second object position, wherein the first object position of the movable virtual object is at a single common anchor point; Based, the second object position of the movable virtual object is based on the single common anchor point.

Optionally, the one or more common anchor points comprises a plurality of common anchor points, and the processing unit activates one of the common anchor points to place the virtual content within the first display screen. configured to select.

Optionally, the processing unit establishes a common anchor for placing the virtual content by selecting one of the common anchor points closest to or within a distance threshold from the action of the virtual content. configured to select one of the points.

Optionally, the position and/or movement of the virtual content is controllable by the first user's first handheld device.

Optionally, the position and/or movement of the virtual content can also be controlled by the second user's second handheld device.

Optionally, the processing unit is configured to localize the first user and the second user to the same mapping information based on one or more common anchor points.

Optionally, the processing unit causes the first display screen to display the virtual content such that the virtual content would appear in a spatial relationship to physical objects in the first user's surroundings. Configured.

Optionally, the processing unit is configured to obtain the one or more sensor inputs, the processing unit determining, based on the one or more sensor inputs, the first user's intent with the virtual content. configured to assist in accomplishing

Optionally, the one or more sensor inputs are indicative of the first user's eye gaze direction, arm kinematics, body position, body orientation, or any combination of the foregoing.

Optionally, the processing unit is configured to assist the first user in accomplishing a goal by applying one or more limits on position and/or angular velocity of system components.

Optionally, the processing unit is configured to assist the first user in accomplishing a goal by gradually reducing the distance between the virtual content and another element.

Optionally, the processing unit comprises a first processing portion in communication with the first display screen and a second processing portion in communication with the second display screen.

virtual content in an environment in which a first user wearing a first display screen and a second user wearing a second display screen can interact with each other; A method implemented by an apparatus configured to provide: obtaining a first location of a first user; and based on the first location of the first user, one or more determining a first set of overriding anchor points; obtaining a second location of the second user; and one or more anchor points based on the second location of the second user. determining one or more common anchor points within both the first set and the second set; determining one or more common anchor points in both the first set and the second set; providing virtual content for experience by the first user and/or the second user based on at least one of:

Optionally, the one or more common anchor points comprise a plurality of common anchor points, and the method further comprises selecting a subset of common anchor points from the plurality of common anchor points.

Optionally, the subset of common anchor points is selected to reduce localization errors of the first user and the second user with respect to each other.

Optionally, the one or more common anchor points comprise a single common anchor point.

Optionally, the method further includes determining a position and/or orientation for the virtual content based on at least one of the one or more common anchor points.

Optionally, each of the one or more anchor points in the first set is a point in a Persistent Coordinate Frame (PCF).

Optionally, the virtual content is provided as movable virtual objects within the first display screen and/or the second display screen for display.

Optionally, the virtual object is provided for display such that it appears within the first display screen as if the virtual object were moving in space between the first user and the second user. be.

Optionally, the one or more common anchor points comprises a first common anchor point and a second common anchor point, wherein the movable virtual object is for display in the first display screen: A movable virtual object is provided having a first object position relative to the first display screen and a second object position relative to the first display screen, the first object position of the movable virtual object being the first A second object position of the movable virtual object is based on the second common anchor point.

Optionally, the method further includes selecting a first common anchor point for placing the virtual object at the first object position based on where the action of the virtual object occurs.

Optionally, the one or more common anchor points comprises a single common anchor point, the moveable virtual object is for display in the first display screen, the moveable virtual object is in the first display screen. and a second object position relative to the first display screen, wherein the first object position of the movable virtual object is based on the single common anchor point and the position of the movable virtual object A second object position is based on a single common anchor point.

Optionally, the one or more common anchor points comprises a plurality of common anchor points, the method further comprising selecting one of the common anchor points for placing the virtual content within the first display screen. Including choosing.

Optionally, the act of selecting comprises selecting one of the common anchor points closest to the action of the virtual content or within a distance threshold from the action of the virtual content.

Optionally, the position and/or movement of the virtual content is controllable by the first user's first handheld device.

Optionally, the position and/or movement of the virtual content can also be controlled by the second user's second handheld device.

Optionally, the method further comprises localizing the first user and the second user to the same mapping information based on one or more common anchor points.

Optionally, the method further includes displaying the virtual content with the first display screen such that the virtual content appears in a spatial relationship to physical objects in the first user's surroundings.

Optionally, the method further comprises obtaining one or more sensor inputs and assisting the first user in accomplishing a goal with the virtual content based on the one or more sensor inputs. including doing.

Optionally, the one or more sensor inputs are indicative of the first user's eye gaze direction, arm kinematics, body position, body orientation, or any combination of the foregoing.

Optionally, the act of assisting the first user in accomplishing the objective includes applying one or more limits on position and/or angular velocity of system components.

Optionally, the act of assisting the first user in accomplishing the objective includes gradually reducing the distance between the virtual content and another element.

Optionally, the device comprises a first processing portion in communication with the first display screen and a second processing portion in communication with the second display screen.

A processor-readable non-transitory medium stores a set of instructions, execution of which by a processing unit causes a method to be performed, the processing unit in which a first user and a second user A portion of a device configured to provide virtual content within an environment capable of interacting with each other, the method comprising: obtaining a first location of a first user; determining a first set of one or more anchor points based on a first location of one user; obtaining a second location of a second user; determining a second set of one or more anchor points based on a second position of and one or more common Determining anchor points and providing virtual content for experience by the first user and/or the second user based on at least one of the one or more common anchor points. including.

Additional and other objects, features, and advantages of the present disclosure are set forth in the detailed description, figures, and claims.

The drawings illustrate the design and utility of various embodiments of the present disclosure. Note that the figures are not drawn to scale and that elements of similar structure or function are represented by similar reference numerals throughout the figures. To better understand how to obtain the aforementioned and other advantages and objectives of the various embodiments of the present disclosure, a more detailed description of the disclosure, briefly described above, will be presented with the specifics thereof illustrated in the accompanying drawings. will be given by reference to the embodiments. With the understanding that these drawings depict only typical embodiments of the disclosure and are therefore not to be considered limiting of its scope, the present disclosure will be illustrated with additional specificity through the use of the accompanying drawings. and will be explained and described in detail.

FIG. 1A illustrates an image display system having an image display device, according to some embodiments.

FIG. 1B illustrates an image display device that displays frames in multiple depth planes.

FIG. 2 illustrates a method, according to some embodiments.

FIG. 3 illustrates a method, according to some embodiments.

FIG. 4 illustrates a method, according to some embodiments.

5A-5L illustrate examples of two users interacting with each other within a virtual or augmented environment. 5A-5L illustrate examples of two users interacting with each other within a virtual or augmented environment. 5A-5L illustrate examples of two users interacting with each other within a virtual or augmented environment. 5A-5L illustrate examples of two users interacting with each other within a virtual or augmented environment. 5A-5L illustrate examples of two users interacting with each other within a virtual or augmented environment. 5A-5L illustrate examples of two users interacting with each other within a virtual or augmented environment. 5A-5L illustrate examples of two users interacting with each other within a virtual or augmented environment. 5A-5L illustrate examples of two users interacting with each other within a virtual or augmented environment. 5A-5L illustrate examples of two users interacting with each other within a virtual or augmented environment. 5A-5L illustrate examples of two users interacting with each other within a virtual or augmented environment. 5A-5L illustrate examples of two users interacting with each other within a virtual or augmented environment. 5A-5L illustrate examples of two users interacting with each other within a virtual or augmented environment.

FIG. 6 illustrates an example of two users interacting with each other within a virtual or augmented environment based on a single anchor point.

7A-7D illustrate examples of two users interacting with each other within a virtual or augmented environment based on multiple anchor points. 7A-7D illustrate examples of two users interacting with each other within a virtual or augmented environment based on multiple anchor points. 7A-7D illustrate examples of two users interacting with each other within a virtual or augmented environment based on multiple anchor points. 7A-7D illustrate examples of two users interacting with each other within a virtual or augmented environment based on multiple anchor points.

FIG. 8 illustrates a method, according to some embodiments.

FIG. 9 illustrates a processing unit of an apparatus, according to some embodiments.

FIG. 10 illustrates a method, according to some embodiments.

FIG. 11 illustrates a specialized processing system, according to some embodiments.

DETAILED DESCRIPTION Various embodiments of the present disclosure are directed to methods, apparatus, and articles of manufacture for providing input for head-mounted video imaging devices. Other objects, features, and advantages of the present disclosure are set forth in the detailed description, figures, and claims.

Various embodiments are described below with reference to the figures. Note that the figures are not drawn to scale and that elements of similar structure or function are represented by like reference numerals throughout the figures. Also, it should be noted that the figures are only intended to facilitate the description of the embodiments. They are not intended as a comprehensive description of the invention or as limitations on the scope of the invention. Additionally, an illustrated embodiment need not have all the aspects or advantages shown. Aspects or advantages described in conjunction with a particular embodiment are not necessarily limited to that embodiment, nor are any other implementations, even if not so illustrated or explicitly described as such. can be practiced in the form

The description that follows relates to illustrative VR, AR, and/or MR systems with which the embodiments described herein can be practiced. However, embodiments are also suitable for use in other types of display systems (including other types of VR, AR, and/or MR systems), and thus embodiments are disclosed herein. It should be understood that it is not limited to only the illustrative examples given.

Referring to FIG. 1A, a head-mounted viewing component (image display device) 2, a handheld controller component 4, and an interconnected auxiliary computer that may be configured to be worn on the user as a beltpack or equivalent. An augmented reality system 1 is shown, featuring a lighting or controller component 6 . Each of these components communicates with each other and cloud computing or It may be operably coupled (10, 12, 14, 16, 17, 18) to other connected resources 8, such as cloud storage resources. Using various embodiments such as two depicted optical elements 20, the user can see the world around them along with visual components that can be produced by associated system components for an augmented reality experience. . As illustrated in FIG. 1A, such a system 1 also includes, but is not limited to, various camera type sensors (monochrome, color/RGB, and/or thermal imaging components, etc.) (22, 24, 26). , a depth camera sensor 28, and/or a sound sensor 30, such as a microphone, configured to provide information about the user's surrounding environment. A need exists for compact and persistently connected wearable computing systems and assemblies such as those described herein that can be utilized to provide users with a perceptually rich augmented reality experience.

System 1 also includes apparatus 7 for providing input for image display device device 2 . Device 7 will be described in more detail below. The image display device 2 may be either a VR device, AR device, MR device, or other type of display device. As shown in the figure, the image display device 2 includes a frame structure worn by the end user and a display subsystem carried by the frame structure such that the display subsystem is positioned in front of the end user's eyes. and a speaker is positioned adjacent to the end-user's ear canal (optionally, another speaker (not shown) is positioned adjacent to the end-user's other ear canal to provide stereo/adjustable sound control ) and a loudspeaker carried by the frame structure. The display subsystem is designed to present light patterns to the end-user's eye that can be comfortably perceived as an extension to physical reality, with a high level of image quality and three-dimensional perception, and to display two-dimensional content. It is also possible to present A display subsystem presents a sequence of frames at high frequency that provides the perception of a single coherent scene.

In the illustrated embodiment, the display subsystem employs an "optical see-through" display through which a user can view light from real objects directly through transparent (or translucent) elements. . A transparent element, often referred to as a "combiner," superimposes light from the display across the user's view of the real world. To this end, the display subsystem comprises a partially transparent display or a fully transparent display. The display is positioned in the end user's field of view between the end user's eyes and the surrounding environment such that direct light from the surrounding environment is transmitted through the display to the end user's eyes.

In the illustrated embodiment, the image projection assembly provides light to the partially transparent display such that it is combined with direct light from the surrounding environment and transmitted from the display to the user's eyes. The projection subsystem may be a fiber optic scanning-based projection device, and the display may be a single unit into which the scanned light from the projection subsystem is, for example, closer than infinity (eg, arm's length). Injected to produce images at one optical viewing distance, images at multiple discrete optical viewing distances or focal planes, and/or image layers stacked at multiple viewing distances or focal planes to represent a stereoscopic 3D object. , may be a waveguide-based display. These layers in the light field may be stacked close enough together to appear persistent to the human sub-visual system (i.e., one layer is within the disturbance cone of an adjacent layer). . Additionally or alternatively, picture elements are blended across two or more layers, the layers being more sparsely stacked (i.e., one layer overlaps the disturbance cone of the adjacent layer). It may increase the perceived continuity of transitions between layers in the light field, even when outside the gamut). The display subsystem may be monocular or binocular.

The image display device 2 may also include one or more sensors mounted to the frame structure for detecting the position and movement of the end user's head and/or the end user's eye position and inter-eye distance. good. Such sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and/or gyroscopes), or any combination of the foregoing. Many of these sensors operate on the assumption that the frames on which they are affixed are, in turn, substantially fixed to the user's head, eyes, and ears.

Image display device 2 may also include a user orientation detection module. A user orientation module detects the instantaneous position of the end-user's head (e.g., via a sensor coupled to the frame) and predicts the position of the end-user's head based on the position data received from the sensor. You may Detecting the instantaneous position of the end-user's head facilitates the determination of the specific real object that the end-user is looking at and thereby the specific virtual object to be generated in relation to that real object. It provides an indication of the object and also provides an indication of the location where the virtual object will be displayed. The user orientation module may also track the end user's eyes based on tracking data received from the sensors.

Image display device 2 may also include a control subsystem, which may take any of a wide variety of forms. The control subsystem includes several controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), application specific integrated circuits (ASICs). , programmable gate arrays (PGAs), eg, field PGAs (FPGAs), and/or programmable logic controllers (PLUs).

The control subsystem of image display device 2 includes a central processing unit (CPU), a graphics processing unit (GPU), one or more frame buffers, and a 3D database for storing 3D scene data. may contain. The CPU may control the overall operation, while the GPU renders frames from the 3D data stored in the 3D database (i.e., transforms the 3D scene into a 2D image) and renders these Frames may be stored in a frame buffer. One or more additional integrated circuits may control the loading of frames into and out of the frame buffer and the operation of the image projection assembly of the display subsystem.

Device 7 represents various processing components for system 1 . In the figure the apparatus 7 is shown as part of the image display device 2 . In other embodiments, device 7 may be implemented within handheld controller component 4 and/or controller component 6 . In further embodiments, the various processing components of device 7 may be implemented within distributed subsystems. For example, the processing component of apparatus 7 may be one of image display device 2, handheld controller component 4, controller component 6, or another device (that communicates with image display device 2, handheld controller component 4, and/or controller component 6). may be located within two or more of

Couplings 10, 12, 14, 16, 17, 18 between the various components described above provide one or more wired interfaces or ports to provide wire or optical communication, or wireless communication. may include one or more wireless interfaces or ports, such as via RF, microwave, and IR, for In some implementations all communications may be wired, while in other implementations all communications may be wireless. Therefore, the particular choice of wired or wireless communication should not be considered limiting.

Some image display systems (e.g., VR systems, AR systems, MR systems, etc.) incorporate multiple volume phase holograms, surface Using relief holograms, or light directing optical elements. In other words, a diffraction pattern or diffractive optical element (“DOE”) causes collimated light (a light beam with a substantially plane wavefront) to be diffracted at multiple locations as it is substantially totally internally reflected along the LOE. embedded within or imprinted/embossed onto a light guiding optical element ("LOE", e.g., planar waveguide) to intersect the pattern and emit toward the user's eye good. The DOE is configured such that light exiting the LOE through it is converged so that it appears to originate from a particular depth plane. Collimated light may be produced using an optical collection lens (“concentrator”).

For example, the first LOE may be configured to deliver collimated light to the eye that appears to originate from an optical infinity depth plane (0 diopters). Another LOE may be configured to deliver collimated light that appears to originate from a distance of 2 meters (1/2 diopter). Yet another LOE may be configured to deliver collimated light that appears to originate from a distance of 1 meter (1 diopter). It should be appreciated that by using stacked LOE assemblies, multiple depth planes can be created, each LOE configured to display an image that appears to originate from a particular depth plane. It should be appreciated that the stack may contain any number of LOEs. However, at least N stacked LOEs are required to generate N depth planes. Additionally, N, 2N, or 3N stacked LOEs may be used to generate an RGB color image in N depth planes.

In order to present 3D virtual content to a user, an image display system 1 (e.g., VR system, AR system, MR system, etc.) is configured such that they are varied in the Z direction (i.e., orthogonally away from the user's eyes). An image of the virtual content is projected into the user's eye such that it appears to originate from the depth plane of . In other words, not only can the virtual content change in the X and Y directions (i.e., the 2D plane, orthogonal to the central visual axis of the user's eye), but it can also change when the user sees objects in very close proximity. Or it can appear to vary in the Z direction as well, so that it can be perceived as being at infinite distance, or any distance in between. In other embodiments, the user may perceive multiple objects at different depth planes simultaneously. For example, it may appear to the user that a virtual dragon appears from infinity and runs towards the user. Alternatively, the user may simultaneously see a virtual bird at a distance of 3 meters from the user and a virtual coffee cup at arm's length (approximately 1 meter) from the user.

Multi-focus systems create the perception of variable depth by projecting images onto some or all of multiple depth planes located at discrete fixed distances in the Z direction from the user's eyes. Referring now to FIG. 1B, it should be appreciated that the multi-plane focus system may display frames in fixed depth planes 150 (eg, the six depth planes 150 shown in FIG. 1B). Although an MR system can include any number of depth planes 150, one exemplary multiplanar focal system has six fixed depth planes 150 in the Z direction. When generating virtual content in one or more of the six depth planes 150, 3D perception is such that the user perceives one or more virtual objects at variable distances from the user's eyes. created. Given that the human eye is more sensitive to objects that are close in distance than objects that appear to be far away, more depth planes 150 are generated closer to the eye, as shown in FIG. 1B. . In some embodiments, the depth planes 150 may be placed equidistantly away from each other.

Depth plane position 150 may be measured in diopters, a unit of refractive power equal to the reciprocal of the focal length measured in meters. For example, in some embodiments, depth plane 1 may be 1/3 diopters away, depth plane 2 may be 0.3 diopters away, and depth plane 3 may be 0.2 diopters away. Alternatively, depth plane 4 may be 0.15 diopters away, depth plane 5 may be 0.1 diopters away, and depth plane 6 may represent infinity (ie, 0 diopters away). It should be appreciated that other embodiments may generate the depth plane 150 at other distances/diopters. Therefore, when generating virtual content in strategically placed depth planes 150, the user is able to perceive virtual objects in three dimensions. For example, a user may perceive it to be near when a first virtual object appears in depth plane 1 while another virtual object appears at infinity in depth plane 6 . Alternatively, the virtual object may be displayed first in depth plane 6, then in depth plane 5, and so on until the virtual object appears very close to the user. It should be understood that the above examples have been significantly simplified for illustrative purposes. In another embodiment, all six depth planes may be centered on a particular focal length away from the user. For example, if the virtual content to be displayed is a coffee cup 0.5 meters away from the user, all six depth planes are generated at different cross-sections of the coffee cup to provide the user with a high-grain image of the coffee cup. A 3D view can be given.

In some embodiments, the image display system 1 (eg, VR system, AR system, MR system, etc.) may function as a multi-focus system. In other words, all six LOEs may be illuminated simultaneously such that images appearing to arise from six fixed depth planes are generated in rapid succession, and the light sources provide image information to LOE1, LOE1, It then rapidly propagates to LOE2, then LOE3, and so on. For example, a portion of the desired image, including an image of the sky at optical infinity, may be thrown in at time 1, and the LOE (e.g., depth plane 6 from FIG. 1B), which reserves light collimation, is may be used. An image of a closer tree branch may then be thrown in at time 2, configured to create an image that appears to arise from a depth plane 10 meters away, LOE (e.g., depth from FIG. 1B). Face 5) may be utilized. An image of the pen may then be thrown in at time 3, and LOE may be utilized, configured to create an image that appears to originate from a depth plane one meter away. This type of paradigm can be repeated in a fast time-sequential (eg, 360 Hz) fashion so that the user's eyes and brain (eg, visual cortex) perceive the input as being all parts of the same image.

The image display system 1 may project images (i.e. light beam by diverging or converging ). As used herein, a beam of light includes, but is not limited to, a directional projection of light energy (including visible and invisible light energy) emitted from a light source. Generating images that appear to arise from different depth planes matches the user's eye convergence-divergence and accommodation for that image, minimizing convergence-divergence-accommodation conflicts. To or eliminate.

In some cases, a localization map of the environment is obtained to localize the user of the head-mounted image display device relative to the user's environment. In some embodiments, the location map may be stored in a non-transitory medium that is part of system 1 . In other embodiments, the location map may be received wirelessly from a database. After the localization map is acquired, real-time input images from the camera system of the image display device are then matched against the localization map to locate the user. For example, corner features of the input image may be detected from the input image and matched against the corner features of the localization map. In some embodiments, to obtain a set of corners as features from an image for use in localization, the image must first undergo corner detection to obtain an initial set of detected corners. can be. The initial set of detected corners may then be further processed, e.g., undergo non-maximum suppression, spatial binning, etc. to obtain a final set of detected corners for localization purposes. In some cases, filtering may be performed to identify a subset of detected corners within the initial set to obtain a final set of corners.

Also, in some embodiments, the localization map of the environment may be dictated by the user pointing the image display device 2 in different directions (e.g., turning their head while wearing the image display device 2). ), may be created. As the image display device 2 is directed to different spaces within the environment, sensors on the image display device 2 sense properties of the environment, which properties are then used by the system 1 to create a localization map. obtain. In one implementation, the sensors may include one or more cameras and/or one or more depth sensors. The camera provides camera images, which are processed by device 7 to identify different objects in the environment. Additionally or alternatively, depth sensors provide depth information, which is processed by the device to determine different surfaces of objects in the environment.

In various embodiments, a user may be wearing an augmented reality system, such as the one depicted in FIG. may also be referred to as a "spatial computing" system, in connection with the interaction of Such a system may, for example, comprise a head wearable display component 2 and may feature environmental sensing capabilities, such as various types of cameras, which map the environment around the user or can be used to map walls, walls, etc. It may be configured to create a "mesh" of such an environment, with different points representing the geometry of different objects in the user's surrounding environment, such as floors, chairs, and the like. . Spatial Computing Systems, which map or mesh the environment around a user, are developed by Magic Leap, Inc. (Planation, Florida) that utilizes a map or mesh of the room to allow the user to navigate in the three-dimensional space around the user. may be configured to assist in placing, manipulating, visualizing, creating, and modifying various objects and elements in . Referring back to FIG. 1A, the system may be operatively coupled to additional resources, such as other computing systems, by cloud or other connectivity arrangements. One of the challenges in spatial computing is, for example, in making useful and/or important decisions for a user, such as in computer vision and/or object recognition tasks, which may relate to the three-dimensional world around the user. , relates to the use of data captured by various operably coupled sensors (such as elements 22, 24, 26, 28 of the system of FIG. 1A).

For example, referring to FIG. 2, a system such as that illustrated in FIG. ) is illustrated. A first user (which may be referred to as "User 1") boots up its ML1 system and mounts a head-mounted component 2 on its head, and ML1 configures the head-mounted component 2. sensors to scan the local environment around the user's head 1 and perform a simultaneous localization and mapping (known as "SLAM") activity to create a local map of the environment around the user's 1 head or It may also be configured to create a mesh (which in this scenario may be referred to as "Local Map 1"), which User 1 can use to determine its real or near real-time position and orientation relative to the local environment. , can be "located" in this local map 1 by the SLAM activity (40). Referring again to FIG. 2, User 1 navigates around the environment, sees and interacts with real and virtual objects, and uses ongoing SLAM activity to map/mesh neighborhoods of the environment. , and generally reap the benefits of their spatial computing (42). Referring to FIG. 3, an additional step and arrangement is that User 1 encounters one of more predetermined anchor points or points in what may be known as a "persistent coordinate frame" or "PCF". These anchor points and/or PCFs may be known to User 1's local ML1 system by previous installations and/or User 1 may be added to a cloud-based map (local Map 1, which may be larger and/or more refined) via cloud connectivity (i.e. connected resources such as those illustrated in FIG. 1A, i.e. edge (44) by element 8), which may be computing, cloud computing, and other connected resources. Referring again to FIG. 3, anchor points and/or PCFs are utilized to locate various virtual objects or assets intentionally placed by others for User 1 (e.g., for users hiking, It may assist User 1 in his spatial computing tasks, such as by displaying virtual signs (such as virtual signs indicating the presence of sinkholes in the hiking trail) at given fixed locations near the user. (46).

Referring to FIG. 4, a multi-user (or "multi-player" in a game scenario) configuration similar to that described above with reference to FIG. , mounting it in a head-mounted configuration, ML1 scans the environment around the user's head 1, performs SLAM activities, and generates a local map or mesh of the environment around the user's 1 head ("local Map 1"), and User 1 is 'located' in the local map 1 by the SLAM activity so that its real or near-real-time position and orientation can be determined relative to the local environment (40 ). User 1 may encounter one of a number of predetermined anchor points or points within what may be known as a "persistent coordinate frame" or "PCF". These anchor points and/or PCFs may be known to User 1's local ML1 system due to previous installations and/or User 1 may use a cloud-based map (larger and/or more refined than Local Map 1). may be known (48) via cloud connectivity. A separate user "User 2" may boot up another ML1 system and mount it in a head-mounted configuration. This second ML1 system scans the environment around User 2's head, performs SLAM activities, and creates a local map or mesh ("Local Map 2") of the environment around User 2's head. User 2 is then "located" 50 in local map 2 by the SLAM activity so that its real or near real-time position and orientation can be determined relative to the local environment. Similar to User 1, User 2 may encounter one of more predetermined anchor points or points within what may be known as a "persistent coordinate frame" or "PCF". These anchor points and/or PCFs may be known to User 2's local ML1 system due to previous installations and/or User 2 may use a cloud-based map (larger than Local Map 1 or Local Map 2 and/or or may be more refined), known through cloud connectivity (52). Referring again to FIG. 4, User 1 and User 2 may be physically close enough that their ML1 systems begin to encounter common anchor points and/or PCFs. From an overlapping set between two users, using resources such as cloud computing connection resources 8, the system may be configured to select a subset of anchor points and/or PCFs, which is the user's Minimizing localization error with respect to each other, this subset of anchor points and/or PCFs may be utilized to position and orient virtual content for users within a common experience, such that certain content and/or virtual Assets may be experienced by both users from each of their unique perspectives, along with position and orientation localization with respect to the Handheld 4 and other components configured to form part of such a common experience. good.

5A-5L, an exemplary common, collaborative, or multi-user experience is available from Magic Leap, Inc. under the trademark "Pancake Pals"(R). (Plantation, Florida) in the form of a pancake flip game. Referring to FIG. 5A, a first user (which may also be referred to as "user 1"), element 60, has at one end an office environment 66, head-mounted component 2, interconnected auxiliary computing Or shown holding a handheld component 4 fitted with a controller component 6 and making up an ML1 system similar to that shown in FIG. 1A. User 1 60's system displays for him a virtual frying pan element 62 extending from the handheld component 4 in virtual form as if the handheld component 4 were the handle of the frying pan element 62. configured as User 1 60's system is configured to reposition and reorient virtual frying pan 62 as User 1 60 reorients and repositions its handheld component. Both the head-mounted 2 and handheld 4 components may be tracked in terms of position and orientation relative to each other in real or near-real time, for example, utilizing the tracking features of the system. The system may include a virtual frying pan 62, or other virtual or other real element, such as one of the user's two hands, or perhaps a real one, which may be configured to be trackable by the system. A frying pan or other user controlled element (such as a paddle) causes User 1 to turn the pancake 64 over so that it lands in its frying pan 62 and/or move the virtual pancake 64 away from User 1 . interact with the virtual elements of the pancake 64 using simulated physics, for example using the soft body physics capabilities of the environment, such as Unity (RTM), so that it can be thrown or tossed on a certain trajectory may be configured to allow it to act. The soft body physics ability may be configured to cause the pancake to bounce around any edge of the virtual frying pan 62 and fly, for example, in a trajectory and fashion that a real pancake would take. Referring to FIG. 5B, the virtual pancake 64 has animated properties that allow the user to score, create sounds, music, and heart or rainbow images, and the like. may be configured to provide the perception that the is happy to be flipped and/or landed. FIG. 5C illustrates User 1 60 successfully landing an inverted virtual pancake 64 in its virtual frying pan 62 . FIG. 5D illustrates User 1 (60) preparing to launch a virtual pancake 64 in front of User 1 (60). FIG. 5E illustrates a flung virtual pancake 64 flying away from User 1 (60).

Referring to FIG. 5F, in a multi-user experience such as the one described above with reference to FIG. 4, the system is configured to locate two players within the same environment 66 relative to each other. be. Here, User 1 (60) and User 2 (61) are depicted occupying two different ends of the same corridor in an office (see FIGS. 5K and 5L above, both users are shown together). ), the same virtual pancake 64 is similarly head-mounted, with a simulated physics trajectory, as it is flown by User 1 60 in FIG. 5E. Utilizing the ML1 system with Mold 2, Handheld 4, and Compute Pack 6 components, it has its own virtual frying pan element 63 and interacts with the virtual pancake 64 using simulated physics. , is flown towards user 2 (61). Referring to FIGS. 5G and 5H, user 2 (61) successfully aligns its virtual frying pan 63 and catches virtual pancake 64, and referring to FIGS. 5I and 5J, user 2 (61) , can throw a virtual pancake 64 back toward User 1 60, and User 1 throws its virtual pancake 62 back for another successful catch of virtual pancake 64, as shown in FIG. 5K. or alternatively, in FIG. 5L, User 1 misses the positioning and orientation of his virtual frying pan 62 and virtual pancake 64 appears to be facing the floor. Thus, multi-user or multi-player configurations are presented in which two users can collaborate or interact with various elements such as virtual dynamic elements.

With respect to PCFs and anchor points, as explained above, a local map, such as one created by a local user, may contain certain persistent anchor points or coordinate frames, which represent certain positions of various elements. and/or orientation. Maps, which may be promoted, stored, or created up to eight levels of external resources, such as maps, which may be promoted to cloud-based computing resources, may be merged with maps generated by other users. Indeed, a given user may be located within the cloud or part thereof, which may be larger or more refined than that generated in situ by the user, as explained above. Further, similar to local maps, cloud maps may be configured to contain certain persistent anchor points or PCFs, which may correspond to real-world positions and/or orientations, and which may correspond to maps or may be agreed upon by various devices within the same area or portion of the environment. Once the user is located (e.g., after first booting up and starting a scan with the ML1 system, or after losing coordination with the local map, or after SLAM activity locates the user) After walking a distance in the environment, the user can be located based on nearby map features that correspond to observable features in the real world. Persistent anchor points and PCFs may correspond to real-world locations, but they may also be fixed relative to each other until the map itself is updated. For example, if PCF-A and PCF-B are 5 meters apart, they may be configured to remain 5 meters apart even if the user is relocated (i.e., the system can The PCF may be configured to not move, ie only the user's presumed map match and the user's location within it move). Referring previously to FIGS. 6 and 7A-7C, the farther the user is from the high-confidence PCF (ie, closer to the user's localization point), the more error will occur in terms of position and orientation. For example, a 2 degree error in PCF alignment on any axis can be associated with an offset of 35 centimeters in 10 meters (tan(2 degrees) x 10), i.e. see above FIGS. 6 and 7A-7C. As discussed above, it is preferable to utilize persistent anchor points and PCFs that are close to the user or object being tracked.

As described with reference to FIG. 4, neighboring users may be configured to receive the same neighboring PCF and persistent anchor point information. From an overlapping set of PCFs and persistent anchor points between two players, the system may be configured to select one or more PCFs and/or persistent anchor points, which reduces the average error to Minimally, these shared anchors/PCFs may be utilized to set the position/orientation of shared virtual content. In one embodiment, a host system, such as a cloud computing resource, may be configured to provide meshing/mapping information and transmit this to all users when starting a new collaborative session or game. . The mesh may be positioned by the local user into a suitable installation location using its common anchors with the host, and the system allows the player to perceive very large "headroom" or ceiling heights. (useful for scenarios such as flipping pancakes between two users when maximum "air time" is desired) It may be configured to "cut" or "crop" off physically. Various aspects of mapping or mesh-based limitations, such as ceiling height, may optionally be avoided or ignored (e.g., in one embodiment, only the floor mesh/map may be used for certain virtual objects such as flying pancakes). may be used to confirm that the element hits the ground).

As mentioned above, soft body physics simulations may be utilized. In various embodiments, to prevent particles from getting stuck when they land between virtual frying pans 62, 63 or other colliding elements, such colliding elements are It may be configured to have extensions that grow on opposite sides of the pancake so that it can be properly resolved.

The user's system and associated connected resources 8 allow the user to access the game through an associated social network resource (such as a predetermined group of friends who also have an ML1 system) and by such user's geographic location. may be configured to allow the initiation of For example, when a particular user attempts to play a game such as illustrated in FIGS. It may be configured to automatically suppress the person.

Referring back to FIGS. 5A-5L, the system may be configured to deal with only one virtual pancake 64 at a time so that computational efficiency can be achieved (i.e., the entire game state is In various embodiments, the closest player/user to a given virtual pancake 64 at any given time is entitled to such virtual pancake 64. , which could mean that they control where it is for other users). In various embodiments, the networked state updates the physics and renders everything with an update frequency in the range of 60 Hz each, which enforces authority boundaries (i.e. panning from one user to another). It is chosen to keep the visual presentation consistent even when overriding the power of cake. In various embodiments, a Tnet configuration (packet-switched point-to-point system area network) is utilized, along with some custom packet configurations and other more standard packet configurations such as those described in known RFC publications. may

Referring to FIG. 6, two users (User 1 60, User 2 61) are placed in very close proximity to each other with one PCF 68 positioned immediately between them. Located within the same local environment 66 . With such a configuration and only one trusted PCF 68, games such as those illustrated in FIGS. can be done in the state However, referring to FIGS. 7A-7C, although the users (User 1 (60), User 2 (61)) are positioned relatively far apart, fortunately, multiple PCFs (69, 70, 71, 72) is positioned between the two users. As mentioned above, it is desirable to use a PCF that is as close to action as possible to prevent offset and/or drift. Thus, in FIG. 7A, with User 2 (61) being the flipping/flying side and generally having proximity to and control over the virtual pancake 64, neighboring PCFs (eg, 70, 69 , or both 69 and 70) may be utilized. Referring to FIG. 7B, as a virtual pancake 64 is flung between two users (61, 60), the closest PCF to the flying virtual pancake (eg, 70, 71, or 70 and 71) both) may be utilized. As shown in FIG. 7C, when the virtual pancake is thrown at a lower position, the other PCFs (eg, 69, 70, or both 69 and 70) closest to the flying virtual pancake are available may be Referring to FIG. 7D, as the virtual pancake (64) continues its trajectory toward user 1 (60), the PCF closest to the flying virtual pancake (eg, 69, 70, or 69 and 70) may be utilized. Therefore, a dynamic PCF selection configuration can help minimize drift and maximize accuracy.

Referring to FIG. 8, User 1 and User 2 are arranged so that they can engage in a common spatial computing experience such that certain content and/or virtual assets can be experienced by both users from each of their unique perspectives. may be located 80 within the same map. Local computing capabilities, etc., residing within the user's ML1 system, or connected systems, such as certain cloud computing resources 8 that may be interconnected, can be used to determine eye gaze, upper extremity kinematics, and body position and orientation relative to the local environment. It may also be configured to attempt to infer certain aspects of intent from user activity such as (82). For example, in one embodiment, the system utilizes the first user's captured gaze information to allow the first user to attempt to target a pancake location, such as a second user's approximate location, that the first user may attempt to target. 64 may be configured to infer a certain destination for throwing a virtual element. Similarly, the system may be configured to infer targeting or other variables from upper extremity position, orientation, velocity, and/or angular velocity. The information utilized to assist the user may be from real or near real-time sampling, or may be based on sampling from a larger time domain such as a particular user's play or participation history ( For example, a Convolutional Neural Network (“CNN”) configuration indicates that when a particular user attempts to hit a particular target or in a particular manner, he or she always glances in a particular way or always raises his arm. may be used to understand moving in a particular way, and such configuration may be used to assist the user). Based, at least in part, on one or more interferences related to the user's intent, the system may be configured to assist the user in accomplishing the intended purpose (84). For example, in one embodiment, the system establishes functional limits on the position or angular velocity of a given system component relative to the local environment when tremor is detected in a user's hand or arm (i.e., normal tremor , thereby smoothing out the user's commands to the system), or when it is determined that the conflict is desired In the example of small children with slight pulls toward each other over time (i.e., with relatively undeveloped gross motor skills), the system may help the child to be more successful in the game or use case. Additionally, it may be configured to assist the child in aiming or moving, for example, placing the virtual frying pan 62 too far to the right for a child who consistently fails to catch the virtual pancake 64. Thereby, the system may be configured to direct pancakes toward pans, or pans toward pancakes, or both.

In various other embodiments, the system may be configured to have other interactions with the local real world. For example, in one embodiment, a user playing a game such as that illustrated in FIGS. If it's straight enough at speed, the virtual pancake stays stuck to the wall for the rest of the game, or even permanently as part of the local mapping information or associated with neighboring PCFs. You may do so.

processing unit

FIG. 9 illustrates a processing unit 1002, according to some embodiments. Processing unit 1002 may, in some embodiments, be an example of apparatus 7 described herein. In other embodiments, the processing unit 1002 or any portion of the processing unit 1002 may be implemented using separate devices that communicate with each other. As shown, processing unit 1002 includes communication interface 1010 , positioner 1020 , graphics generator 1030 , non-transitory media 1040 , controller input 1050 and task assistant 1060 . In some embodiments, communication interface 1010, positioner 1020, graphics generator 1030, non-transitory media 1040, controller input 1050, task assistant 1060, or any combination of the foregoing are implemented using hardware. may As non-limiting examples, the hardware comprises one or more FPGA processors, one or more ASIC processors, one or more signal processors, one or more math processors, one or more It may include more integrated circuits, or any combination of the foregoing. In some embodiments, any component of processing unit 1102 may be implemented using software.

In some embodiments, processing unit 1002 may be implemented as separate components that are communicatively coupled together. For example, processing unit 1002 may include a first substrate carrying communication interface 1010, positioner 1020, graphics generator 1030, controller input 1050, task assistant 1060, and another substrate carrying non-transitory media 1040. may have As another example, all components of processing unit 1002 may be carried by the same substrate. In some embodiments, any, some, or all of the components of processing unit 1002 may be implemented in image display device 2 . In some embodiments, any, some, or all of the components of processing unit 1002 may be implemented in a device separate from image display device 2, such as handheld control component 4, control component 6, mobile phone, server, etc. good. In further embodiments, processing unit 1002 or any of the components of processing unit 1002 (such as positioner 1020) may be worn by different individual users, implemented in different display devices, or associated with different individual users. may be implemented in different devices that are located (eg, in close proximity to it).

The processing unit 1002 receives position information (eg, from a sensor in the image display device 2 or from an external device) and/or control information from the controller component 4, and based on the position information and/or control information, generates a virtual It is configured to present content within the screen of the image display device 2 for display. For example, as illustrated with reference to FIGS. 5A-5L, location information may indicate the location of user 60, and control information from controller 4 may indicate the location of controller 4 and/or through controller 4. Actions being performed by user 60 may be indicated. In such a case, processing unit 1002 generates an image of a virtual object (eg pancake 64 in the above example) based on the position of user 60 and control information from controller 4 . In one embodiment, the control information indicates the position of controller 4 . In such a case, processing unit 1002 generates an image of pancake 64 such that the position of pancake 64 is related to the position of controller 4 (as shown in FIG. 5D). For example, movement of pancake 64 will follow movement of controller 4 . In another example, when user 60 uses controller 4 to perform a maneuver such as throwing virtual pancakes 64 using controller 4 , the control information is associated with the direction of movement and movement of controller 4 . It would contain information about the speed and/or acceleration to be delivered. In such a case, processing unit 1002 then generates a graphic showing the movement of virtual pancake 64 (such as those shown in FIGS. 5E-5G). Movement is processed based on the position of where the pancake 64 popped out of the virtual frying pan 63 and based on the movement model (which receives as input the movement direction of the controller 4 and the velocity and/or acceleration of the controller 4). It may be along a movement trajectory calculated by unit 1002 . Mobility models will be described in further detail herein.

Returning to FIG. 9, communication interface 1010 is configured to receive location information. As used herein, the term "location information" refers to any information that represents the location of an entity or that can be used to derive the location of an entity. In some embodiments, communication interface 1010 is communicatively coupled to the camera and/or depth sensor of image display device 2 . In such embodiments, communication interface 1010 receives images directly from the camera and/or depth signals from the depth sensor. In some embodiments, the communication interface 1010 processes images from the camera and/or processes depth signals from the depth sensor before passing to the communication interface 1010 as location information, such as a separate processing unit. may be coupled to another device. In other embodiments, communication interface 1010 may be configured to receive GPS information, or any information that may be used to derive location. Also, in some embodiments, communication interface 1010 may be configured to obtain location information that is output wirelessly or via physical conductive transmission lines.

In some embodiments, if different sensors are present in the image display device 2 to provide different types of sensor output, the communication interface 1010 of the processing unit 1002 may receive different individual sensor outputs by: It may have different individual sub-communication interfaces. In some embodiments, sensor output may include images captured by a camera on image display device 2 . Alternatively or additionally, the sensor output may include range data captured by a depth sensor in image display device 2 . The range data may be data generated based on time-of-flight techniques. In such a case, the signal generator in the image display device 2 will transmit the signal and the signal will be reflected from objects in the user's surrounding environment. The reflected signal is received by a receiver in image display device 2 . Based on the time it takes for the signal to reach the object and reflect back to the receiver, the sensor or processing unit 1002 may then determine the distance between the object and the receiver. In other embodiments, the sensor output may include any other data that can be processed to determine the location of entities (users, objects, etc.) within the environment.

The positioner 1020 of the processing unit 1002 is configured to determine the position of the user of the image display device and/or the position of the virtual object to be displayed within the image display device. In some embodiments, the position information received by communication interface 1010 may be sensor signals, and positioner 1020 is configured to process the sensor signals to determine the position of the user of the image display device. be done. For example, the sensor signals may be camera images captured by one or more cameras of the image display device. In such cases, the positioner 1020 of the processing unit 1002 determines a localization map based on the camera image and/or uses features in the camera image and/or features in the generated localization map to localize the user. is configured to match features of In one implementation, positioner 1020 is configured to perform the actions described with reference to FIGS. 2 and/or 3 for user location. In other embodiments, the location information received by communication interface 1010 may already indicate the user's location. In such cases, positioner 1020 then uses the location information as the user's location.

As shown in FIG. 9, positioner 1020 includes anchor point module 1022 and anchor point selector 1024 . Anchor point module 1022 is configured to determine one or more anchor points, which are used by processing unit 1002 to locate the user and/or place the virtual object relative to the environment surrounding the user. It may be used by installing In some embodiments, an anchor point may be a point in a localization map, where each point in the localization map corresponds to an identified feature (e.g., corner, edge, object, etc.) in the physical environment. ). Also, in some embodiments, each anchor point may be a Persistent Coordinate Frame (PCF) determined previously or in the current session. In some embodiments, communication interface 1010 may receive a previously determined anchor point from another device. In such cases, anchor point module 1022 may obtain the anchor point by receiving the anchor point from communication interface 1010 . In other embodiments, anchor points may be stored in non-transitory media 1040 . In such cases, anchor point module 1022 may obtain the anchor point by reading the anchor point from non-transitory media 1040 . In a further embodiment, the anchor point module 1022 may be configured to determine anchor points within the mapping session. In a map-making session, a user wearing an image display device walks through the environment and/or views the image display device at different viewing angles such that the camera of the image display device captures images of different features in the environment. Oriented to Processing unit 1002 may then perform feature identification to identify one or more features in the environment for use as anchor points. In some embodiments, anchor points for a physical environment have already been determined in a previous session. In such a case, when the user enters the same physical environment, the camera in the image display device worn by the user will capture an image of the physical environment. The processing unit 1002 may identify features in the physical environment and ascertain whether one or more of the features match the previously determined anchor points. If applicable, the matched anchor points are made available by the anchor point module 1022 so that the processing unit 1002 can use them for user location and/or placement of virtual content. Will.

Also, in some embodiments, as the user moves about the physical environment, the anchor point module 1022 of the processing unit 1002 will identify additional anchor points. For example, when the user is at a first position in the environment, the anchor point module 1022 of the processing unit 1002 identifies anchor points AP1, AP2, AP3 that are in close proximity to the user's first position in the environment. You may When the user moves from a first position to a second position within the physical environment, the anchor point module 1022 of the processing unit 1002 moves closer to the user's second position within the environment, anchor points AP3, AP4, AP5 may be identified.

Additionally, in some embodiments, the anchor point module 1022 is configured to obtain anchor points associated with multiple users. For example, two users in the same physical environment may be standing apart from each other. A first user may be at a first location with a first set of anchor points associated therewith. Similarly, a second user may be at a second location with a second set of anchor points associated therewith. Initially, the first set of anchor points and the second set of anchor points may not have any overlap, as the two users are distant from each other. However, as one or both of the users move toward each other, the configuration of anchor points within the respective first and second sets will change. If they get close enough, the first and second set of anchor points will start to have overlap.

Anchor point selector 1024 is selected by processing unit 1002 (provided by anchor point module 1022) for use in locating the user and/or placing virtual objects relative to the environment surrounding the user. ) configured to select a subset of anchor points. In some embodiments, if the anchor point module 1022 provides multiple anchor points that are associated with a single user, and no other users are involved, the anchor point selector 1024 is used to locate the user and/or One or more of the anchor points may be selected for placement of virtual content relative to the physical environment. In other embodiments, the anchor point module 1022 is configured for different individual users (e.g., users wearing individual image display devices) who wish to interact with each other virtually within the same physical environment. A set of multiple anchor points may be provided that are associated with . In such cases, anchor point selector 1024 is configured to select one or more common anchor points that are common among the different sets of anchor points. For example, as shown in Figure 6, one common anchor point 68 may be selected to allow users 60, 61 to interact with the same virtual content (pancake 64). FIG. 7A shows another example in which four common anchor points 69, 70, 71, 72 are selected to allow users 60, 61 to interact with virtual content (pancake 64). An example is shown. Processing unit 1002 may then utilize the selected common anchor points for placement of virtual content so that users can interact with the virtual content within the same physical environment.

In some embodiments, anchor point selector 1024 may be configured to perform the actions described with reference to FIG.

Returning to FIG. 9, controller input 1050 of processing unit 1002 is configured to receive input from controller component 4 . The input from controller component 4 may be positional information regarding the position and/or orientation of controller component 4 and/or control information based on user actions performed via controller component 4 . By way of non-limiting example, control information from controller component 4 may be transmitted by the user translating controller component 4 , rotating controller component 4 , pressing one or more buttons on controller component 4 . actuating a knob, trackball, or joystick on the controller component 4, or any combination of the foregoing. In some embodiments, user input is utilized by processing unit 1002 to insert and/or move virtual objects presented on the screen of image display device 2 . For example, if the virtual object is a virtual pancake 64, described with reference to FIGS. , the virtual pancake 64 is caught, the frying pan 62 is used to move the virtual pancake 64, and/or the virtual pancake 64 is moved by the user so that it appears to move in the real environment. 64 may be manipulated to be thrown away from frying pan 62 . In some embodiments, the handheld controller component 4 is configured to move the virtual object within the two-dimensional display screen such that the virtual object would appear to be in motion within the virtual three-dimensional space. may For example, in addition to moving virtual objects up, down, left, and right, handheld controller component 4 may also move virtual objects in and out of the user's visual depth.

The graphics generator 1030 is configured to generate graphics for display on the screen of the image display device 2 based, at least in part, on the output from the positioner 1020 and/or the output from the controller input 1050. be done. For example, the graphics generator 1030 may control the screen of the image display device 2 to display the virtual object such that the virtual object appears within the environment when viewed by the user through the screen. As non-limiting examples, virtual objects include virtual movable objects (e.g., balls, shuttles, bullets, missiles, fire, heat waves, energy waves), weapons (e.g., swords, axes, hammers, knives, bullets, etc.), Any object that can be found inside a room (e.g. a pencil, a ball of crumpled paper, a cup, a chair, etc.) Any object that can be found outside a building (e.g. a rock, tree branch, etc.) It may be a vehicle (eg, car, airplane, space shuttle, rocket, submarine, helicopter, motorcycle, bicycle, tractor, all terrain vehicle, snowmobile, etc.), and the like. In some embodiments, the graphics generator 1030 also renders images of the virtual objects to the screen for display such that the virtual objects would appear to interact with real physical objects in the environment. You can generate above. For example, the graphics generator 1030 moves and configures the image of the virtual object on the screen such that when viewed by a user through the screen of the image display device 2, the virtual object appears to move through space within the environment. may be displayed in Also, in some embodiments, the graphics generator 1030 causes the virtual objects to appear to distort, damage, or otherwise alter physical objects in the environment when viewed by a user through the screen of the image display device 2 . An image of the virtual object may be displayed on the screen so that it appears to deform or damage the virtual object. In some cases, this allows the graphics generator 1030 to generate images of deformation marks (e.g., dent marks, crease lines, etc.), images of burn marks, images showing thermal changes, This may be accomplished by generating interactive images such as fire images, explosion images, wreckage images, and the like.

As mentioned above, in some embodiments, the graphics generator 1030 renders the virtual content such that the virtual objects appear to move within the three-dimensional space of the physical environment surrounding the user. It may be configured to be presented as a moveable virtual object. For example, the movable virtual object may be the flying pancake 64 described with reference to Figures 5A-5L. In some embodiments, the graphics generator 1030 uses the trajectory model and based on one or more anchor points provided by the anchor point module 1022 or the anchor point selector 1024 to fly. may be configured to generate a graphic of the pancake 64 that is in place. For example, processing unit 1002 may determine an initial trajectory for flying pancake 64 based on a trajectory model, where the initial trajectory is the desired action of flying pancake 64 to occur. indicate where The graphics generator 1030 then generates a sequence of images of the pancake 64, corresponding to the initial trajectory (eg, following the initial trajectory as much as possible) and displaying a video of the pancake 64 flying through the air. Form. The position of each image of pancake 64 to be presented in the video may be determined by processing unit 1002 based on the proximity of the action of pancake 64 to one or more neighboring anchor points. For example, as discussed with reference to FIG. 7A, when the action of pancake 64 approaches anchor points 69, 70, one or both of anchor points 69, 70 are generated by graphics generator 1030 to the user. Place the pancake 64 in the desired position relative to the display screen so that when 60, 61 view the pancake 64 in relation to the physical environment, the pancake 64 will be in the correct position relative to the physical environment. may be used to As pancake 64 progresses further along its trajectory, the action of pancake 64 approaches anchor points 70, 71, as shown in FIG. 7B. In such a case, one or both of the anchor points 70, 71 are generated by the graphics generator 1030 so that when the user 60, 61 views the pancake 64 in relation to the physical environment, the pancake 64 is physically It may be used to place the pancake 64 in the desired position relative to the display screen as it would be in the correct position relative to the environment. As pancake 64 progresses further along its trajectory, the action of pancake 64 approaches anchor points 69, 72, as shown in FIG. 7D. In such a case, one or both of the anchor points 69, 72 may be mapped by the graphics generator 1030 so that when the user 60, 61 views the pancake 64 in relation to the physical environment, the pancake 64 is physically It may be used to place the pancake 64 in the desired position relative to the display screen as it would be in the correct position relative to the environment.

As illustrated in the examples above, the actual positioning of the pancake 64 at various locations along its trajectory is based on different anchor points (e.g., features identified within the physical environment, PCF, etc.). Thus, as the pancake 64 moves across space, the pancake 64 is placed precisely against the anchor point in close proximity to the moving pancake 64 (where the action of the pancake 64 takes place). be done. This feature is advantageous because it prevents the pancake 64 from being placed incorrectly relative to the environment, which would otherwise allow the pancake 64 to be at only one anchor point close to the user. This can occur when installed against For example, if the positioning of pancake 64 is based solely on anchor point 70 , the distance between pancake 64 and anchor point 70 increases as pancake 64 moves further away from user 61 . If there is a slight error in the anchor point 70, such as incorrect positioning and/or orientation of the PCF, this will result in the pancake 64 offsetting or drifting away from its intended position. The magnitude of drifting increases as pancake 64 moves further from anchor point 70 . The above technique of selecting different anchor points in close proximity to the pancake 64 to locate the pancake 64 addresses offset and drifting issues. The above features also allow multiple users at distances (e.g., greater than 5 feet, greater than 10 feet, greater than 15 feet, greater than 20 feet, etc.) to accurately interact with each other, and This is advantageous because it allows for/or interacting with the same virtual content. In gaming applications, the above techniques may allow multiple users to accurately interact with the same object even when the users are relatively far apart. For example, in a gaming application, a virtual object may be virtually passed back and forth between remote users. As used herein, the term “close proximity” means less than some predefined value (e.g., 15 feet, 12 feet, 10 feet, 8 feet, 6 feet, 4 feet, 2 feet, 1 Refers to the distance between two items that satisfies a criterion such as distance (e.g., less than feet).

Note that the above technique of placing virtual content based on anchor points in close proximity to the actions of the virtual content is not limited to games involving two users. In other embodiments, the above techniques for placing virtual content may be applied to any application with only a single user or with more than two users (whether it is any gaming application or not). sometimes). For example, in other embodiments, the above techniques for placing virtual content may be utilized in applications that allow users to place virtual content remote from them within their physical environment. The above techniques for placing virtual content work even when the virtual content is far from the user (e.g., more than 5 feet, more than 10 feet, more than 15 feet, more than 20 feet, etc.) This is advantageous because it allows the virtual content to be virtually placed accurately with respect to the user (when viewed by the user through a screen worn by the user).

As discussed with reference to FIG. 9, processing unit 1002 includes non-transitory media 1040 configured to store anchor point information. As non-limiting examples, non-transitory media 1040 may include locations of anchor points, different sets of anchor points associated with different users, sets of common anchor points, location of users and/or placement of virtual content. Common anchor points, etc., selected for the purpose may be stored. Non-transitory media 1040 may store other information in other embodiments. In some embodiments, non-transitory media 1040 may store different virtual content, which may be read by graphics generator 1030 for presentation to the user. In some cases, certain virtual content may be associated with a gaming application. In such cases, when the gaming application is activated, processing unit 1002 may then access non-transitory media 1040 to obtain corresponding virtual content for the gaming application. In some embodiments, the non-transitory medium may also store gaming applications and/or parameters associated with gaming applications.

Additionally, as disclosed herein, in some embodiments the virtual content may be a movable object that moves within the screen based on the trajectory model. Trajectory models may be stored in non-transitory media 1040 in some embodiments. In some embodiments, the trajectory model may be straight. In such a case, when the trajectory model is applied to the movement of the virtual object, the virtual object will move within a straight line path defined by the straight lines of the trajectory model. As another example, the trajectory model may be a parabolic equation that defines a path based on the virtual object's initial velocity Vo and initial direction of movement, and based on the weight of the virtual object. Therefore, different virtual objects with different individual assigned weights will move along different parabolic paths.

Non-transitory media 1040 is not limited to a single storage unit and may be either integrated or separate but communicatively connected (eg, wirelessly or by wires). , may include a plurality of storage units.

In some embodiments, as the virtual object moves virtually through the physical environment, the processing unit 1002 tracks the position of the virtual object relative to one or more objects identified within the physical environment. do. In some cases, when a virtual object contacts or comes in close proximity to a physical object, the graphics generator 1030 generates graphics to illustrate interactions between the virtual object and physical objects in the environment. can be shown. For example, a graphic may indicate that a virtual object is deflected from a physical object (eg, wall) or another virtual object by changing the path of travel of the virtual object. As another example, when a virtual object contacts or comes in close proximity to a physical object (eg, wall) or another virtual object, the graphics generator 1030 determines whether the virtual object is a physical object or other virtual object. An interaction image may be placed in spatial association with where the object is touched. The interactive image may show that the wall is cracked, dented, scratched, soiled, and the like.

In some embodiments, different interaction images may be stored in non-transitory media 1040 and/or in a server in communication with processing unit 1002 . Interaction images may be stored in association with one or more attributes associated with the interaction of two objects. For example, an image of wrinkles may be stored in association with the attribute "blanket". In such cases, if the virtual object is displayed as being supported on a physical object, which may be identified as a "blanket," the graphics generator 1030 may , an image of wrinkles between the virtual object and the physical object may be displayed such that the virtual object sits on top of the blanket so that the blanket appears wrinkled.

The virtual content, which can be displayed virtually relative to the physical environment based on one or more anchor points, is not limited to the described examples; virtual content can be other items as well. Good thing to note. Also, as used herein, the term "virtual content" is not limited to virtualized physical items, but virtualized energy (e.g., laser beams, sound waves, energy waves, heat, etc.) can refer to the virtualization of any item such as The term "virtual content" can also refer to any content such as text, symbols, cartoons, animations, and the like.

task assistant

As shown in FIG. 9, processing unit 1002 also includes task assistant 1060 . Task assistant 1060 of processing unit 1002 receives the one or more sensor information based on the one or more sensor inputs to assist the user of the image display device in accomplishing objectives with virtual content. configured to For example, in some embodiments, one or more sensor inputs may indicate a user's eye gaze direction, upper extremity kinematics, body position, body orientation, or any combination of the foregoing. In some embodiments, processing unit 1002 is configured to assist a user in accomplishing objectives by applying one or more limits on the position and/or angular velocity of system components. Alternatively or additionally, the processing unit 1002 may be configured to assist the user in accomplishing an objective by gradually reducing the distance between the virtual content and another element. Using the above pancake tossing game, described with reference to FIGS. and based on the velocity and/or based on the trajectory of the pancake 64 it may detect an attempt to catch the pancake 64 . In such a case, task assistant 1060 moves pancake 64 away from the determined trajectory by moving pancake 64 away from the determined trajectory so that pancake 64 is closer toward frying pan 62 . The distance between the pancake 64 and the frying pan 62 may be gradually reduced, such as by deviating. Alternatively, task assistant 1060 may discretely increase the size of pancake 64 (i.e., calculated and not graphically) and/or increase the size of frying pan 62 (i.e., calculated , and not graphically), thereby allowing the user to catch the pancake 64 with the frying pan 62 more easily.

In some embodiments, assisting a user in accomplishing a task involving virtual content may be performed in response to the satisfaction of criteria. For example, using the pancake catch game described herein, in some embodiments, the processing unit 1002 determines whether the user pans based on the trajectory of the moving pancake 64 and the movement trajectory of the controller 4 . It may be configured to determine (eg, predict) whether it will be close to catching the cake 64 (eg, within a distance threshold of 5 inches, 3 inches, 1 inch, etc.). If applicable, task assistant 1060 will control graphics generator 1030 to output a graphic showing pancake 64 being caught by frying pan 62 . On the other hand, if processing unit 1002 determines (eg, predicts) that the user will not be close to catching pancakes 64 in skillet 62, task assistant 1060 prompts the user to perform the task. would not take any action for assistance.

Note that the tasks that the task assistant 1060 may assist the user to accomplish are not limited to the example of catching flying virtual objects. In other embodiments, task assistant 1060 indicates that processing unit 1002 will be very close to a state where the task will be accomplished (eg, greater than 80%, 85%, 90%, 95%, etc.). When making decisions (eg, predictions), it may assist the user in accomplishing other tasks. For example, in other embodiments, the task involves a user pointing a virtual object to another user, through an opening (e.g., a basketball hoop), to an object (e.g., a shooting range target), etc. may involve throwing or shooting at

In other embodiments, task assistant 1060 is optional and processing unit 1002 does not include task assistant 1060 .

A method performed by a processing unit and/or an application within the processing unit

FIG. 10 illustrates method 1100, according to some embodiments. Method 1100 may be performed by a device configured to provide virtual content within a virtual or augmented environment. Also, in some embodiments, the method 1100 allows a first user wearing a first display screen and a second user wearing a second display screen to interact with each other. It may be implemented by a device configured to provide virtual content within a virtual or augmented reality environment that can interact with. Each image display device may be image display device 2 in some embodiments. In some embodiments, method 1100 may be performed by any of the image display devices or by multiple image display devices described herein. Also, in some embodiments, at least a portion of method 1100 may be performed by processing unit 1002 or by multiple processing units (eg, processing units within separate image display devices). Further, in some embodiments, method 1100 may be performed by an apparatus that is separate from a server or image display device worn by an individual user.

As shown in FIG. 10, a method 1100 includes obtaining a first location of a first user (item 1102), and based on the first location of the first user, one or more determining a first set of anchor points (item 1104); obtaining a second location of the second user (item 1106); Determining a second set of one or more anchor points (item 1108) and determining one or more common anchor points within both the first set and the second set. (item 1110) and providing virtual content for experience by the first user and/or the second user based on at least one of the one or more common anchor points (item 1112 ) and

Optionally, in method 1100, the one or more common anchor points comprises a plurality of common anchor points, and the method further comprises selecting a subset of common anchor points from the plurality of common anchor points.

Optionally, in method 1100, the subset of common anchor points is selected to reduce localization error of the first user and the second user with respect to each other.

Optionally, in method 1100, the one or more common anchor points comprises a single common anchor point.

Optionally, method 1100 further includes determining a position and/or orientation for the virtual content based on at least one of the one or more common anchor points.

Optionally, in method 1100 each of the one or more anchor points in the first set is a point in a Persistent Coordinate Frame (PCF).

Optionally, in method 1100, the virtual content is provided as movable virtual objects within the first display screen and/or the second display screen for display.

Optionally, in method 1100, the virtual object appears for display within the first display screen as if the virtual object were moving in space between the first user and the second user. provided as follows.

Optionally, in method 1100, the one or more common anchor points comprises a first common anchor point and a second common anchor point, and the moveable virtual object is displayed on the first display for display. A movable virtual object is provided within the screen having a first object position relative to the first display screen and a second object position relative to the first display screen, the first object position of the movable virtual object is based on the first common anchor point and the second object position of the movable virtual object is based on the second common anchor point.

Optionally, method 1100 further includes selecting a first common anchor point for placing the virtual object at the first object position based on where the action of the virtual object occurs.

Optionally, in method 1100, the one or more common anchor points comprises a single common anchor point, the movable virtual object is placed in a first display screen for display, and the movable virtual object is placed in a second display screen for display. a first object position relative to one display screen and a second object position relative to the first display screen, the first object position of the movable virtual object being based on a single common anchor point; A second object position of the movable virtual object is based on the single common anchor point.

Optionally, in method 1100, the one or more common anchor points comprises a plurality of common anchor points, the method further comprising: , including selecting one of

Optionally, in method 1100, the act of selecting includes selecting one of the common anchor points closest to the action of the virtual content or within a distance threshold from the action of the virtual content.

Optionally, in method 1100 the position and/or movement of the virtual content can be controlled by the first user's first handheld device.

Optionally, in method 1100 the position and/or movement of the virtual content can also be controlled by the second user's second handheld device.

Optionally, method 1100 further includes localizing the first user and the second user to the same mapping information based on one or more common anchor points.

Optionally, method 1100 further includes displaying the virtual content with the first display screen such that the virtual content appears in a spatial relationship to physical objects in the first user's surroundings.

Optionally, method 1100 further includes obtaining one or more sensor inputs and assisting the first user in accomplishing a goal with the virtual content based on the one or more sensor inputs. including doing.

Optionally, in method 1100, the one or more sensor inputs are indicative of the first user's eye gaze direction, arm kinematics, body position, body orientation, or any combination of the foregoing.

Optionally, in method 1100, the act of assisting the first user to accomplish a goal includes applying one or more limits on position and/or angular velocity of system components.

Optionally, in method 1100, the act of helping the first user accomplish an objective includes gradually reducing the distance between the virtual content and another element.

Optionally, in method 1100, the apparatus comprises a first processing portion in communication with the first display screen and a second processing portion in communication with the second display screen.

In some embodiments, method 1100 may be performed in response to a processing unit executing instructions stored in a non-transitory medium. Thus, in some embodiments, a non-transitory medium includes stored instructions, execution of which by a processing unit causes a method to be performed. The processing unit is part of a device configured to provide virtual content within a virtual or augmented reality environment in which the first user and the second user can interact with each other. good too. A method (which is caused to be performed by a processing unit executing instructions) includes obtaining a first location of a first user; determining a first set of one or more anchor points; obtaining a second location of the second user; and based on the second location of the second user, one or more determining one or more common anchor points that are within both the first set and the second set; determining one or more common anchor points that are in both the first set and the second set; providing virtual content for experience by the first user and/or the second user based on at least one of the common anchor points.

Special treatment system

In some embodiments, the method 1100 described herein may be performed by system 1 (eg, processing unit 1002) executing an application or by the application. An application may contain a set of instructions. In one implementation, a specialized processing system may be provided having a non-transitory medium that stores a set of instructions for an application. Execution of the instructions by processing unit 1102 of system 1 will cause processing unit 1102 and/or image display device 2 to implement the features described herein. For example, execution of instructions by processing unit 1102 will cause method 1100 to be performed in some embodiments.

In some embodiments, system 1, image display device 2, or apparatus 7 may also be considered a specialized processing system. In particular, system 1, image display device 2, or apparatus 7 executes instructions stored in its non-transitory medium for execution by processing unit 1102 to provide unique tangible effects in the real world. It is a special processing system in that it contains The features provided by the image display device 2 (as a result of the execution of instructions by the processing unit 1102) provide improvements within the art of augmented reality and virtual reality.

FIG. 11 is a block diagram illustrating an embodiment of a specialized processing system 1600 that can be used to implement various features described herein. For example, in some embodiments, processing system 1600 may be used to implement at least a portion of system 1, such as image display device 2, processing unit 1002, and the like. Also, in some embodiments, processing system 1600 may be used to implement processing unit 1102 or one or more components therein (eg, positioner 1020, graphics generator 1030, etc.). .

Processing system 1600 includes a bus 1602 or other communication mechanism for communicating information, and a processor 1604 coupled with bus 1602 for processing information. Processor system 1600 also includes main memory 1606 , such as random access memory (RAM) or other dynamic storage device, coupled to bus 1602 for storing information and instructions to be executed by processor 1604 . Main memory 1606 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 1604 . Processor system 1600 also includes a read only memory (ROM) 1608 or other static storage device coupled to bus 1602 for storing static information and instructions for processor 1604 . A data storage device 1610, such as a magnetic disk, solid state disk, or optical disk, is provided and coupled to bus 1602 for storing information and instructions.

Processor system 1600 may be coupled via bus 1602 to a display 1612, such as a screen, for displaying information to a user. In some cases, display 1612 may be a touch screen when processing system 1600 is part of a device that includes a touch screen. An input device 1614 , including alphanumeric and other keys, is coupled to bus 1602 for communicating information and command selections to processor 1604 . Another type of user input device is a cursor control 1616 , such as a mouse, trackball, or cursor direction keys for communicating direction information and command selections to processor 1604 and for controlling cursor movement on display 1612 . is. The input device typically has two axes, a first axis (e.g. x) and a second axis (e.g. y) that allow the device to define a position in a plane. has two degrees of freedom in In some cases, if processing system 1600 is part of a device that includes a touch screen, input device 1614 and cursor control may be the touch screen.

In some embodiments, processor system 1600 can be used to perform various functions described herein. According to some embodiments, such use may be performed by the processor 1604 in response to executing one or more sequences of one or more instructions contained within main memory 1606 . Provided by system 1600 . Those skilled in the art will know how to prepare such instructions based on the functions and methods described herein. Such instructions may be read into main memory 1606 from another processor-readable medium, such as storage device 1610 . Execution of the sequences of instructions contained in main memory 1606 causes processor 1604 to perform the process steps described herein. One or more processors in a multi-processing arrangement may also be employed to execute the sequences of instructions contained within main memory 1606 . In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement various embodiments described herein. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

The term "processor-readable medium," as used herein, refers to any medium that participates in providing instructions to processor 1604 for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical, solid state, or magnetic disks, such as storage device 1610 . Non-volatile media may be considered an example of non-transitory media. Volatile media includes dynamic memory, such as main memory 1606 . A volatile medium may be considered an example of a non-transitory medium. Transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise bus 1602 . Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

Common forms of processor-readable media include, for example, a floppy disk, hard disk, magnetic tape, or any other magnetic medium, CD-ROM, any other optical medium, any other physical medium with a pattern of holes. , RAM, PROM and EPROM, FLASH-EPROM, solid-state disks, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a processor can read.

Various forms of processor readable media may be involved in carrying one or more sequences of one or more instructions to processor 1604 for execution. For example, the instructions may initially be carried on a magnetic or solid state disk of the remote computer. A remote computer can load the instructions into its dynamic memory and send the instructions over a network such as the Internet. Processing system 1600 can receive data about network lines. Bus 1602 carries the data to main memory 1606, from which processor 1604 retrieves and executes the instructions. The instructions received by main memory 1606 may optionally be stored on storage device 1610 either before or after execution by processor 1604 .

Processing system 1600 also includes a communication interface 1618 coupled to bus 1602 . Communications interface 1618 provides bi-directional data communications coupling to network link 1620 , which is connected to local network 1622 . For example, communication interface 1618 may be a local area network (LAN) card for providing data communication connections to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 1618 sends and receives electrical, electromagnetic or optical signals that carry data streams representing various types of information.

Network link 1620 typically provides data communication through one or more networks to other devices. For example, network link 1620 may provide connectivity through local network 1622 to a host computer 1624 or device 1626 . Data streams transported over network link 1620 may include electrical, electromagnetic or optical signals. The signals through the various networks and the signals on network link 1620 and through communication interface 1618, which carry the data to and from processing system 1600, are exemplary forms of carrier waves transporting the information. Processing system 1600 can send messages and receive data, including program code, through the network(s), network link 1620 and communication interface 1618 .

The term "image", as used herein, refers to an image that is displayed and/or is not in display form (e.g., an image or image data stored in media or being processed). Note that we get

Also, as used herein, the term “action” for virtual content is not limited to virtual content that moves, but can also refer to stationary virtual content that can be moved (e.g., a pointer can also refer to any virtual content that can be or is being “dragged” by a user using , or an action can be performed on or near it.

Various exemplary embodiments are described herein. Reference is made to these examples in a non-limiting sense. They are provided to illustrate the more broadly applicable aspects of the invention. Various changes may be made to the described embodiments and equivalents may be substituted without departing from the true spirit and scope of the invention. In addition, many modifications may be made to adapt a particular situation, material, composition of matter, process, process action or step to the objective, spirit or scope of the present invention. Moreover, as will be appreciated by those skilled in the art, each of the individual variations described and illustrated herein may be used in some other embodiments without departing from the scope or spirit of the invention. has discrete components and features that are easily separated from or combined with features of any of All such modifications are intended to be within the scope of the claims associated with this disclosure.

The embodiments include methods that can be performed using the devices of the present subject matter. The method may comprise an act of providing such suitable device. Such offerings may be performed by end users. In other words, the act of "providing" simply means that the end user acquires, accesses, approaches, locates, configures, activates the device required in the subject method. , power it on, or otherwise act to provide it. Methods recited herein may be performed in any order of the recited events that is logically possible and in the recited order of events.

Unless such exclusive terminology is used, the term "comprising" in a claim associated with the present invention is used regardless of whether a given number of elements are recited in the claim. , shall allow for the inclusion of any additional elements, or addition of features may be viewed as transforming the nature of such claimed elements. Unless specifically defined herein, all technical and scientific terms used herein are to be understood in the broadest possible manner while maintaining legitimacy of the claims. should be given some meaning.

Illustrative aspects of the disclosure are described above, along with details regarding material selection and manufacturing. As to other details of the present disclosure, these may be understood in connection with the above-referenced patents and publications and generally known or appreciated by those of ordinary skill in the art. The same may be true with respect to the method-based aspects of this disclosure in terms of additional acts as commonly or logically employed.

Additionally, while the disclosure has optionally been described with reference to several embodiments incorporating various features, the disclosure is described or illustrated as being considered with respect to each variation of the disclosure. is not limited to Various changes may be made to the disclosure as described and equivalents (whether recited herein or not included for purposes of some brevity) may be incorporated into the disclosure. may be substituted without departing from the true spirit and scope of Additionally, when a range of values is provided, all intervening values between the upper and lower limits of that range and any other stated or intervening value within a stated range are included within this disclosure. should be understood to be included.

Also, any optional feature of the described variations of the invention may be described and claimed independently or in combination with any one or more of the features described herein. It is considered possible. Reference to an item in the singular includes the possibility that there are pluralities of the same item present. More specifically, as used in this specification and the claims associated herewith, the singular forms "a," "an," "said," and "the" are not otherwise specified. Including plural references unless stated otherwise. It is further noted that any claim may be drafted to exclude any optional element. Accordingly, the text does not use exclusive terminology such as "merely," "only," and equivalents in connection with the recitation of claim elements, or as antecedents for the use of "negative" limitations. intended to play a role.

Additionally, as used herein, a phrase referring to “at least one of” a list of items refers to one item or any combination of items, including single elements. As an example, "at least one of A, B, or C" encompasses A, B, C, A and B, A and C, B and C, and A, B, and C is intended. Conjunctive sentences, such as the phrase "at least one of X, Y, and Z," generally refer to at least one of X, Y, or Z, unless specifically stated otherwise. understood differently in contexts such as those used to convey that a Thus, such conjunctions generally imply that certain embodiments require that at least one of X, at least one of Y, and at least one of Z each be present. is not intended to

The scope of this disclosure should not be limited to the examples and/or the specification provided, but rather should be limited only by the scope of the claim terms associated with this disclosure.

In the foregoing specification, the disclosure has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of this disclosure. For example, the foregoing process flows are described with reference to a specific order of process actions. However, the order of many of the process actions described may be changed without affecting the scope or operation of the present disclosure. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Claims (45)

An apparatus for providing virtual content within an environment, wherein a first user and a second user may interact within the environment, the apparatus comprising:
A communication interface, said communication interface configured to communicate with a first display screen worn by said first user and/or a second display screen worn by said second user. , a communication interface, and
A processing unit, the processing unit comprising:
obtaining a first location of the first user;
determining a first set of one or more anchor points based on a first location of the first user;
obtaining a second location of the second user;
determining a second set of one or more anchor points based on a second location of the second user;
determining one or more common anchor points within both the first set and the second set;
providing the virtual content for experience by the first user and/or the second user based on at least one of the one or more common anchor points. An apparatus comprising: a processing unit; 2. The method of claim 1, wherein the one or more common anchor points comprise a plurality of common anchor points, and wherein the processing unit is configured to select a subset of common anchor points from the plurality of common anchor points. Apparatus as described. 3. The apparatus of claim 2, wherein the processing unit is configured to select the subset of common anchor points to reduce localization errors of the first user and the second user with respect to each other. 2. The apparatus of claim 1, wherein the one or more common anchor points comprise a single common anchor point. 2. The apparatus of claim 1, wherein the processing unit is configured to position and/or orient the virtual content based on at least one of the one or more common anchor points. 2. The apparatus of claim 1, wherein each of said one or more anchor points in said first set is a point in a Persistent Coordinate Frame (PCF). 2. The processing unit of claim 1, wherein the processing unit is configured to provide the virtual content as a movable virtual object within the first display screen and/or the second display screen for display. equipment. The processing unit is configured to display the virtual object on the first display for displaying the virtual object such that the virtual object appears to move in a space between the first user and the second user. 8. The device of claim 7, configured to present within a screen. the one or more common anchor points comprises a first common anchor point and a second common anchor point;
a processing unit for displaying the moveable virtual object such that the moveable virtual object has a first object position relative to the first display screen and a second object position relative to the first display screen; configured to provide within said first display screen to
wherein the first object position of the movable virtual object is based on the first common anchor point;
the second object position of the movable virtual object is based on the second common anchor point;
8. Apparatus according to claim 7. 3. The processing unit is configured to select the first common anchor point for placing the virtual object at the first object position based on where an action of the virtual object occurs. 9. Apparatus according to 9. the one or more common anchor points comprising a single common anchor point;
a processing unit for displaying the moveable virtual object such that the moveable virtual object has a first object position relative to the first display screen and a second object position relative to the first display screen; configured to provide within said first display screen to
the first object position of the movable virtual object is based on the single common anchor point;
the second object position of the movable virtual object is based on the single common anchor point;
8. Apparatus according to claim 7. The one or more common anchor points comprises a plurality of common anchor points, and the processing unit selects one of the common anchor points to place the virtual content within the first display screen. 2. The apparatus of claim 1, configured to select one. The processing unit for placing the virtual content by selecting one of the common anchor points closest to the action of the virtual content or within a distance threshold from the action of the virtual content. 13. Apparatus according to claim 12, configured to select one of the common anchor points. 2. The apparatus of claim 1, wherein the position and/or movement of said virtual content is controllable by said first user's first handheld device. 15. The apparatus of claim 14, wherein the position and/or movement of said virtual content is also controllable by said second user's second handheld device. 2. The method of claim 1, wherein the processing unit is configured to locate the first user and the second user to the same mapping information based on the one or more common anchor points. Apparatus as described. The processing unit causes the first display screen to display the virtual content such that the virtual content would appear in a spatial relationship to physical objects in the first user's surroundings. 2. The apparatus of claim 1, configured to: the processing unit is configured to obtain one or more sensor inputs;
the processing unit is configured to assist the first user in accomplishing a goal with the virtual content based on the one or more sensor inputs;
A device according to claim 1 . 19. The apparatus of claim 18, wherein the one or more sensor inputs are indicative of the first user's eye gaze direction, arm kinematics, body position, body orientation, or any combination of the foregoing. 3. The processing unit is configured to assist the first user in accomplishing the objective by applying one or more limits on the positions and/or angular velocities of system components. 19. Apparatus according to 18. 19. The processing unit of claim 18, wherein the processing unit is configured to assist the first user in accomplishing the objective by gradually reducing the distance between the virtual content and another element. Apparatus as described. 2. The apparatus of claim 1, wherein said processing unit comprises a first processing portion in communication with said first display screen and a second processing portion in communication with said second display screen. A method performed by a device configured to provide virtual content within an environment, the environment comprising a first user wearing a first display screen and a second display screen within the environment. may interact with a second user wearing a
obtaining a first location of the first user;
determining a first set of one or more anchor points based on a first location of the first user;
obtaining a second location of the second user;
determining a second set of one or more anchor points based on a second location of the second user;
determining one or more common anchor points within both the first set and the second set;
providing said virtual content for experience by said first user and/or said second user based on at least one of said one or more common anchor points. . 24. The method of claim 23, wherein the one or more common anchor points comprise a plurality of common anchor points, the method further comprising selecting a subset of the common anchor points from the plurality of common anchor points. the method of. 25. The method of claim 24, wherein the subset of common anchor points is selected to reduce localization errors of the first user and the second user with respect to each other. 24. The method of claim 23, wherein the one or more common anchor points comprise a single common anchor point. 24. The method of claim 23, further comprising determining a position and/or orientation for said virtual content based on at least one of said one or more common anchor points. 24. The method of claim 23, wherein each of said one or more anchor points in said first set is a point in a Persistent Coordinate Frame (PCF). 24. The method of claim 23, wherein said virtual content is provided as movable virtual objects within said first display screen and/or said second display screen for display. The virtual object appears for display within the first display screen such that the virtual object appears to move in a space between the first user and the second user. 30. A method according to claim 29 provided. the one or more common anchor points comprises a first common anchor point and a second common anchor point;
The movable virtual object is configured for display in the first display screen so that the movable virtual object has a first object position relative to the first display screen and a second object position relative to the first display screen. provided to have an object position and
wherein the first object position of the movable virtual object is based on the first common anchor point;
the second object position of the movable virtual object is based on the second common anchor point;
30. The method of claim 29. 32. The method of claim 31, further comprising selecting the first common anchor point for placing the virtual object at the first object position based on where an action of the virtual object occurs. the one or more common anchor points comprising a single common anchor point;
The movable virtual object is configured for display in the first display screen so that the movable virtual object has a first object position relative to the first display screen and a second object position relative to the first display screen. provided to have an object position and
the first object position of the movable virtual object is based on the single common anchor point;
the second object position of the movable virtual object is based on the single common anchor point;
30. The method of claim 29. The one or more common anchor points comprises a plurality of common anchor points, the method further comprising one of the common anchor points for placing the virtual content within the first display screen. 24. The method of claim 23, comprising selecting . 35. The method of claim 34, wherein the act of selecting comprises selecting one of the common anchor points closest to the virtual content action or within a distance threshold from the virtual content action. . 24. The method of claim 23, wherein the position and/or movement of said virtual content is controllable by said first user's first handheld device. 37. The method of claim 36, wherein the position and/or movement of said virtual content is also controllable by said second user's second handheld device. 24. The method of claim 23, further comprising locating the first user and the second user to the same mapping information based on the one or more common anchor points. 24. Displaying the virtual content with the first display screen such that the virtual content appears in a spatial relationship to physical objects in the first user's surroundings. The method described in . obtaining one or more sensor inputs;
24. The method of claim 23, further comprising: assisting the first user in fulfilling an objective with the virtual content based on the one or more sensor inputs. 41. The method of claim 40, wherein the one or more sensor inputs are indicative of the first user's eye gaze direction, arm kinematics, body position, body orientation, or any combination of the foregoing. 41. The method of claim 40, wherein the act of assisting the first user in accomplishing the objective comprises applying one or more limits on position and/or angular velocity of system components. 41. The method of claim 40, wherein the act of assisting the first user in accomplishing the objective comprises gradually reducing the distance between the virtual content and another element. 24. The method of claim 23, wherein the device comprises a first processing portion in communication with the first display screen and a second processing portion in communication with the second display screen. A processor-readable non-transitory medium storing a set of instructions, wherein execution of the set of instructions by a processing unit causes a method to be performed, the processing unit providing virtual content within an environment and within said environment a first user and a second user may interact with each other, said method comprising:
obtaining a first location of the first user;
determining a first set of one or more anchor points based on a first location of the first user;
obtaining a second location of the second user;
determining a second set of one or more anchor points based on a second location of the second user;
determining one or more common anchor points within both the first set and the second set;
providing said virtual content for experience by said first user and/or said second user based on at least one of said one or more common anchor points. A readable non-transitory medium.

Images