WO2016118344A1 - Objets à réalité augmentée de taille fixe - Google Patents

Objets à réalité augmentée de taille fixe Download PDF

Info

Publication number
WO2016118344A1
WO2016118344A1 PCT/US2016/012778 US2016012778W WO2016118344A1 WO 2016118344 A1 WO2016118344 A1 WO 2016118344A1 US 2016012778 W US2016012778 W US 2016012778W WO 2016118344 A1 WO2016118344 A1 WO 2016118344A1
Authority
WO
WIPO (PCT)
Prior art keywords
augmented reality
eye
real world
reality object
eye display
Prior art date
Application number
PCT/US2016/012778
Other languages
English (en)
Inventor
Scott RAMSBY
Dan Osborn
Shawn Wright
Anatolie GAVRILIUC
Forest Woodcroft GOUIN
Megan SAUNDERS
Jesse Rapczak
Stephen Latta
Adam G. Poulos
Daniel MCCULLOCH
Wei Zhang
Original Assignee
Microsoft Technology Licensing, Llc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US14/717,771 external-priority patent/US9934614B2/en
Application filed by Microsoft Technology Licensing, Llc filed Critical Microsoft Technology Licensing, Llc
Priority to CN201680006372.2A priority Critical patent/CN107209565B/zh
Publication of WO2016118344A1 publication Critical patent/WO2016118344A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0132—Head-up displays characterised by optical features comprising binocular systems
    • G02B2027/0134—Head-up displays characterised by optical features comprising binocular systems of stereoscopic type
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2016—Rotation, translation, scaling
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays

Definitions

  • Stereoscopic displays can simultaneously present images to the left and right eyes of a viewer. By presenting different views of the same object at different positions in the right and left eye fields of view, a three-dimensional perception of the object can be achieved.
  • An example wearable, head-mounted display system includes a left near-eye, see-through display configured to display a left-eye augmented reality image with a left-eye display size at left-eye display coordinates, a right near-eye, see-through display configured to display a right-eye augmented reality image with a right-eye display size at right-eye display coordinates, the left-eye augmented reality image and right-eye augmented reality image collectively forming an augmented reality object perceivable at an apparent real world depth by a wearer of the head-mounted display system, and a controller.
  • the controller sets the left-eye display coordinates relative to the right-eye display coordinates as a function of the apparent real world depth of the augmented reality object.
  • the function maintains an aspect of the left-eye display size and the right-eye display size throughout a non-scaling range of apparent real world depths of the augmented reality object, and the function scales the left-eye display size and the right-eye display size with changing apparent real world depth of the augmented reality object outside the non-scaling range of apparent real world depths.
  • FIG. 1 shows an example environment including a user wearing a near-eye, see-through display device.
  • FIG. 2 schematically shows an example stereoscopic, near-eye, see-through display device.
  • FIG. 3 is a diagram schematically illustrating example apparent real world size and depth for an augmented reality object scaled according to a first scaling function.
  • FIG. 4 is a diagram schematically illustrating example apparent real world size and depth for an augmented reality object scaled according to a second scaling function.
  • FIG. 5 is a flow chart illustrating a method for displaying an augmented reality object.
  • FIGS. 6A-6E are diagrams illustrating example scaling functions.
  • FIG. 7 schematically shows a first example view of an augmented reality object.
  • FIG. 8 schematically shows a second example view of an augmented reality object.
  • FIG. 9 schematically shows a third example view of an augmented reality object.
  • FIG. 10 schematically shows a fourth example view of an augmented reality object.
  • FIG. 11 shows an example computing system.
  • FIG. 12 shows an example head-mounted display device.
  • Near-eye see-through display devices may be configured to display augmented reality images to provide the illusion that augmented reality objects, sometimes referred to as holograms, are present in the real world environment surrounding the near- eye display device.
  • the displayed augmented reality objects may scale in size as a perceived depth of the augmented reality object changes.
  • size preservation may lessen the realism of the object, as the object will not scale exactly as a real object would scale.
  • such size preservation may make it easier to see objects that would be too small or too large if scaled as a real object would scale and/or may provide an increased ability to read or otherwise interact with content displayed on the object.
  • augmented reality content such as user interface elements, holographic icons, or the like
  • different types of augmented reality content may be sized according to different scaling functions.
  • user interface control elements such as cursors
  • holograms displayed as part of an immersive game environment may be scaled linearly with changing depth. In this way, the user interface control element may be maintained at a size that is visible to a user of the display device, even if the user interface control element is displayed at a relatively distant apparent depth.
  • scaling functions may also increase a user' s ability to visualize content displayed on an augmented reality object.
  • a holographic newspaper floating above a table across the room from the user may itself be visible, but the headlines on the newspaper may only be visible if the scaling techniques as described above are employed.
  • a user may have difficulty noticing the 3D effect of a
  • FIG. 1 shows an example environment 100 in which a user 102 is wearing near-eye, see-through display device, herein embodied as a head-mounted display (HMD) 104.
  • the HMD provides user 102 a see-through view of environment 100.
  • the HMD also displays augmented reality images to the user.
  • the HMD is a stereoscopic display device, wherein two separate augmented reality images are each displayed on respective left-eye and right-eye displays of the HMD.
  • the two augmented reality images When viewed by a wearer of the HMD (e.g., user 102), the two augmented reality images collectively form an augmented reality object perceivable by the wearer as part of environment 100.
  • FIG. 1 depicts example augmented reality objects 106a and 106b. However, it is to be understood that the depicted augmented reality objects are not visible to others in environment 100, and the augmented reality objects can only be seen by user 102 via HMD 104.
  • HMD 104 can display augmented reality images such that perceived augmented reality objects are body-locked and/or world-locked.
  • a body -locked augmented reality object moves as the six-degree-of-freedom pose (i.e., 6DOF: x, y, z, yaw, pitch, roll) of HMD 104 changes.
  • 6DOF six-degree-of-freedom pose
  • a body-locked augmented reality object appears to occupy the same portion of the field of view of user 102 and appears to be at the same distance from user 102, even as the user moves, turns, etc.
  • a world-locked augmented reality object appears to remain in a fixed location relative to the surrounding environment. Even as a user moves and the user's perspective changes, a world-locked augmented reality object will appear to be in the same position/orientation relative to the surrounding environment.
  • an augmented reality chess piece may appear to be on the same square of a real world chess board regardless of the vantage point from which a user views the chess board.
  • an HMD may track the 6DOF pose of the HMD and a geometric mapping/modeling of surface aspects of the surrounding environment.
  • the apparent real world- size of an augmented reality object, or portions of an augmented reality object may be changed as a function of the apparent real world depth of the augmented reality object.
  • the size of an augmented reality object may be increased as the augmented reality object is displayed at farther perceived distances, and the size of the augmented reality object may be decreased as the augmented reality object is displayed at nearer perceived distances.
  • the scaling function may be tuned so that the augmented reality object, or portions of the augmented reality object, will occupy the same proportion of a user's field of view (FOV) regardless of the perceived distance at which the augmented reality object is displayed. That is, the apparent real world size of an augmented reality object, or a portion of an augmented reality object, may be increased or decreased to maintain the same angular size relative to the user.
  • FOV field of view
  • user 102 is creating an augmented reality drawing via gesture input.
  • user 102 is creating a first drawing, depicted as augmented reality object 106a, along a first wall 108 that is relatively close to user 102 and HMD 104.
  • One or more aspects of augmented reality object 106a may be set such that augmented reality object 106a is visible to user 102.
  • the line thickness of the augmented reality object 106a may be set based on the distance between the user and the first wall 108 where the augmented reality object 106a is placed, in order to ensure that the augmented reality object is visible and to reduce eye strain of the user.
  • the augmented reality object changes in apparent depth, for example if the augmented reality object is placed such that its apparent depth increases, one or more aspects of the augmented reality object may be maintained in order to maintain visibility of the object. As shown in FIG. 1, the drawing created by the user is moved to be at a greater apparent depth.
  • the moved drawing depicted as augmented reality object 106b, is placed on second wall 110, which is a farther distance from user 102 and HMD 104 than first wall 108. Accordingly, the apparent real world depth of the augmented reality object has increased, and thus the apparent real world size of the augmented reality object decreases, in order to provide the perception of three dimensions.
  • the line thickness of the drawing is maintained, in order to maintain visibility of the drawing. As described herein, the line thickness of the drawings being maintained refers to the user-perceived line thickness being maintained. In some examples, maintaining the user-perceived line thickness may include adjusting one or more aspects of the actual displayed line.
  • some types of augmented reality objects may be scaled such that one or more aspects (e.g., line thickness) are constant throughout a range of different apparent depths.
  • one or more aspects e.g., line thickness
  • the aspect of the object may be set to a predetermined level that is constant throughout the range.
  • FIG. 2 is a schematic view 200 showing aspects of a wearable stereoscopic display system 202 including a controller 203.
  • the illustrated display system resembles ordinary eyewear and is one non-limiting example of HMD 104 of FIG. 1.
  • the display system includes a right display 206 and a left display 204.
  • the right and left displays are wholly or partly transparent from the perspective of the wearer, to give the wearer a clear view of his or her surroundings. This feature enables computerized display imagery to be admixed with imagery from the surroundings, for an illusion of augmented reality.
  • display imagery is transmitted in real time to display system 202 from a remote computing system operatively coupled to display system 202 (not shown).
  • the display imagery may be transmitted in any suitable form— viz., type of transmission signal and data structure.
  • the signal encoding the display imagery may be carried over a wired or wireless communication link of any kind to controller 203 of the display system.
  • controller 203 is operatively coupled to the right and left optical systems.
  • the controller is concealed within the display-system frame, along with the right and left optical systems.
  • the controller may include suitable input/output (10) componentry to enable it to receive display imagery from the remote computing system.
  • the controller may also include position-sensing componentry— e.g., a global-positioning system (GPS) receiver, a gyroscopic sensor or accelerometer to assess head orientation and/or movement, etc.
  • GPS global-positioning system
  • controller 203 sends appropriate control signals to the right optical system which cause the right optical system to form a right display image in right display 206.
  • the controller sends appropriate control signals to the left optical system which cause the left optical system to form a left display image in left display 204.
  • the wearer of the display system views the right and left display images through the right and left eyes, respectively.
  • an 'augmented reality object' may be an object of any desired complexity and need not be limited to a singular object. Rather, an augmented reality object may comprise a complete virtual scene having both foreground and background portions. An augmented reality object may also correspond to a portion or locus of a larger augmented reality object.
  • left display 204 and right display 206 are each displaying a respective augmented reality image (i.e., an image of a tree).
  • Left display 204 is displaying left augmented reality image 208 and right display 206 is displaying right augmented reality image 210.
  • Each of left display 204 and right display 206 may comprise a suitable display, such as an LCD display, configured to form a display image based on the control signals from controller 203.
  • Each display includes a plurality of individual, addressable pixels arranged on a rectangular grid or other geometry.
  • Each of left display 204 and right display 206 may further comprise optics for delivering the displayed image to the eyes. Such optics may include waveguides, beam splitters, partially reflective mirrors, etc.
  • the left augmented reality image 208 and right augmented reality image 210 create augmented reality object 212 when viewed by a wearer of the display system 202. While left augmented reality image 208 and right augmented reality image 210 are depicted in FIG. 2 as being identical, it is to be understood that each of the left and right augmented reality images may be the same, or each may be different (e.g., each may comprise an image of the same object, but from slightly different perspectives).
  • the augmented reality object 212 has an apparent real world size and an apparent real world depth determined by the size and location of each of the left augmented reality image 208 and right augmented reality image 210 on its respective display.
  • the apparent location, including apparent real world depth (i.e., z coordinate), apparent real world lateral position (i.e., x coordinate), and apparent real world vertical coordinate (i.e., y coordinate) of the augmented reality object 212 may be dictated by the display coordinates for each of the left and right augmented reality images 208, 210.
  • the apparent size may be dictated by the display size and the apparent depth for that object.
  • the display coordinates of an augmented reality image include the x,y location of each pixel comprising the augmented reality image.
  • the display size of an augmented reality image is a length measurement in one or more dimensions as dictated by the number of pixels comprising the augmented reality image, e.g., the proportion of the display taken up by the augmented reality image.
  • augmented reality image refers to an actual image displayed on a display
  • augmented reality object refers to the augmented reality content perceived by the wearer of the display system, when the wearer views both the right and left displays.
  • an augmented reality object may comprise any suitable augmented reality content, including but not limited to graphical user interfaces, user interface control elements, virtual user markings, holograms, animations, video simulations, and the like.
  • the right display coordinates and/or left display coordinates may be set relative to each other.
  • the left and right display coordinates may be set to be closer to each other.
  • the tree image may move towards the nose on the left and right displays.
  • the left and right display coordinates may be set to be farther from each other.
  • the tree image may move away from the nose on the left and right displays.
  • the right display size and/or left display size may be adjusted.
  • the right and/or left display sizes may be increased to increase the apparent real world size of an augmented reality object.
  • the apparent real world size of an augmented reality object may be the size of the augmented reality object relative to other, real objects at the same apparent depth.
  • the apparent real world size of the augmented reality object may scale as a function of the apparent real world depth.
  • each scaling function may set left and right display coordinates relative to each other to set an augmented reality object at a desired apparent real world depth as well as scale one or more aspects of augmented reality image display size based on the apparent real world depth.
  • Each function may perform the scaling differentially, such as linearly, non-linearly, scale only within a specified range of depths, or other suitable function.
  • augmented reality image display size may be scaled linearly with changing apparent real world depth outside a non-scaling range of apparent real world depths, while within the non-scaling range of apparent real world depths, augmented reality image display size may be maintained. In doing so, the apparent real world size of the augmented reality object may change with changing apparent real world depth such that the augmented reality object stays at a constant proportion of the field of view of the wearer of the display system.
  • FIG. 3 is a diagram 300 schematically illustrating example apparent real world size and depth for an augmented reality object scaled according to a first scaling function.
  • An augmented reality image 302 is displayed on a near-eye, see-through display 304, such as a display included in HMD 104 of FIG. 1 and/or display system 202 of FIG. 2.
  • the augmented reality image 302 appears to be an augmented reality object 308. While only one augmented reality image is depicted in FIG. 3, it is to be understood that display 304 may include two displays, each displaying respective augmented reality images.
  • FIG. 3 also includes a timeline 310.
  • the augmented reality image 302 is displayed with a first display size, DS1, and with display coordinates that set the augmented reality object at a first apparent depth, AD1. Due to the display size and apparent depth, the augmented reality object has a first apparent size, AS1.
  • the apparent depth of the augmented reality object is increased, as shown by apparent depth AD2.
  • the first scaling function applied in the example of FIG. 3 specifies that the display size of the augmented reality image 302 be maintained while apparent depth changes, and thus the display size DS2 is equal to the display size DS1 of time Tl .
  • the apparent size of the augmented reality object 308 increases, as shown by apparent size AS2.
  • the relative proportion of the field of view of the user taken up by the augmented reality image, and the augmented reality object remains constant from time Tl to time T2.
  • FIG. 4 is a diagram 400 schematically illustrating example apparent real world size and depth for an augmented reality object scaled according to a second scaling function.
  • an augmented reality image 402 is displayed on a near-eye, see- through display 404, such as a display included in HMD 104 of FIG. 1 and/or display system 202 of FIG. 2.
  • display 404 When viewed by eyes of a user 406, the augmented reality image 402 appears to be an augmented reality object 408. While only one augmented reality image is depicted in FIG. 4, it is to be understood that display 404 may include two displays, each displaying respective augmented reality images.
  • FIG. 4 also includes a timeline 410.
  • the augmented reality image 402 is displayed with a third display size, DS3, and with display coordinates that set the augmented reality object at a third apparent depth, AD3. Due to the display size and apparent depth, the augmented reality object has a third apparent size, AS3.
  • the third display size DS3 is equal to the first display size DS1 of FIG. 3.
  • the third apparent depth AD3 and third apparent size AS3 are each equal to the first apparent depth AD1 and first apparent size AS1, respectively, of FIG. 3.
  • the apparent depth of the augmented reality object is increased, as shown by apparent depth AD4.
  • the second scaling function applied in the example of FIG. 4 specifies that the display size of the augmented reality image 302 be scaled linearly with apparent depth.
  • the display size DS4 decreases relative to the display size DS3 at time Tl .
  • the apparent size of the augmented reality object 408 at time T2 stays the same, as shown by AS4.
  • the apparent size of the augmented reality object at time Tl, AS3 is equal to the apparent size AS4 at time T2.
  • the relative proportion of the field of view of the user taken up by the augmented reality image and augmented reality object decreases at time T2 relative to time Tl .
  • method 500 for displaying an augmented reality object is illustrated.
  • Method 500 may be enacted in a wearable, head-mounted stereoscopic display system, such as HMD 104 of FIG. 1 or display system 202 of FIG. 2 described hereinabove, or HMD 1200 of FIG. 12, described below.
  • method 500 includes obtaining an augmented reality object for display on the display system.
  • the augmented reality object may include any suitable augmented reality content and may be displayed as part of a graphical user interface, game, guidance or assistance system, or any suitable augmented or immersive environment.
  • the augmented reality object may be obtained from a remote service, from a memory of the display system, or other suitable source in response to user input, predetermined sequence of an executed game or other content, or other suitable action.
  • the augmented reality object may be comprised of a right-eye and left-eye augmented reality image, each configured to be displayed on respective right-eye and left-eye displays of the display system. Accordingly, obtaining the augmented reality object may include obtaining corresponding left-eye and right-eye augmented reality images.
  • the method includes determining an augmented reality object type and associated scaling function.
  • the augmented reality object may be classified into one or more types of objects.
  • Example types of augmented reality objects include graphical user interfaces, user interface control elements (e.g., cursors, arrows), virtual user markings (e.g., drawings), navigation and/or assistance icons, holograms, and other suitable types of augmented reality objects.
  • Each type of augmented reality object may have an associated scaling function that dictates how the display sizes of the augmented reality images forming the augmented reality object scale as a function of the apparent real word depth of the augmented reality object.
  • the apparent real world depth of the augmented reality object is determined.
  • the augmented reality object may be displayed at a suitable apparent real world depth.
  • the apparent real world depth of the augmented reality object may be set according to one or more suitable parameters, including but not limited to user command (e.g., if a user issues a gesture, voice, or other command indicating the augmented reality object be placed at a given location), association with one or more real world objects, and preset parameters of the augmented reality object (e.g., the augmented reality object may have a preset depth selected to reduce eye strain of the user).
  • method 500 includes displaying the augmented reality object at the apparent real world depth and at an apparent real world size according to the scaling function.
  • method 500 includes displaying a left- eye augmented reality image on a left near-eye, see-through display with a left-eye display size at left-eye display coordinates according to the scaling function, as indicated at 510.
  • method 500 includes displaying a right-eye augmented reality image on a right near-eye, see-through display with a right-eye display size at right-eye display coordinates according to the scaling function, as indicated at 512.
  • the apparent real world depth of the augmented reality object may be dictated by the respective right-eye and left-eye display coordinates.
  • the appropriate apparent real world size of the augmented reality object may be set as a function of the apparent real world depth, according to the scaling function.
  • the augmented reality object may have a default apparent real world size for a given apparent real world depth. The default size may be based on the type of augmented reality object, context and/or environment in which the augmented reality object is placed, user input, and/or other suitable factors.
  • the scaling function may then alter this apparent real world size based on the determined real world depth.
  • the right-eye and left-eye display sizes of the right-eye and left-eye augmented reality images may be adjusted, as explained above.
  • Example scaling functions that may be applied during execution of method 500 are illustrated in FIGS. 6A-6E.
  • Each of diagrams 601, 603, 605, 607, and 609 plots augmented reality image display size as a function of apparent real world depth of the corresponding augmented reality object.
  • the example functions may apply to one or more dimensions of an augmented reality image (e.g., height, or width, or height and width).
  • the example functions may apply to another aspect of an augmented reality image, such as a line thickness.
  • a first linear function illustrated by line 602 adjusts the display size linearly
  • the first linear scaling function may be used to scale augmented reality objects that are intended to mimic elements within a user's environment, e.g., objects within a game environment. While a linear function such as the one illustrated by line 602 may accurately represent how an object changes in perceived size as the depth of the object changes, it may result in an object becoming too small to be accurately perceived or so large that it occludes the user's field of view.
  • FIG. 604. Another example of a linear scaling function is illustrated by line 604.
  • the display size of the augmented reality image remains constant regardless of apparent real world depth. While such an approach to sizing the augmented reality object may be simple to execute, it also suffers from the same issues as the first linear scaling function, e.g., the augmented reality object being too small or too large at some depths. Realism is also lessened, because augmented reality objects scaled in this manner do not mimic the scaling of real world objects.
  • first segmented function is illustrated as line 606.
  • the display size is maintained constant over a first non-scaling range of apparent depths, and adjusts linearly with changing depth at depths outside the first non-scaling range.
  • left-eye and right-eye display sizes are scaled as a function of apparent real world depth (e.g., decreasing size with increasing depth) until the apparent real world depth reaches a first threshold depth Tl .
  • the display sizes remain constant throughout the non-scaling range of depths until a second threshold depth T2 is reached.
  • the left-eye and right-eye display sizes are again scaled as a function of apparent real world depth.
  • the first segmented scaling function may be applied to scale augmented reality obj ects that do not necessarily correlate to real obj ects or the real world environment. This may include user interface control elements, such as cursors, graphical interfaces, and virtual user markings such as drawings.
  • the apparent real world size of the augmented reality object may be smaller at lesser depths and greater at greater depths, thus occupying the same, constant proportion of the user's field of view throughout the first non-scaling range of depths. In doing so, the augmented reality object may be easily visualized and/or interacted with by the user, even at relatively far depths. Further, by scaling the display size as a function of depth outside the first non-scaling range, the first segmented scaling function prevents the augmented reality object from becoming too large and occluding the user's field of view.
  • a second segmented scaling function is illustrated by line 608.
  • the second segmented scaling function is similar to the first segmented scaling function, and includes a second non-scaling range of depths between a first threshold depth Tl and a second threshold depth T2 where the display sizes of the augmented reality images are maintained at a constant size.
  • the second non-scaling range of depths may be different than the first non-scaling range, e.g., the second non-scaling range may be a larger range of depths than the first non-scaling range.
  • a third segmented scaling function is illustrated by line 610.
  • the third segmented scaling function linearly scales the display sizes of the augmented reality images as a function of depth within a scaling range of depths, but maintains the display sizes at one or more constant sizes outside the scaling range of depths. For example, the display sizes are maintained at a first, relatively large display size at close range depths, scale linearly in the scaling range of depths, and then are maintained at a second, relatively small display size at far range depths.
  • the example scaling functions described above may each be associated with a respective different type of augmented reality object and automatically applied each time the associated augmented reality object is displayed.
  • a respective scaling function may be applied to an augmented reality function in response to a user request or other input.
  • each displayed augmented reality object may be scaled according to its respective scaling function.
  • some augmented reality objects when displayed together, may be scaled similarly, while other augmented reality objects may be scaled differently.
  • a displayed object that is part of a game e.g., a holographic tree, such as the one illustrated in FIG. 2
  • a control object such as a cursor used to control aspects of the game, may be scaled according to the first segmented scaling function to maintain visibility of the cursor.
  • the left-eye display coordinates may be set relative to the right-eye display coordinates as a function of the apparent real world depths for both first and second augmented reality objects.
  • An aspect (e.g., overall image size) of the left-eye display size and the right-eye display size may be maintained throughout a non- scaling range of apparent real world depths for only the first augmented reality object.
  • the left-eye display size and the right-eye display size may be scaled with changing apparent real world depth for both the first and second augmented reality objects outside the non- scaling range of apparent real world depths.
  • the left-eye display size and the right-eye display size may be scaled with changing apparent real world depth throughout the non- scaling range of apparent real world depths for only the second augmented reality object.
  • scaling functions described above with respect to FIGS. 6A-6E are exemplary in nature, and other scaling functions may be used. Scaling functions having any number of constant, linear, or non-linear segments may be used. Different scaling segments of the same function may have different scaling properties. For example, the slope of a scaling segment before a constant segment may be greater than the slope of a scaling segment after the constant segment. [0060] Other variations from the functions illustrated in FIGS. 6A-6E are contemplated. For example, the slope of the first linear function may be smaller or greater than illustrated. In another example, the first segmented scaling function may scale in size during the non-scaling range of depths, but at a much lower rate than outside the non-scaling range of depths.
  • the function may only scale a proportion of the necessary scale required to maintain the same angular size, blending both concerns of giving cues that the user is moving relative to the augmented reality object while at the same time mostly maintaining its angular size to allow the user to more easily view and interact with it.
  • the scaling functions may be user configurable in some examples.
  • Some scaling functions may have restrictions on the maximum and minimum apparent real world sizes, which would result in the angular size of an augmented reality object appearing to change if the user moves beyond the corresponding physical distances to the object.
  • the scaling operations may be triggered by virtually any object positioning change, and are not restricted to only positioning due to collisions with other real world or augmented reality obj ects .
  • a floating user interface element may continuously update its apparent real world size to maintain its angular size (e.g., proportion of the user's field of view) based upon placement against a real world surface that the user is gazing at, while a line the user draws may size itself to maintain a target angular size based upon the distance to the target physical surface it is drawn upon, but then not change in world space size after that point.
  • angular size e.g., proportion of the user's field of view
  • some scaling functions may adjust aspects of displayed augmented reality images alternative or in addition to image display size. For example, hue, color, transparency, lighting effects, and/or feature density of an augmented reality image may be adjusted based on apparent real world depth.
  • the example scaling functions were described above with respect to how the overall apparent real world size of the augmented reality object changes based on apparent real world depth.
  • one or more specific aspects of the augmented reality object may be adjusted alternative to or in addition to the adjustment of the overall apparent real world size.
  • One example aspect that may be adjusted is the line thickness of the augmented reality object, which is described in more detail below.
  • Another example aspect that may be adjusted includes object orientation.
  • an augmented reality object such as a book
  • the augmented reality object may be easily visible when viewed head-on.
  • the side angle e.g. 90 degrees
  • the book is effectively impossible to read.
  • the augmented reality object may be automatically rotated to face the user.
  • This effect can be referred to as billboarding.
  • a billboarding effect may be keyed to apparent real world depth.
  • billboarding may be implemented only within a range of apparent real world depths.
  • FIG. 7 shows an example view 700 through a near-eye, see-through display
  • the user can see real world walls 702a, 702b, 702c, 702d, and floor 704.
  • the user can see an augmented reality object of a first instance of a virtual user marking, herein depicted as horizontal line 706' on wall 702b and an augmented reality object of a second instance of the same horizontal line 706" on wall 702d.
  • horizontal line 706" is five feet away from the user and occupies a 0.95 degree vertical angular spread.
  • the horizontal line 706" may appear to be one inch tall in world space coordinates.
  • the same horizontal line 706' may still occupy a 0.95 degree vertical angular spread but appear to be two inches tall in world space coordinates.
  • the line occupies the same proportion of an HMD's field of view at different distances, and the line will have the same weight regardless of the apparent real world depth at which it is drawn. Maintaining the weight at different distances may make it easier for a user to perceive augmented reality lines at farther depths.
  • the horizontal line length may be scaled as a function of depth. As shown, the perceived size of the horizontal line 706' is shorter than the perceived size of the horizontal line 706". However, in other examples the line length may be kept constant, similar to the line weight.
  • a user interface control element herein depicted as a cursor
  • a segmented scaling function such as the first segmented scaling function described above.
  • FIG. 8 shows a view 800 with a first instance of an augmented reality cursor 802' at a relatively far distance and a second instance of the same augmented reality cursor 802" at a relatively near distance.
  • the augmented reality cursor is maintained at the same proportion of the user's field of view.
  • the segmented scaling function maintains the same display size for the left-eye and right-eye displays of the augmented reality images comprising the augmented reality cursor, at least throughout a non-scaling range of depths.
  • the overall size of augmented reality objects comprising many constituent elements may be scaled so as to have a larger corresponding apparent real world size when at a relatively far distance and a smaller corresponding apparent real world size when at a relatively close distance.
  • FIG. 9 shows view 900 with an augmented reality object of a first instance of a picture 902' at a relatively far distance and a second instance of the same picture 902" at a relatively near distance.
  • the augmented reality objects are scaled so as to occupy the same proportion of an HMD's field of view at the different distances. As a result, picture 902' has a greater real world size than picture 902".
  • an augmented reality object may be a parent object comprising multiple child objects (e.g., sub-objects).
  • the object illustrated in FIG. 9 includes a square frame with two circles contained inside the frame.
  • scaling functions may be applied differentially to different children of the parent augmented reality object. In this way, aspects of particular child objects may be scaled and/or maintained based on depth, while aspects of other child objects may not be scaled or maintained based on depth.
  • the circles and the frame may be scaled based on depth while the thickness of the lines that is used to render these objects is maintained at the same display size, as illustrated in FIG. 10, described below.
  • the overall size of an augmented reality object may remain the same relative to the surrounding environment, but one or more of the constituent elements may scale.
  • the overall size of an icon may appear to be smaller when displayed at a farther perceived distance, but the thickness of the lines making up the icon may appear the same at both near and far perceived distances.
  • FIG. 10 shows view 1000 with an augmented reality object of a first instance of a picture 1002' at a relatively far distance and a second instance of the same picture 1002" at a relatively near distance.
  • the augmented reality objects are scaled such that the overall real world dimensions of the objects remain consistent at different distances.
  • the farther instance of picture 1002' occupies less of an HMD's field of view than the nearer instance of picture 1002".
  • the constituent lines that make up the pictures are scaled so as to occupy the same proportion of the HMD's field of view at the different distances.
  • the methods and processes described herein may be tied to a computing system of one or more computing devices.
  • such methods and processes may be implemented as a computer-application program or service, an application-programming interface (API), a library, and/or other computer-program product.
  • API application-programming interface
  • FIG. 11 schematically shows a non-limiting embodiment of a computing system 1100 that can enact one or more of the methods and processes described above.
  • HMD 104 of FIG. 1, display system 202 of FIG. 2, and/or HMD 1200 of FIG. 12, described below, are non-limiting examples of computing system 1100.
  • Computing system 1100 is shown in simplified form.
  • Computing system 1100 may take the form of one or more personal computers, server computers, tablet computers, home-entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phone), and/or other computing devices.
  • Computing system 1100 includes a logic machine 1102 and a storage machine 1104.
  • Computing system 1100 may optionally include a display subsystem 1106, input subsystem 1108, communication subsystem 1110, and/or other components not shown in FIG. 11.
  • Logic machine 1102 includes one or more physical devices configured to execute instructions.
  • the logic machine may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs.
  • Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.
  • the logic machine may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machine may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. Processors of the logic machine may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing. Individual components of the logic machine optionally may be distributed among two or more separate devices, which may be remotely located and/or configured for coordinated processing. Aspects of the logic machine may be virtualized and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration.
  • Storage machine 1104 includes one or more physical devices configured to hold instructions executable by the logic machine to implement the methods and processes described herein. When such methods and processes are implemented, the state of storage machine 1104 may be transformed— e.g., to hold different data. [0077] Storage machine 1104 may include removable and/or built-in devices.
  • Storage machine 1104 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.), among others.
  • Storage machine 1 104 may include volatile, nonvolatile, dynamic, static, read/write, readonly, random-access, sequential-access, location-addressable, file-addressable, and/or content-addressable devices.
  • storage machine 1104 includes one or more physical devices.
  • aspects of the instructions described herein alternatively may be propagated by a communication medium (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for a finite duration.
  • a communication medium e.g., an electromagnetic signal, an optical signal, etc.
  • logic machine 1102 and storage machine 1104 may be integrated together into one or more hardware-logic components.
  • Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC / ASICs), program- and application-specific standard products (PSSP / ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.
  • FPGAs field-programmable gate arrays
  • PASIC / ASICs program- and application-specific integrated circuits
  • PSSP / ASSPs program- and application-specific standard products
  • SOC system-on-a-chip
  • CPLDs complex programmable logic devices
  • module may be used to describe an aspect of computing system 1 100 implemented to perform a particular function.
  • a module, program, or engine may be instantiated via logic machine 1102 executing instructions held by storage machine 1104. It will be understood that different modules, programs, and/or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and/or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc.
  • module may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.
  • a “service”, as used herein, is an application program executable across multiple user sessions.
  • a service may be available to one or more system components, programs, and/or other services.
  • a service may run on one or more server-computing devices.
  • display subsystem 1106 may be used to present a visual representation of data held by storage machine 1104.
  • This visual representation may take the form of a graphical user interface (GUI).
  • GUI graphical user interface
  • Display subsystem 1106 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with logic machine 1102 and/or storage machine 1104 in a shared enclosure, or such display devices may be peripheral display devices.
  • input subsystem 1108 may comprise or interface with one or more user-input devices such as a keyboard, mouse, touch screen, or game controller.
  • the input subsystem may comprise or interface with selected natural user input (NUI) componentry.
  • NUI natural user input
  • Such componentry may be integrated or peripheral, and the transduction and/or processing of input actions may be handled on- or off-board.
  • NUI componentry may include a microphone for speech and/or voice recognition; an infrared, color, stereoscopic, and/or depth camera for machine vision and/or gesture recognition; a head tracker, eye tracker, accelerometer, and/or gyroscope for motion detection and/or intent recognition; as well as electric-field sensing componentry for assessing brain activity.
  • communication subsystem 1 110 may be configured to communicatively couple computing system 1100 with one or more other computing devices.
  • Communication subsystem 1110 may include wired and/or wireless communication devices compatible with one or more different communication protocols.
  • the communication subsystem may be configured for communication via a wireless telephone network, or a wired or wireless local- or wide-area network.
  • the communication subsystem may allow computing system 1100 to send and/or receive messages to and/or from other devices via a network such as the Internet.
  • FIG. 12 shows a non-limiting example of a head-mounted, near-eye, see- through display system, also referred to as an HMD 1200, in the form of wearable glasses with a see-through display 1202.
  • HMD 1200 is a non-limiting example of the HMD 104 of FIG. 1, the display system 202 of FIG. 2, and/or the computing system 1100 of FIG. 11.
  • An HMD may take any other suitable form in which a transparent, semi-transparent, and/or non-transparent display is supported in front of a viewer's eye or eyes.
  • embodiments described herein may be used with any other suitable computing device, including but not limited to mobile computing devices, laptop computers, desktop computers, tablet computers, other wearable computers, etc.
  • an augmented reality image may be displayed on a mobile phone's display along with real world imagery captured by the mobile phone's camera.
  • the HMD 1200 includes a see-through display 1202 and a controller 1204.
  • the see-through display 1202 may enable images such as augmented reality images (also referred to as holographic objects) to be delivered to the eyes of a wearer of the HMD.
  • the see-through display 1202 may be configured to visually augment an appearance of a real- world, physical environment to a wearer viewing the physical environment through the transparent display.
  • the display may be configured to display one or more UI objects of a graphical user interface.
  • the UI objects presented on the graphical user interface may be virtual objects overlaid in front of the real -world environment.
  • the UI objects presented on the graphical user interface may incorporate elements of real -world objects of the real -world environment seen through the see-through display 1202.
  • the display may be configured to display one or more other graphical objects, such as virtual objects associated with games, videos, or other visual content.
  • the see-through display 1202 may include image-producing elements located within lenses 1206 (such as, for example, a see-through Organic Light- Emitting Diode (OLED) display).
  • the see-through display 1202 may include a display device (such as, for example a liquid crystal on silicon (LCOS) device or OLED microdisplay) located within a frame of HMD 1200.
  • the lenses 1206 may serve as, or otherwise include, a light guide for delivering light from the display device to the eyes of a wearer.
  • Such a light guide may enable a wearer to perceive a 3D holographic image located within the physical environment that the wearer is viewing, while also allowing the wearer to directly view physical objects in the physical environment, thus creating a mixed reality environment.
  • the see-through display 1202 may present left-eye and right-eye augmented reality images via respective left-eye and right-eye displays, as discussed above with respect to FIG. 2.
  • the HMD 1200 may also include various sensors and related systems to provide information to the controller 1204.
  • sensors may include, but are not limited to, one or more inward facing image sensors 1208a and 1208b, one or more outward facing image sensors 1210, an inertial measurement unit (IMU) 1212, and one or more microphones 1220.
  • the one or more inward facing image sensors 1208a, 208b may be configured to acquire image data in the form of gaze tracking data from a wearer' s eyes (e.g., sensor 208a may acquire image data for one of the wearer's eye and sensor 1208b may acquire image data for the other of the wearer's eye).
  • the HMD may be configured to determine gaze directions of each of a wearer's eyes in any suitable manner based on the information received from the image sensors 1208a, 1208b.
  • one or more light sources 1214a, 1214b such as infrared light sources, may be configured to cause a glint of light to reflect from the cornea of each eye of a wearer.
  • the one or more image sensors 1208a, 1208b may then be configured to capture an image of the wearer's eyes. Images of the glints and of the pupils as determined from image data gathered from the image sensors 1208a, 1208b may be used by the controller 1204 to determine an optical axis of each eye.
  • the controller 1204 may be configured to determine a direction the wearer is gazing.
  • the controller 1204 may be configured to additionally determine an identity of a physical and/or virtual object at which the wearer is gazing by projecting the user's gaze vector onto a 3D model of the surrounding environment.
  • the one or more outward facing image sensors 1210 may be configured to measure physical environment attributes of the physical environment in which the HMD 1200 is located (e.g., light intensity). Data from the outward facing image sensors 1210 may be used to detect movements within a field of view of the display 1202, such as gesture- based inputs or other movements performed by a wearer or by a person or physical object within the field of view. In one example, data from the outward facing image sensors 1210 may be used to detect a selection input performed by the wearer of the HMD, such as a gesture (e.g., a pinching of fingers, closing of a fist, etc.), that indicates selection of a UI object displayed on the display device.
  • a gesture e.g., a pinching of fingers, closing of a fist, etc.
  • Data from the outward facing sensors may also be used to determine direction/location and orientation data (e.g. from imaging environmental features) that enables position/motion tracking of the HMD 1200 in the real -world environment.
  • Data from the outward facing camera also may be used to construct still images and/or video images of the surrounding environment from the perspective of the HMD 1200.
  • the IMU 1212 may be configured to provide position and/or orientation data of the HMD 1200 to the controller 1204.
  • the IMU 1212 may be configured as a three-axis or three-degree of freedom (3DOF) position sensor system.
  • This example position sensor system may, for example, include three gyroscopes to indicate or measure a change in orientation of the HMD 1200 within 3D space about three orthogonal axes (e.g., roll, pitch, and yaw).
  • the orientation derived from the sensor signals of the IMU may be used to display, via the see-through display, one or more AR images with a realistic and stable position and orientation.
  • the IMU 1212 may be configured as a six-axis or six- degree of freedom (6DOF) position sensor system.
  • a six-axis or six- degree of freedom (6DOF) position sensor system may include three accelerometers and three gyroscopes to indicate or measure a change in location of the HMD 1200 along three orthogonal spatial axes (e.g., x, y, and z) and a change in device orientation about three orthogonal rotation axes (e.g., yaw, pitch, and roll).
  • position and orientation data from the outward facing image sensors 1210 and the IMU 1212 may be used in conjunction to determine a position and orientation of the HMD 1200.
  • the HMD 1200 may also support other suitable positioning techniques, such as GPS or other global navigation systems. Further, while specific examples of position sensor systems have been described, it will be appreciated that any other suitable position sensor systems may be used. For example, head pose and/or movement data may be determined based on sensor information from any combination of sensors mounted on the wearer and/or external to the wearer including, but not limited to, any number of gyroscopes, accelerometers, inertial measurement units, GPS devices, barometers, magnetometers, cameras (e.g., visible light cameras, infrared light cameras, time-of-flight depth cameras, structured light depth cameras, etc.), communication devices (e.g., WTFI antennas/interfaces), etc.
  • gyroscopes e.g., accelerometers, inertial measurement units, GPS devices, barometers, magnetometers, cameras (e.g., visible light cameras, infrared light cameras, time-of-flight depth cameras, structured light depth cameras, etc.), communication devices (
  • the controller 1204 may be configured to record multiple eye gaze samples over time based on information detected by the one or more inward facing image sensors 1208a, 1208b.
  • eye tracking information and, in some embodiments, head tracking information may be used to estimate an origin point and a direction vector of that eye gaze sample to produce an estimated location at which the eye gaze intersects the see-through display.
  • Examples of eye tracking information and head tracking information used to determine an eye gaze sample may include an eye gaze direction, head orientation, eye gaze velocity, eye gaze acceleration, change in angle of eye gaze direction, and/or any other suitable tracking information.
  • eye gaze tracking may be recorded independently for both eyes of the wearer of the HMD 1200.
  • Controller 1204 may be configured to generate or update a three-dimensional model of a surrounding environment using information from outward facing image sensors 1210. Additionally or alternatively, information from outward facing image sensors 1210 may be communicated to a remote computer responsible for generating and/or updating a model of the surrounding environment. It either case, the relative position and/or orientation of the HMD relative to the surrounding environment may be assessed so that augmented reality images may be accurately displayed in desired real world locations with desired orientations.
  • the HMD 1200 may also include one or more microphones, such as microphone 1220, that capture audio data.
  • the one or more microphones 1220 may comprise a microphone array including two or more microphones.
  • the microphone array may include four microphones, two positioned above the right lens and two positioned above the left lens of the HMD.
  • audio outputs may be presented to the wearer via one or more speakers, such as speaker 1222.
  • the controller 1204 may include a logic machine and a storage machine, discussed in more detail above with respect to FIG. 11, in communication with the display and the various sensors of the HMD.
  • An example wearable, head-mounted display system comprises a left near- eye, see-through display configured to display a left-eye augmented reality image with a left-eye display size at left-eye display coordinates, a right near-eye, see-through display configured to display a right-eye augmented reality image with a right-eye display size at right-eye display coordinates, the left-eye augmented reality image and right-eye augmented reality image collectively forming an augmented reality object perceivable at an apparent real world depth by a wearer of the head-mounted display system, and a controller.
  • the controller sets the left-eye display coordinates relative to the right-eye display coordinates as a function of the apparent real world depth of the augmented reality object, the function maintaining an aspect of the left-eye display size and the right-eye display size throughout a non-scaling range of apparent real world depths of the augmented reality object, and the function scaling the left-eye display size and the right-eye display size with changing apparent real world depth of the augmented reality object outside the range of apparent real world depths.
  • the augmented reality object comprises virtual user markings.
  • maintaining the aspect of the left-eye display size and the right-eye display size comprises maintaining a line thickness of the virtual user markings throughout the non-scaling range.
  • Such an example additionally or alternatively includes scaling a line length of the virtual user markings as a function of apparent real world depth throughout the non-scaling range.
  • Such an example additionally or alternatively includes wherein the function decreases a distance between the left-eye display coordinates and the right-eye display coordinates with decreasing apparent real world depth.
  • Such an example additionally or alternatively includes wherein maintaining the aspect of the left-eye display size and the right-eye display size throughout the non-scaling range of apparent real world depths comprises changing an apparent real world size of a corresponding aspect of the augmented reality object throughout the non-scaling range of apparent real world depths, such that the augmented reality object occupies a constant proportion of the wearer's field of view.
  • the function decreases the left-eye display size and the right- eye display size at apparent real world depths greater than the non-scaling range, and increases the left-eye display size and the right-eye display size at apparent real world depths less than the non-scaling range.
  • the augmented reality object is a first augmented reality object, and wherein the controller sets left-eye coordinates of a second augmented reality obj ect relative to right-eye coordinates of the second augmented reality object as a second function of apparent real world depth of the second augmented reality object.
  • Such an example additionally or alternatively includes wherein the second function maintains an aspect of a left-eye display size and a right-eye display size of the second augmented reality object throughout a second, different non-scaling range of apparent real world depths of the second augmented reality object.
  • the augmented reality object is a child object of a parent augmented reality object, and wherein the function scales a left-eye display size and a right-eye display size of the parent augmented reality object with changing apparent real world depth of the parent augmented reality object throughout the non-scaling range of apparent real world depths of the parent augmented reality object.
  • Another example provides a method for a wearable, head-mounted display system comprising displaying, on a left near-eye, see-through display, a left-eye augmented reality image with a left-eye display size at left-eye display coordinates according to a scaling function, displaying, on a right near-eye, see-through display, a right-eye augmented reality image with a right-eye display size at right-eye display coordinates according to the scaling function, the left-eye augmented reality image and right-eye augmented reality image collectively forming an augmented reality object perceivable at an apparent real world depth by a wearer of the head-mounted display system, the scaling function setting the left-eye display coordinates relative to the right-eye display coordinates as a function of the apparent real world depth of the augmented reality object, the scaling function maintaining an aspect of the left-eye display size and the right-eye display size throughout a non-scaling range of apparent real world depths of the augmented reality object, and the scaling function scaling the left-eye display size
  • Such an example additionally or alternatively includes wherein scaling the left-eye display size and the right-eye display size with changing apparent real world depth of the augmented reality object outside the non-scaling range comprises increasing the left-eye display size and the right-eye display size with decreasing apparent real world depth and decreasing the left-eye display size and right-eye display size with increasing apparent real world depth outside the non-scaling range of real world depths.
  • Such an example additionally or alternatively includes wherein maintaining an aspect of the left-eye display size and the right-eye display size throughout the non-scaling range comprises maintaining the augmented reality object as a constant proportion of a field of view of the wearer throughout the non-scaling range.
  • Such an example additionally or alternatively includes wherein maintaining the augmented reality object as a constant proportion of the field of view of the wearer comprises changing a real world size of the augmented reality object relative to real world objects at a same depth of the augmented reality object as the apparent real world depth of the augmented reality object changes.
  • the augmented reality object comprises a virtual user marking, and wherein maintaining an aspect of the left-eye display size and the right-eye display size throughout a non-scaling range of apparent real world depths comprises maintaining a line thickness of the virtual user marking. Any or all of the above- described examples may be combined in any suitable manner in various implementations.
  • a wearable, head-mounted display system comprising a left near-eye, see-through display configured to display a first left-eye augmented reality image and a second left-eye augmented reality image, the first and second left-eye augmented reality images displayed with different left-eye display sizes at different left-eye display coordinates, a right near-eye, see-through display configured to display a first right-eye augmented reality image and a second right-eye augmented reality image, the first and second right-eye augmented reality images displayed with different right-eye display sizes at different right-eye display coordinates, the first left-eye and first right-eye augmented reality images collectively forming a first augmented reality object, the second left-eye and second right-eye augmented reality images collectively forming a second augmented reality object, the first and second augmented reality objects perceivable at respective apparent real world depths by a wearer of the head-mounted display system, and a controller to set the left-eye display coordinates relative to the right-eye display coordinates
  • Such an example additionally or alternatively includes wherein the first augmented reality object comprises a user interface control element, and wherein the second augmented reality object comprises a holographic game element. Such an example additionally or alternatively includes wherein the first augmented reality object is a child of the second augmented reality object.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Hardware Design (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Architecture (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Optics & Photonics (AREA)
  • Controls And Circuits For Display Device (AREA)
  • User Interface Of Digital Computer (AREA)
  • Processing Or Creating Images (AREA)

Abstract

La présente invention concerne un exemple de système d'affichage pouvant être porté qui comprend un contrôleur, un dispositif d'affichage pour afficher une image à réalité augmentée pour l'œil gauche avec une taille d'affichage pour l'œil gauche au niveau de coordonnées d'affichage pour l'œil gauche, et un dispositif d'affichage droit pour afficher une image à réalité augmentée pour l'œil droit avec une taille d'affichage pour l'œil droit au niveau de coordonnées d'affichage pour l'œil droit, les images à réalité augmentée pour l'œil gauche et pour l'œil droit forment collectivement un objet à réalité augmentée perceptible au niveau d'une profondeur du monde réel apparent par un porteur du système d'affichage. Le contrôleur règle les coordonnées d'affichage d'œil gauche associées aux coordonnées d'affichage d'œil droit comme une fonction de la profondeur du monde réel apparent de l'objet à réalité augmentée. La fonction maintient un aspect des tailles d'affichage de l'œil gauche et de l'œil droit sur toute une plage de non-mise à l'échelle des profondeurs du monde réel apparent de l'objet à réalité augmentée, et la fonction met à l'échelle les tailles d'affichage de l'œil gauche et de l'œil droit avec un changement de profondeur du monde réel apparent à l'extérieur de la plage de non-mise à l'échelle.
PCT/US2016/012778 2015-01-20 2016-01-11 Objets à réalité augmentée de taille fixe WO2016118344A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201680006372.2A CN107209565B (zh) 2015-01-20 2016-01-11 用于显示固定大小的增强现实对象的方法和系统

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US201562105672P 2015-01-20 2015-01-20
US62/105,672 2015-01-20
US14/717,771 US9934614B2 (en) 2012-05-31 2015-05-20 Fixed size augmented reality objects
US14/717,771 2015-05-20

Publications (1)

Publication Number Publication Date
WO2016118344A1 true WO2016118344A1 (fr) 2016-07-28

Family

ID=55349938

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2016/012778 WO2016118344A1 (fr) 2015-01-20 2016-01-11 Objets à réalité augmentée de taille fixe

Country Status (2)

Country Link
CN (1) CN107209565B (fr)
WO (1) WO2016118344A1 (fr)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3501985A1 (fr) * 2017-12-22 2019-06-26 The Boeing Company Systèmes et procédés pour affichages de réalité virtuelle en vol pour passager et assistance d'équipage
WO2019152255A1 (fr) * 2018-02-01 2019-08-08 Microsoft Technology Licensing, Llc Affichage d'un contenu visuel stéréo modifié
US10514801B2 (en) 2017-06-15 2019-12-24 Microsoft Technology Licensing, Llc Hover-based user-interactions with virtual objects within immersive environments
WO2023044050A1 (fr) * 2021-09-17 2023-03-23 Apple Inc. Assistant numérique pour fournir une visualisation de fragments d'informations
US11837237B2 (en) 2017-05-12 2023-12-05 Apple Inc. User-specific acoustic models
US11979836B2 (en) 2007-04-03 2024-05-07 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
US12009007B2 (en) 2013-02-07 2024-06-11 Apple Inc. Voice trigger for a digital assistant
US12026197B2 (en) 2017-05-16 2024-07-02 Apple Inc. Intelligent automated assistant for media exploration
US12061752B2 (en) 2018-06-01 2024-08-13 Apple Inc. Attention aware virtual assistant dismissal
US12067990B2 (en) 2014-05-30 2024-08-20 Apple Inc. Intelligent assistant for home automation
US12118999B2 (en) 2014-05-30 2024-10-15 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US12136419B2 (en) 2023-08-31 2024-11-05 Apple Inc. Multimodality in digital assistant systems

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107592520B (zh) 2017-09-29 2020-07-10 京东方科技集团股份有限公司 Ar设备的成像装置及成像方法
US10817047B2 (en) * 2018-09-19 2020-10-27 XRSpace CO., LTD. Tracking system and tacking method using the same
CN111083391A (zh) * 2018-10-19 2020-04-28 舜宇光学（浙江）研究院有限公司 虚实融合系统及其方法
CN111506188A (zh) * 2019-01-30 2020-08-07 托比股份公司 用于动态地调整hud的方法和hmd

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6198484B1 (en) * 1996-06-27 2001-03-06 Kabushiki Kaisha Toshiba Stereoscopic display system
US20130050432A1 (en) * 2011-08-30 2013-02-28 Kathryn Stone Perez Enhancing an object of interest in a see-through, mixed reality display device
US20130050258A1 (en) * 2011-08-25 2013-02-28 James Chia-Ming Liu Portals: Registered Objects As Virtualized, Personalized Displays
US20130326364A1 (en) * 2012-05-31 2013-12-05 Stephen G. Latta Position relative hologram interactions
US20130328927A1 (en) * 2011-11-03 2013-12-12 Brian J. Mount Augmented reality playspaces with adaptive game rules

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8711206B2 (en) * 2011-01-31 2014-04-29 Microsoft Corporation Mobile camera localization using depth maps
KR101918057B1 (ko) * 2011-11-30 2019-01-30 삼성전자주식회사 이미지의 깊이 정보를 복원하는 방법 및 장치
US10502876B2 (en) * 2012-05-22 2019-12-10 Microsoft Technology Licensing, Llc Waveguide optics focus elements
CN102981616B (zh) * 2012-11-06 2017-09-22 中兴通讯股份有限公司 增强现实中对象的识别方法及系统和计算机

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6198484B1 (en) * 1996-06-27 2001-03-06 Kabushiki Kaisha Toshiba Stereoscopic display system
US20130050258A1 (en) * 2011-08-25 2013-02-28 James Chia-Ming Liu Portals: Registered Objects As Virtualized, Personalized Displays
US20130050432A1 (en) * 2011-08-30 2013-02-28 Kathryn Stone Perez Enhancing an object of interest in a see-through, mixed reality display device
US20130328927A1 (en) * 2011-11-03 2013-12-12 Brian J. Mount Augmented reality playspaces with adaptive game rules
US20130326364A1 (en) * 2012-05-31 2013-12-05 Stephen G. Latta Position relative hologram interactions

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11979836B2 (en) 2007-04-03 2024-05-07 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
US12009007B2 (en) 2013-02-07 2024-06-11 Apple Inc. Voice trigger for a digital assistant
US12118999B2 (en) 2014-05-30 2024-10-15 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US12067990B2 (en) 2014-05-30 2024-08-20 Apple Inc. Intelligent assistant for home automation
US11837237B2 (en) 2017-05-12 2023-12-05 Apple Inc. User-specific acoustic models
US12026197B2 (en) 2017-05-16 2024-07-02 Apple Inc. Intelligent automated assistant for media exploration
US10514801B2 (en) 2017-06-15 2019-12-24 Microsoft Technology Licensing, Llc Hover-based user-interactions with virtual objects within immersive environments
US10991138B2 (en) 2017-12-22 2021-04-27 The Boeing Company Systems and methods for in-flight virtual reality displays for passenger and crew assistance
EP3501985A1 (fr) * 2017-12-22 2019-06-26 The Boeing Company Systèmes et procédés pour affichages de réalité virtuelle en vol pour passager et assistance d'équipage
US10523912B2 (en) 2018-02-01 2019-12-31 Microsoft Technology Licensing, Llc Displaying modified stereo visual content
WO2019152255A1 (fr) * 2018-02-01 2019-08-08 Microsoft Technology Licensing, Llc Affichage d'un contenu visuel stéréo modifié
US12061752B2 (en) 2018-06-01 2024-08-13 Apple Inc. Attention aware virtual assistant dismissal
WO2023044050A1 (fr) * 2021-09-17 2023-03-23 Apple Inc. Assistant numérique pour fournir une visualisation de fragments d'informations
US12136172B2 (en) 2022-09-15 2024-11-05 Apple Inc. Digital assistant for providing visualization of snippet information
US12136419B2 (en) 2023-08-31 2024-11-05 Apple Inc. Multimodality in digital assistant systems

Also Published As

Publication number Publication date
CN107209565B (zh) 2020-05-05
CN107209565A (zh) 2017-09-26

Similar Documents

Publication Publication Date Title
US9934614B2 (en) Fixed size augmented reality objects
US10672103B2 (en) Virtual object movement
CN107209565B (zh) 用于显示固定大小的增强现实对象的方法和系统
US10740971B2 (en) Augmented reality field of view object follower
US9824499B2 (en) Mixed-reality image capture
EP3619599B1 (fr) Contenu virtuel affiché avec un ancrage partagé
US10754496B2 (en) Virtual reality input
US10373392B2 (en) Transitioning views of a virtual model
EP3137982B1 (fr) Transitions de la réalité augmentée entre un mode body locked et un mode world locked
US10127725B2 (en) Augmented-reality imaging
US10134174B2 (en) Texture mapping with render-baked animation
CN106489171B (zh) 立体图像显示器
US20180046352A1 (en) Virtual cursor movement
US10523912B2 (en) Displaying modified stereo visual content
US20180143693A1 (en) Virtual object manipulation
US20180330546A1 (en) Wind rendering for virtual reality computing device
US10025099B2 (en) Adjusted location hologram display
US11150470B2 (en) Inertial measurement unit signal based image reprojection

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16704076

Country of ref document: EP

Kind code of ref document: A1

REEP Request for entry into the european phase

Ref document number: 2016704076

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE