WO2019046334A1

WO2019046334A1 - System and method of compensating for real world illumination changes in augmented reality

Info

Publication number: WO2019046334A1
Application number: PCT/US2018/048395
Authority: WO
Inventors: David WYBLE
Original assignee: Pcms Holdings, Inc.
Priority date: 2017-08-29
Filing date: 2018-08-28
Publication date: 2019-03-07

Abstract

A method in some embodiments is performed using an augmented reality head-mounted display (HMD) that includes a primary display and a dimmable filter that is external to the primary display. An illuminance sensor measures the illuminance of a real-world scene. Based on the illuminance, a target brightness of a virtual object and a target dimming level of the dimmable filter are selected. The virtual object is displayed on the primary display with the target brightness, and the dimmable filter is adjusted to the target dimming level. Changes to virtual object brightness and dimming level of the filter may be performed gradually to avoid sudden brightness changes that may be distracting to a user. In some embodiments, the luminance sensor is used to measure a white point of the real-world scene, and the virtual object is color balanced based on the measured white point.

Description

SYSTEM AND METHOD OF COMPENSATING FOR REAL WORLD ILLUMINATION CHANGES IN

AUGMENTED REALITY

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application is a non-provisional filing of, and claims benefit under 35 U.S.C. §119(e) from, U.S. Provisional Patent Application Serial No. 62/551 ,512, filed August 29, 2017, entitled "System and Method of Compensating for Real World Illumination Changes in Augmented Reality," which is incorporated herein by reference in its entirety.

BACKGROUND

[0002] The synthetic content inserted into augmented reality systems is expected to maintain a natural appearance. This poses a challenge because ambient conditions in which the content is being viewed may be continuously changing. The premise of augmented reality systems is often to allow for users to move about freely, and the resulting change of locations could include a change in ambient illumination that would impose an unnatural or undesirable color shift in the synthetic content.

[0003] When a real-world scene changes in average luminance, especially across large changes, the perception of the color of the synthetic imagery can change. An item of virtual content may be displayed using, for example, a set of RGB values for different pixels. Even if those pixel values do not change, the virtual content may appear different under different circumstances. An item of virtual content that appears normal under average indoor lighting conditions may appear unrealistically luminous when ambient lighting is dim— and it may appear washed-out when ambient lighting is bright— even though the RGB values used to display the object have not changed. In particular, large ambient brightness changes will significantly change the perception of the color of synthetic imagery. When unaccounted for, these changes in appearance impose an artificial and unnatural impression of the inserted content.

SUMMARY

[0004] Examples of systems and methods described herein improve the ability to maintain proper color perception of synthetic imagery under changing ambient illumination conditions. One such method is performed using an augmented reality display device such as a head-mounted display (HMD). The display device includes a primary display for displaying virtual objects and a dimmable filter that is external to the primary display. The display device includes an illuminance sensor, which may be a specialized sensor or a camera (e.g. a front-facing camera) for measuring the illuminance of a real-world scene. Internal to the dimmable filter, the real-world scene is visible with an apparent illuminance level, the apparent illuminance level being dependent on the illuminance of the scene and on the current dimming level of the dimmable filter. A target brightness of at least a portion of a virtual object is selected based at least in part on the measured illuminance. A target dimming level of the dimmable filter is also selected based at least in part on the measured illuminance. Display of the virtual object on the primary display is adjusted to the target brightness, and the dimming level of the dimmable filter is adjusted to the target dimming level. Changes to virtual object brightness and dimming level of the filter may be performed gradually to avoid sudden brightness changes that may be distracting to a user. In some embodiments, the luminance sensor is used to measure a white point of the real-world scene, and the virtual object is color balanced based on the measured white point.

[0005] In another example, a method is performed using an augmented reality display device that includes a primary display and a dimmable filter external to the primary display. The device measures the illuminance of a real-world scene. Internal to the dimmable filter, the real-world scene is visible with an apparent illuminance level, the apparent illuminance being dependent on the illuminance of the scene and on a current dimming level of the dimmable filter. The device also displays at least one virtual object on the primary display. The device selects a target brightness of at least a portion of the virtual object based at least in part on the measured illuminance. The device also selects a target dimming level of the dimmable filter based at least in part on the measured illuminance. The device adjusts the brightness of the portion of the displayed virtual object toward the target brightness. The brightness of the displayed object may be adjusted substantially instantaneously, or it may be adjusted gradually (e.g. in a stepwise manner with a plurality of incremental changes). The device further adjusts a dimming level of the dimmable filter toward the target dimming level. The dimming level of the dimmable filter may be adjusted substantially instantaneously, or it may be adjusted gradually (e.g. in a stepwise manner with a plurality of incremental changes).

[0006] In some embodiments, the target brightness is selected so as to be based on the target level of apparent illuminance (e.g. substantially proportional to the target level of apparent illuminance). In other embodiments, the target brightness is selected so as to be based on a current level of apparent illuminance (e.g. substantially proportional to the current level of apparent illuminance). In some embodiments, the brightness of the displayed virtual object is adjusted in real time to correspond to the current level of apparent illuminance.

[0007] The selection of the target dimming level may include selecting a target level of apparent illuminance and selecting the target dimming level so as to achieve the target level of apparent illuminance given the measured illuminance. The target level of apparent illuminance may be a maximum level of apparent illuminance, wherein the maximum level of apparent illuminance is selected to limit saturation of pixels of the primary display.

[0008] In some embodiments, the target brightness is selected so as to be (i) substantially proportional to measured illuminance for measured illuminance below a threshold illuminance level and (ii) substantially constant for measured illuminance above the threshold illuminance level. In some embodiments, the target dimming level (i) is selected so as to be substantially at a minimum dimming level for measured illuminance below a threshold illuminance level and (ii) is selected to provide a substantially constant level of apparent illuminance for measured illuminance above the threshold illuminance level. In some embodiments, the threshold illuminance level is based on the color of virtual objects displayed on the primary display, for example with the presence of lighter-colored virtual objects leading to a lower threshold illuminance level. The threshold illuminance level may be set so as to limit oversaturation of the primary display (such as may otherwise occur with attempts to render light-colored objects with a realistic level of brightness given the apparent illuminance).

[0009] In some embodiments, an increase in the illuminance of the real-world scene is detected while the virtual object is being displayed. One or more of the following actions may be taken in response to the increase in illuminance: (i) increasing the dimming level of the dimmable filter and (ii) increasing the brightness of the displayed virtual object. In some embodiments, only one of those actions is taken in response to the increase in illuminance.

[0010] In some embodiments, an augmented reality display device (such as a head-mounted display) includes a primary display, a dimmable filter external to the primary display, an illuminance sensor, a processor; and a non-transitory computer-readable storage medium storing instructions operative to perform the methods described herein, including the functions of: measuring the illuminance of a real-world scene; displaying at least one virtual object on the primary display; selecting a target brightness of at least a portion of the virtual object based at least in part on the measured illuminance; selecting a target dimming level of the dimmable filter based at least in part on the measured illuminance; adjusting the brightness of the portion of the virtual object toward the target brightness; and adjusting a dimming level of the dimmable filter toward the target dimming level.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG. 1 is a flowchart of a method of adjusting brightness and dimming on an augmented reality display.

[0012] FIGs. 2A-2D are schematic cross-sectional views of illustrative augmented reality head- mounted displays used in some embodiments. [0013] FIG. 3 is a flow diagram of a method of adjusting virtual object luminance and head-mounted display dimming according to some embodiments.

[0014] FIG. 4 is a flow diagram of another method of adjusting virtual object luminance and head- mounted display dimming according to some embodiments.

[0015] FIGs. 5A and 5B are schematic graphs illustrating relationships among environmental illuminance, dimming levels, and augmented object luminance used in some embodiments.

[0016] FIGs. 6A and 6B are schematic graphs illustrating additional relationships among environmental illuminance, dimming levels, and augmented object luminance used in some embodiments.

[0017] FIG. 7 is a flow diagram of a method of adjusting virtual object luminance and head-mounted display dimming according to some embodiments, taking into consideration different color components.

[0018] FIG. 8 is a message flow diagram of a method of adjusting virtual object luminance and head- mounted display dimming according to some embodiments.

[0019] FIG. 9 is a system diagram illustrating an example wireless transmit/receive unit (WTRU) that may be used as a head-mounted display in some embodiments.

DETAILED DESCRIPTION

Overview of Some Embodiments.

[0020] In an example of an embodiment, a method is provided for operating an augmented reality display device that includes a controllable ambient light blocking layer. In the method, information is received on the characteristics of a camera and on characteristics of optical transmission thorough augmented reality goggles which include an ambient light blocking layer. The camera captures an image of the scene and estimates a scene white point. A representation is received of synthetic content including a synthetic image and a reference white point. A degree of brightening used in rendering the synthetic content is determined, and a degree of ambient light to block is determined based on the estimated scene white point and the reference white point. The rate of temporal change in the control value used for the blocking layer and brightening applied to the rendered image may be controlled to prevent distractingly rapid brightness changes. The ambient light blocking layer is controlled based on the control value. In some embodiments, the selection of values used for blocking layer and for brightening are selected to balance the environmental dimming and the introduction of artifacts due to brightening the rendered content. In some embodiments, spatial dithering is used to achieve greyscale for a blocking layer with limited depth of modulation.

[0021] In another embodiment, an augmented reality display system is configured with a calibrated forward-facing camera capturing data from the scene as viewed by the observer. Data from this camera are used to estimate both the positional coordinates of the real world (to allow for accurate positioning of synthetic content) and the average absolute luminance of the scene. The luminance may be estimated in terms of absolute colorimetric coordinates or spectral radiance. Positional estimation may be made directly via scene markers or indirectly by alternative estimating schemes. The absolute luminance, in either colorimetric coordinates or spectral radiance, may be measured with a forward-facing detector. The detector may provide a spot measurement or an image-based measurement. A spot detector measuring absolute luminance may be correlated with camera data to estimate an image map of absolute luminance for the entire scene.

[0022] In some embodiments, synthetic content that is to be inserted into the scene is color balanced to the white point of the scene. This balancing may account for both the color coordinates of the white point and the absolute luminance. The rendered color of the synthetic content will then have a natural appearance under a given baseline scene luminance.

[0023] In some embodiments, the synthetic content is periodically adjusted to account for changes in the color or absolute luminance level. For very bright real-world scenes, the display properties of the AR display device may not be sufficient to retain acceptable contrast. For these situations, the AR display device operates to dim regions of the real-world scene that are not occluded by the synthetic content.

[0024] In some embodiments, dimming does not occur for the region in which the synthetic content is inserted. The positional information is used to maintain a natural location of the inserted content, and the dimming takes place across the scene except where the inserted imagery is currently located. The color of the rendered imagery is processed to account for the absolute scene luminance as it appears to a user after the dimming of the AR system. In some embodiments, the level of dimming is increased until the color of the synthetic content can be rendered in an accurate manner. The dimming may be done in a way that either preserves the real-world scene white point (neutral dimming, with no color correction) or it the dimming may be done so as to apply a color shift to change the perceived white point (if color correction is desired).

[0025] The control of accurate color is preferably not accomplished only by dimming the real-world imagery. Therefore, to the extent possible in the AR display, the synthetic content may also be boosted or suppressed in concert with dimming. This may produce more accurate and representative color of the synthetic content. In response to changes in the absolute luminance of the scene, the accompanying change in appearance of the synthetic content (after re-rendering and possible dimming) will follow natural expectations, and the perception of the synthetic content will be more accurate and realistic.

[0026] Illustrative embodiments provide for an increased range of real-world luminance levels that can be accommodated while maintaining high color fidelity of the synthetic content. Without dimming, a very bright real-world scene would quickly overwhelm the brightness capability of the AR display. [0027] The amount of real world dimming and synthetic content boosting may be selected so as to balance the boosting limitations of the AR display (e.g. the maximum luminance that can be displayed by the AR display), and the experience of the observer (e.g. limiting the dimming of the real world so as not to distract from the observer's perception of the scene).

[0028] When the initial inserted content is over a high-luminance real world scene, maintaining a good perceptual experience when luminance drops may be implemented by reducing the brightness of the synthetic content by an amount corresponding to the luminance drop in the real world, thereby providing an acceptable observer experience.

[0029] Upon the initial insertion of synthetic content, some dimming and/or boosting may be used to improve perception of the synthetic content.

Examples of Methods and Devices.

[0030] FIG. 1 is a flowchart of an exemplary process of inserting synthetic content and adjusting the brightness of the content ("boost") and/or dimming the real-world scene to account for luminance changes in the real-world scene. As illustrated in FIG. 1 , a calibrated forward-facing camera captures the field of view of the wearer (step 150) and transmits the image to a processor. The captured image may be a color image. The processor operates to determine the current scene illuminance level (step 152) based on the captured image and the stored properties of the head-mounted display (e.g. AR goggles). If it is determined (in step 154) that the virtual content has not yet been inserted into the scene, the system stores the measured luminance (step 164). Based on the measured luminance, the system determines (step 166) whether any dimming or boosting of the content is called for. If not, the content is rendered and displayed (step 168) without dimming or boosting of the content. If dimming or boosting is called for, that dimming or boosting may be applied using, e.g. the process described in detail below.

[0031] If it is determined (in step 154) that the virtual content has already been inserted into the scene, then the newly-measured luminance is compared to a previously-stored luminance (step 156) to determine whether there has been any noticeable change in luminance. If it is determined (in step 158) that there has not been a noticeable change, the virtual content may be rendered (step 170) without any adjustment to the amount of dimming or boosting of the object. The rendering may, however, take into consideration any change in the position of the virtual content and/or in the position or direction of the viewer.

[0032] If it is determined (in step 158) that there has been a noticeable change to luminance, adjustment to dimming and/or boosting may be performed as follows. The processor operates to determine desired (e.g. optimum) dimming and boosting levels (step 160). This optimization may consider different metrics, such as the amount of dimming that the observer can tolerate and the effect accomplished by boosting the AR display to its maximum. For especially bright real-world scenes, the result may be a compromise that is apparent to the user; however, this may be preferable to an unacceptably-high level of dimming, which may be even more objectionable to a user.

[0033] In some embodiments, the difference from the prior dimming level to the currently-selected dimming level (as selected in step 160) is divided into a plurality of equal increments, and the difference from the prior boost level to the currently-selected boost level is divided into a plurality of equal increments. The boost levels and dimming levels may be gradually adjusted (step 162) to the currently-selected boost and dimming levels, changing the boost and dimming levels one increment at a time with a predetermined pause in between changes of increment. The virtual content is re-rendered (step 163) with the new dimming and boost levels. The re-rendering may take into consideration any movement of the virtual content and/or of the user.

[0034] In some embodiments, methods to select the degree of brightening possible with a given image are adapted from methods used in determining the amount of backlight dimming to use in an LCD. For instance, in some embodiments, the histogram of image code values is computed, and the brightening factor is selected based on the percentage of pixels, including none, that are clipped. A technique that may be employed in some embodiments is the technique described in Kerofsky, L. "LCD Backlight Selection through Distortion Minimization," IDW2007, DES1-3 (2007): 315-318, which compares a metric based on the percentage of clipped pixels and a sum of square error criteria for backlight. The technique of Kerofsky may be adapted to embodiments of the present disclosure by converting the backlight value of Kerofsky to a brightening amount for images of synthetic objects. Alternative embodiments may use different techniques to select the brightening factor and enhance the image, such as the methods described in Xu, Xinyu & L. Kerofsky, "Improving content visibility for high-ambient-illumination viewable display and energy- saving display," Journal of the Society for Information Display 19.9 (2011): 645-654.

[0035] Some embodiments make use of a dimming filter on the real-world side (external side) of the AR display. In some embodiments, the dimming filter may be spatially-controllable, such that dimming can be different in different portions of the filter. The dimming filter may be of limited resolution and may be binary.

[0036] The dimming filter may have any of several different configurations. In some embodiments, the dimming filter has both imaging and dimming capability. The spatial resolution of such a filter may be limited, e.g. the spatial resolution of the dimming filter may be lower than the spatial resolution of the display. In embodiments in which the dimming filter has at least some imaging capability, the outer filter may dim different areas of the real world. The imaging capability may be limited (e.g. dithering with only binary control of each pixel). In some embodiments, the dimming filter may be operated so as to leave an undimmed mask corresponding to the location of the inserted content. [0037] In some embodiments, the dimming filter is not spatially-controlled and thus applies dimming to the entire real-world scene. In such embodiments, the dimming filter has only global dimming capability. Depending on the degree of "show through" the inner display allows, this may be indistinguishable from an embodiment with spatially-controllable dimming filter. That is, if the augmented content is totally opaque where content is inserted and totally transparent where there is no inserted content, then the observer experience may be identical to an embodiment with spatially-controllable dimming filter. However, if there is "show through" then the brightness of the inserted content may be increased by an amount corresponding to the dimming of the real world by the outer filter.

[0038] In some embodiments, the AR display itself operates as a dimming filter. This may be implemented where there is a controllable amount of transparency in the AR display. In such embodiments, the regions of the display outside the synthetic imagery may be used as neutral or colored filters for the real-world scene.

[0039] In further embodiments, the AR display has no dimming capability and only brightening of the content shown on the AR display is possible. In such embodiments, the brightness of the rendering of image content may be controlled based on the environment. The display of AR content may brighten the content for display depending on the measure of the environment.

[0040] Embodiments described herein may be implemented using various different types of augmented reality displays. Examples of head-mounted displays that can be employed in exemplary embodiments are illustrated schematically in FIGs. 2A-2D. In the embodiment of FIG. 2A, virtual content is rendered using a liquid-crystal display (LCD) 202. A partly-transparent mirror 204 (e.g. a half-silvered mirror or a beam splitter) allows a user to see both the real world and an image of the content displayed by the LCD. Viewing optics 206 is provided to enable the user to focus on the virtual content. The LCD, viewing optics, and partly-transparent mirror are components of a primary display. A dimmable filter 208 is mounted at a position that is external to the primary display. The dimmable filter is provided to enable dimming of the scene from the perspective of the user without dimming of virtual objects displayed by the primary display. The dimmable filter may be, for example, an electrochromic optical device or a polymer dispersed liquid crystal device, among other possibilities. In some embodiments, the dimmable filter has a plurality of separately-controllable dimmable regions (which may be individual pixels).

[0041] The head-mounted display of FIG. 2A includes a forward-facing camera 210 that operates to assist in tracking the current orientation of the user's head. In some embodiments, the forward-facing camera is also used to measure the optical properties (e.g. the illuminance) of the current real-world scene. In other embodiments, one or more separate illuminance sensors are used to measure optical properties of the scene. A control module 212 uses information on the illuminance (and in some embodiments, the white point) of the scene to control the dimming level of the dimmable filter and to control the brightness of the virtual content displayed on the LCD. It should be noted that while LCD displays are used herein as an example, other types of displays may alternatively be used.

[0042] The head-mounted display of FIG. 2B is similar to the head-mounted display of FIG. 2A, except that partly-transparent mirror 205 is curved so as to perform at least some of the functions of the viewing optics 206 of FIG. 2A. The head-mounted display of FIG. 2C is similar to the head-mounted display of FIG. 2B except that the dimmable filter 209 is incorporated into an external aspect of a curved partly- reflective surface 203. The head-mounted display of FIG. 2D employs a waveguide instead of conventional geometric optics to make the image of the LCD visible to the user, with the dimmable filter 208 being external to the waveguide. In some embodiments, the head-mounted display is a device such as those described in US 2016/0055822, filed Aug. 21 , 2014, entitled "Dimming Module for Augmented and Virtual Reality."

[0043] An example of a method for operating a dimmable head-mounted display according to some embodiments is illustrated in FIG. 3. The system operates to measure the ambient illumination, e.g. using a camera of the head-mounted display of using a specialized illuminance sensor (step 302). Based on a current level of dimming by the head-mounted display, the system calculates an apparent level of illumination from the perspective of the user behind the dimming filter (step 304). For example, if the ambient illumination level is measured to be I, and the dimming filter is at a dimming level of 50%, the apparent illumination level may be calculated to be 0.5 J. In an alternative embodiment, the apparent illumination level may be measured directly by an illuminance sensor positioned behind the dimming filter.

[0044] The system then determines a desired luminance of a virtual object (step 306). The desired luminance may be proportional to (or may bear another functional relationship with) the apparent illumination level, so that as the apparent illumination increases, the luminance of the virtual object increases, just as it would if it were a physical object being illuminated by the ambient light. The system then determines whether the desired luminance is possible given limitations on the output of the head- mounted display (step 308). If the desired luminance is possible, the virtual object is displayed with the desired luminance (step 310). If the desired luminance is not feasible, e.g. if an LCD of the head-mounted display is not capable of outputting sufficient light to generate the desired luminance, or if the desired luminance would be uncomfortably bright to a user, then the system increases the level of dimming (step 312). This increase may be performed slowly so as not to draw the user's attention. The method of FIG. 3 may be performed iteratively, with the level of dimming being performed incrementally until the background is sufficiently dim that the desired level of luminance of the virtual object can be achieved given brightness limitations of the display. Conversely, if dimming is not required, the level of dimming may be gradually decreased until the dimming lens is at full transparency or until dimming is required. [0045] The determination of whether a virtual object can be displayed with the desired level of luminance may be performed on a pixel-by-pixel basis. For example, it may be the case that the desired luminance of some pixels may be greater than that achievable by the display, but that this is true only of a few pixels as compared to the virtual object as a whole. It may be desirable to forego background dimming in such a case. Thus, in some embodiments, background dimming occurs only if the desired luminance is unachievable for a threshold number of pixels, or only for a threshold percentage of the pixels comprising the virtual object.

[0046] A further embodiment is illustrated in FIG. 4. In the method of FIG. 4, the system tracks the desired luminance of virtual objects (step 404) and the level of ambient illumination (step 402). Based on the desired luminance and the ambient illumination, the system selects a target brightness (e.g. a target luminance) and dimming level for the different virtual objects, e.g., for each of the pixels that comprise the virtual object (step 406). The system then adjusts the luminance and background dimming levels toward the target level (step 408). This adjustment may be performed at a slow rate to prevent any rapid changes in brightness that might be distracting to a user. The process of FIG. 4 may be performed continually to update luminance and dimming levels as both ambient illumination and the virtual content may be changing over time.

[0047] Selection of dimming levels and target luminance levels may be performed using techniques described with respect to the schematic graphs of FIGs. 5A-5B. Augmented reality objects may have a more realistic appearance when their luminance increases proportionately with the illuminance of the environment, as if those objects were physically present in the environment and were being lit by the ambient illumination. The luminance of lighter-colored virtual objects is selected to increase at a greater slope than the luminance of darker-colored virtual objects, as lighter-colored objects would be expected to reflect a greater proportion of incident light.

[0048] In the example illustrated in FIG. 5A, the luminance of a white virtual object is increased with increasing environmental illuminance until, at a threshold illuminance level A, the luminance of the white virtual object cannot be further increased due to limits of the display. Thus, with increasing illuminance of the environment beyond threshold illuminance level A, the amount of dimming applied by the dimming filter is increased so as to maintain the apparent level of illuminance at illuminance level A. Because the apparent illuminance is kept constant, the luminance of the light-colored dark-colored virtual objects need not be increased beyond the level used for illuminance level A. In the example illustrated in FIG. 5B, no white virtual objects are present, and the illuminance of the environment can reach a higher level- threshold illuminance level B— before dimming is called for.

[0049] In view of the examples of relationships illustrated in FIGs. 5A-5B, a method of selecting target dimming levels and target luminance levels may be performed as follows. A pixel of a virtual object may be associated with a baseline pixel value po (which may be any one of red, green, or blue value, for example, or a luma value). To give the appearance that the virtual object is illuminated by the real-world ambient lighting, it may be desirable to display the pixels with values proportional to the apparent illuminance from the perspective of the user. The actual illuminance may be measured by the user's headset as / (expressed here as a unitless value, which may be measured relative to a predetermined standard). An illumination-compensated pixel value for the pixels may thus be calculated as po*J The primary display, however, may be limited in generating luminance no greater than p_max. For any particular pixel, the value of illuminance that would cause that pixel to reach its maximum value is pmax/po. Thus, in some embodiments, a maximum value of illuminance Imax is determined, where Imax may be selected to prevent oversaturation of an excess number of pixels of virtual objects. A pixel is oversaturated where po*7 > pmax for that pixel. In some embodiments, Imax is selected to be low enough such that no pixels are oversaturated, and Imax≤ Pmax/po for all pixels of all virtual objects in a particular scene. In some embodiments, Imax may be a higher value selected such that, for example, no more than a threshold number of pixels, or no more than a threshold percent of pixels, are oversaturated.

[0050] In this example embodiment, target values of virtual object luminance and target dimming levels may be selected as follows. Illuminance / of the real-world scene is measured. If / is less than or equal to the determined Imax, then the target illuminance of a pixel of a virtual object is selected to be po*/, and the target level of dimming is selected to provide full transparency. If / is greater than the determined Imax, then the target illuminance of a pixel of a virtual object is selected to be po*Imax, and the target level of dimming is selected such that the apparent level of illuminance as seen by the user is equal to Imax. Where the level of dimming is represented as a dimming factor D, the target level of dimming may be expressed as In some embodiments, the target levels of luminance and dimming are not used right away, but rather the display gradually transitions to the target levels to avoid rapid changes in brightness level that may appear unrealistic to a user.

[0051 ] In the examples of FIGs. 5A and 5B, the luminance of an AR object is proportional to the illuminance of the environment for illuminance below a threshold illuminance level (level A or B), and the luminance of the AR object is constant for measured illuminance above the threshold illuminance level. The dimming level is selected so as to be substantially at a minimum dimming level for environmental illuminance below the threshold illuminance level and so as to provide a substantially constant level of apparent illuminance for measured illuminance above the threshold illuminance level.

[0052] FIGs. 6A and 6B are similar to FIGs. 5A-5B except that in FIGs. 6A-6B, the curves representing object luminance and dimming percentage are smoothed to reduce the appearance to a user of sharp changes in brightness. In addition, FIGs. 6A-6B illustrate that luminance of virtual objects does not necessarily fall to zero in the absence of any real-world illumination. In the examples of FIGs. 6A and 6B, the luminance of an AR object is substantially proportional to the illuminance of the environment for illuminance below a threshold illuminance level (level A or B), and the luminance of the AR object is substantially constant for measured illuminance above the threshold illuminance level. The dimming level is selected so as to be substantially at a minimum dimming level for environmental illuminance below the threshold illuminance level and so as to provide a substantially constant level of apparent illuminance for measured illuminance above the threshold illuminance level.

[0053] Some embodiments operate to take into consideration not just the overall intensity of illumination but also the color of the illumination to perform "color balancing" or "white balancing." For example, if the real-world illumination is red-tinted, then the red component of virtual objects may be increased (e.g. through multiplication by a factor) to enhance the impression that the virtual object is part of the ambient scene and is being illuminated by the ambient illumination. If the desired increase in the red component would exceed the capabilities of the display, then dimming by the dimming filter may be applied to reach a point such that the appropriate level of the red component is within the capabilities of the display.

[0054] An example of a method that performs color balancing (or white balancing) is illustrated in FIG. 7. The system operates to measure the red, green and blue (RGB) levels of ambient illumination, e.g. using a camera of the head-mounted display of using a specialized illuminance sensor (step 702). Based on a current level of dimming by the head-mounted display, the system calculates an apparent level of illumination from the perspective of the user behind the dimming filter (step 704). For example, if the ambient illumination levels are measured to be IR, IG, and IB, and the dimming filter is at a dimming level of 50%, the apparent illumination levels may be calculated to be 0.5*/R, 0.5X/G, and 0.5x/s. The system then determines a desired luminance of a virtual object (step 706). The desired luminance of the virtual object may be proportional to (or may bear another functional relationship with) the color components of the apparent illumination level, so that as the apparent illumination of different components increases, the luminance of the corresponding component of the virtual object increases, just as it would if it were a physical object being illuminated by the ambient light. The system then determines whether the desired luminance of the relevant component is feasible given limitations on the output of the head-mounted display (step 708). If the desired luminance is possible, the virtual object is displayed with the desired luminance (step 710). If the desired luminance is not feasible, e.g. if an LCD of the head-mounted display is not capable of outputting sufficient light to generate the desired luminance, then the system increases the level of dimming (step 712). This increase may be performed slowly so as not to draw the user's attention. The method of FIG. 7 may be performed iteratively, with the level of dimming being performed incrementally until the background is sufficiently dim that the desired level of luminance of the virtual object can be achieved given brightness limitations of the display. Conversely, if dimming is not required, the level of dimming may be gradually decreased until the dimming filter has reached full transparency or until additional dimming is required.

[0055] In some embodiments, white balancing or color balancing may be performed on an augmented reality display device that is not equipped with a dimming filter. In such embodiments, the luminance of different color components of virtual objects may be adjusted based on measured ambient illuminance of different color components. If the adjusted luminance of any color component of a virtual object exceeds the maximum brightness of which the display is capable, the display may display that component at the maximum available brightness, or it may scale down the brightness of the entire virtual object to prevent an appearance of oversaturation.

[0056] A method according to some embodiments is illustrated in FIG. 8. In the method of FIG. 8, a camera 802 captures a scene (step 812) and conveys the scene image (step 814) and camera characteristics (step 810) to an image processor 804. The image processor uses the image and the camera characteristics to estimate the white point of the scene (step 816) and to determine an amount of content brightening (step 818) for virtual content. The image processor further operates to determine an amount of dimming to be employed by a blocking layer (step 820). A temporal control may be applied (step 822) such that brightness adjustments are performed gradually. A control signal (e.g. a voltage) is conveyed (step 824) to the blocking layer 806, which operates to apply the signaled level of dimming (step 826). The image processor further computes a rendered image of the virtual content (step 828) and signals the image to a head-mounted display 808, which operates to display the image (step 832).

[0057] In some embodiments, a method is performed using an augmented reality display device (such as a head-mounted display) that has a primary display and a dimmable filter external to the primary display. In the method, the display device measures the illuminance of a real-world scene (e.g. using a camera on the device). The device selects a target brightness of at least a portion (e.g. at least a pixel) of a virtual object based at least in part on the measured illuminance. The device also selects a target dimming level of the dimmable filter based at least in part on the measured illuminance. The device displays the virtual object on the primary display with the target brightness. The device further adjusts a dimming level of the dimmable filter to the target dimming level. In some embodiments, the target brightness of the virtual object is proportional to the target level of apparent illuminance.

[0058] In some embodiments, the target dimming level is selected by (i) selecting a target level of apparent illuminance; and (ii) selecting the target dimming level so as to achieve the target level of apparent illuminance given the measured illuminance. The target level of apparent illuminance may be a maximum level of apparent illuminance, wherein the maximum level of apparent illuminance is selected to limit saturation of pixels of the primary display. [0059] In some embodiments, the device measures a white point of the real-world scene and performs color correction on the virtual object based on the measured white point.

[0060] In some embodiments, the dimmable filter includes a plurality of pixels. Dimming may be performed only for regions of the dimmable filter that do not correspond to any virtual object. The dimming levels of the dimming filter may be achieved by dithering of the pixels.

[0061] In some embodiments, a method is performed by an augmented reality display device (such as a head-mounted display) that includes a primary display and a dimmable filter that is external to the primary display. The device measures the illuminance of a real-world scene (e.g. using a camera of the device). The device calculates an apparent illuminance based on the measured illuminance and a current dimming level of the dimmable filter. The device selects a target brightness of a virtual object based at least in part on the apparent illuminance. (For example, the target brightness may be scaled by the apparent illuminance.) The device determines whether the target brightness exceeds a brightness achievable by the primary display. In response to a determination that the target brightness exceeds a brightness achievable by the primary display, the dimming level of the dimming filter is increased. The device may further display the virtual object on the primary display with the target brightness.

[0062] In some embodiments, a method is performed by an augmented reality display device (such as a head-mounted display) that includes a primary display and a dimmable filter that is external to the primary display. The device determines an apparent illuminance behind the dimmable filter (e.g. through direct measurement by a camera or other illuminance sensor behind the filter, or by a measurement in front of the filter that is scaled based on the dimming level of the filter). The device selects a target brightness of a virtual object based at least in part on the apparent illuminance and determines whether the target brightness exceeds a brightness achievable by the primary display. In response to a determination that the target brightness exceeds a brightness achievable by the primary display, the device increases the dimming level of the dimming filter. The device may further display the virtual object on the primary display with the target brightness.

[0063] In some embodiments, an augmented reality display device is provided that includes a primary display, a dimmable filter external to the primary display, an illuminance sensor, a processor, and a non- transitory computer-readable storage medium. The storage medium stores instructions that are operative to perform functions including (i) measuring the illuminance of a real-world scene with the illuminance sensor; (ii) selecting a target brightness of at least a portion of a virtual object based at least in part on the measured illuminance; (iii) selecting a target dimming level of the dimmable filter based at least in part on the measured illuminance; (iv) displaying the virtual object on the primary display with the target brightness; and (v) adjusting a dimming level of the dimmable filter to the target dimming level. [0064] In some embodiments, an augmented reality display device is provided that includes a primary display, a dimmable filter external to the primary display, an illuminance sensor, a processor, and a non- transitory computer-readable storage medium. The storage medium stores instructions that are operative to perform functions including (i) measuring the illuminance of a real-world scene; (ii) calculating an apparent illuminance based on the measured illuminance and a current dimming level of the dimmable filter; (iii) selecting a target brightness of a virtual object based at least in part on the apparent illuminance; (iv) determining whether the target brightness exceeds a brightness achievable by the primary display; (v) and in response to a determination that the target brightness exceeds a brightness achievable by the primary display, increasing the dimming level of the dimming filter.

Exemplary Logical Hardware

[0065] Note that various hardware elements of one or more of the described embodiments are referred to as "modules" that carry out (i.e., perform, execute, and the like) various functions that are described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable by those of skill in the relevant art for a given implementation. Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and it is noted that those instructions could take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and/or the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM, ROM, etc.

[0066] FIG. 9 is a system diagram illustrating an example wireless transmit-receive unit (WTRU) 102, which may be implemented as a head-mounted display. As shown in FIG. 9, the WTRU 102 may include a processor 118, a transceiver 120, a transmit/receive element 122, a speaker/microphone 124, a keypad 126, a display/touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and/or other peripherals 138, among others. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0067] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit/receive element 122. While FIG. 9 depicts the processor 118 and the transceiver 120 as separate components, it will be appreciated that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0068] The transmit/receive element 122 may be configured to transmit signals to, or receive signals from, a base station over the air interface 116. For example, in one embodiment, the transmit/receive element 122 may be an antenna configured to transmit and/or receive RF signals. In an embodiment, the transmit/receive element 122 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit/receive element 122 may be configured to transmit and/or receive both RF and light signals. It will be appreciated that the transmit/receive element 122 may be configured to transmit and/or receive any combination of wireless signals.

[0069] Although the transmit/receive element 122 is depicted in FIG. 9 as a single element, the WTRU 102 may include any number of transmit/receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit/receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0070] The transceiver 120 may be configured to modulate the signals that are to be transmitted by the transmit/receive element 122 and to demodulate the signals that are received by the transmit/receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11 , for example.

[0071] The processor 118 of the WTRU 102 may be coupled to, and may receive user input data from, the speaker/microphone 124, the keypad 126, and/or the display/touchpad 128 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker/microphone 124, the keypad 126, and/or the display/touchpad 128. In addition, the processor 118 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and/or the removable memory 132. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown). [0072] The processor 118 may receive power from the power source 134, and may be configured to distribute and/or control the power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0073] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 116 from a base station and/or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0074] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and/or hardware modules that provide additional features, functionality and/or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs and/or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a Virtual Reality and/or Augmented Reality (VR/AR) device, an activity tracker, and the like. The peripherals 138 may include one or more sensors, the sensors may be one or more of a gyroscope, an accelerometer, a hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and/or a humidity sensor.

[0075] The WTRU 102 may include a full duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for both the UL (e.g., for transmission) and downlink (e.g., for reception) may be concurrent and/or simultaneous. The full duplex radio may include an interference management unit to reduce and or substantially eliminate self-interference via either hardware (e.g., a choke) or signal processing via a processor (e.g., a separate processor (not shown) or via processor 118). In an embodiment, the WRTU 102 may include a half-duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for either the UL (e.g., for transmission) or the downlink (e.g., for reception)).

[0076] Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

CLAIMS What is claimed is:

1. A method performed using an augmented reality display device comprising a primary display and a dimmable filter external to the primary display, the method comprising:

measuring the illuminance of a real-world scene;

displaying at least one virtual object on the primary display;

selecting a target brightness of at least a portion of the virtual object based at least in part on the measured illuminance;

selecting a target dimming level of the dimmable filter based at least in part on the measured illuminance;

adjusting the brightness of the portion of the virtual object toward the target brightness; and adjusting a dimming level of the dimmable filter toward the target dimming level.

2. The method of claim 1 , wherein selecting the target dimming level comprises:

selecting a target level of apparent illuminance; and

selecting the target dimming level so as to achieve the target level of apparent illuminance given the measured illuminance.

3. The method of claim 2, wherein selecting the target brightness comprises selecting a target brightness proportional to the target level of apparent illuminance.

4. The method of claim 2 or 3, wherein selecting the target level of apparent illuminance comprises

selecting a maximum level of apparent illuminance, wherein the maximum level of apparent illuminance is selected to limit saturation of pixels of the primary display.

5. The method of claim 1 , wherein the target brightness is selected so as to be (i) substantially proportional to measured illuminance for measured illuminance below a threshold illuminance level and (ii) substantially constant for measured illuminance above the threshold illuminance level.

6. The method of claim 5, wherein the target dimming level is selected (i) so as to be substantially at a minimum dimming level for measured illuminance below the threshold illuminance level and (ii) to provide a substantially constant level of apparent illuminance for measured illuminance above the threshold illuminance level.

7. The method of claim 1 , wherein the target dimming level is selected (i) so as to be substantially at a minimum dimming level for measured illuminance below a threshold illuminance level and (ii) to provide a substantially constant level of apparent illuminance for measured illuminance above the threshold illuminance level.

8. The method of any of claims 5-7, further comprising setting the threshold illuminance level based on the color of virtual objects displayed on the primary display.

9. The method of any of claims 5-8, further comprising setting the threshold illuminance level so as to limit oversaturation of the primary display.

10. The method of claim 1 , further comprising:

while displaying the virtual object, detecting an increase in the illuminance of the real-world scene; and

in response to the increase in illuminance, increasing the dimming level of the dimmable filter or increasing the brightness of the displayed virtual object.

11. The method of any of claims 1-10, wherein adjusting the dimming level of the dimmable filter to the target dimming level is performed gradually.

12. The method of any of claims 1-11, wherein adjusting the dimming level of the dimmable filter to the target dimming level is performed using a plurality of incremental changes to the dimming level.

13. The method of any of claims 1-12, wherein adjusting the brightness of the portion of the virtual object toward the target brightness is performed gradually.

14. The method of any of claims 1-13, wherein selecting the target brightness comprises selecting a target brightness proportional to a current level of apparent illuminance.

15. An augmented reality display device comprising:

a primary display;

a dimmable filter external to the primary display

an illuminance sensor;

a processor; and

a non-transitory computer-readable storage medium storing instructions operative to perform functions comprising:

measuring the illuminance of a real-world scene;

displaying at least one virtual object on the primary display;

selecting a target brightness of at least a portion of the virtual object based at least in part on the measured illuminance; selecting a target dimming level of the dimmable filter based at least in part on the measured illuminance;