TW202331352A - Ambient light sensors and camera-based display adjustment in smart glasses for immersive reality applications - Google Patents

Abstract

A headset for use with immersive reality applications is provided. The headset includes a left eyepiece and a right eyepiece mounted on a frame, a display in at least one of the left eyepiece or the right eyepiece, the display comprising an array of multiple light emitting pixels, an ambient light sensor to measure an amount of ambient light, and a processor configured to control a light intensity of the light emitting pixels based on the amount of ambient light. A method for using the above headset is also provided.

Description

Ambient Light Sensor and Camera Based Display Adjustment in Smart Glasses for Immersive Reality Applications

The present invention relates to configuring display settings for different ambient light conditions in smart glasses. More particularly, embodiments disclosed herein relate to optimizing display appearance for a user-embedded environmental context. Cross-references to related applications

This application is related to and claims priority under 35 USC §119(e): U.S. Provisional Application entitled AMBIENT LIGHT SENSOR AND CAMERA BASED DISPLAY ADJUSTMENT filed November 17, 2021 by Sebastian SZTUK et al 63/280,520; and U.S. Provisional Application No. 63/2022, entitled AMBIENT LIGHT SENSORS AND CAMERA-BASED DISPLAY ADJUSTMENT IN SMART GLASSES FOR IMMERSIVE REALITY APPLICATIONS, filed February 23, 2022 by Scott J. WOLTMAN et al. 313,008; and U.S. Nonprovisional Application No. 17/949,934, filed September 21, 2022, by Sebastian SZTUK et al., entitled AMBIENT LIGHT SENSORS AND CAMERA-BASED DISPLAY ADJUSTMENT IN SMART GLASSES FOR IMMERSIVE REALITY APPLICATIONS, which The content of the application is hereby incorporated by reference in its entirety for all purposes.

Smart glasses for augmented reality (AR) applications pose the challenge of adapting brightness and other display characteristics to widely varying environmental conditions. In fact, throughout the day, indoors and outdoors, the range of exterior brightness encountered by a user can vary widely, and the level of contrast required in an AR display will vary accordingly. Not only brightness changes, but also hue and tint as each of the color separations (eg, red-R-, green-G-, and blue-B-) are affected differently by environmental conditions. Current smart glasses displays do not adequately address this challenge, having only a few different configurations with little continuity and adaptability to the wide range of conditions experienced by users.

In a first embodiment, an apparatus includes: a left eyepiece and a right eyepiece mounted on a frame; a display in at least one of the left eyepiece or the right eyepiece, the display including a plurality of light emitting pixels an array; an ambient light sensor; a camera or combination thereof to measure an amount of ambient light, and a processor configured to control one of the light-emitting pixels based on the amount of ambient light Light intensity and color profiles.

In a second embodiment, a computer-implemented method includes: receiving a signal from an ambient light sensor indicative of an amount of ambient light in an environment of a headset; Determining a characteristic of a virtual image provided to a user based on the amount of ambient light in the environment; and controlling one of light-emitting pixels in a display of the head-mounted kit based on the characteristic of the virtual image brightness. The computer-implemented method can also include controlling an appearance of a virtual object based on the amount of ambient light in the environment of the headset.

In a third embodiment, a system includes a memory storing instructions, and a processor configured to execute the instructions that, when executed, cause the system to: receive indications from an ambient light sensor A signal of an amount of ambient light in an environment of a headset; determining an amount of a virtual image provided to a user based on the amount of ambient light in the environment of the headset and controlling a light intensity of a plurality of light-emitting pixels in a display of the head-mounted kit based on the characteristic of the virtual image.

In yet another embodiment, a system includes first means for storing instructions and second means configured to execute the instructions that, when executed, cause the system to: sense from an ambient light The device receives a signal indicative of an amount of ambient light in an environment of a headset; a virtual light provided to a user is determined based on the amount of ambient light in the environment of the headset. a characteristic of the image; and controlling a light intensity of light-emitting pixels in a display of the head-mounted kit based on the characteristic of the virtual image.

These and other specific examples will become apparent based on the following disclosure.

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, to one of ordinary skill in the art, that embodiments of the invention may be practiced without some of these specific details. In other instances, well-known structures and techniques have not been shown in detail in order not to obscure the disclosure.

As users of smart glasses transition from indoors to outdoors, or from inside a vehicle to outside, and from early morning through midday to evening and late at night, AR displays are expected to provide well-balanced, clear, clear and comfortable color gamut and brightness.

To solve the above challenges, the smart glasses disclosed herein include an ambient light sensor (ALS). Accordingly, the display of the smart glasses may have adjustable settings that allow brightness, gray scale, and white balance to be adjusted according to the ALS signal provided by the ALS sensor. This adjustment may be based on a predetermined user condition, such as some degree of color vision deficiency or color blindness.

The ALS sensor can be a single photodiode configured for broadband measurements, a multi-element photodiode, or even have multiple pixels (including points in a two-dimensional (2D) array) RGB pixels at the camera). In the case of a single photodiode, the ALS adjustment is based on a single data point of ambient brightness and color. In the case of cameras, ALS adjustments can more closely compensate for the portion of the real world that exists behind the display from the user's perspective.

In some embodiments, the camera can be used to assess, identify and verify the surrounding environment as initially detected by a single ALS sensor, or in combination with a single photodiode when the ALS signal has low reliability or large fluctuations Work. For example, in some specific instances, the camera can perform a quick scene check to ensure accurate tracking of the surrounding environment (e.g. nighttime scenes by capturing the moon and stars, daytime scenes by detecting the rising or setting sun, and the like).

In some embodiments, the smart glasses include an electronically adjustable mechanism to adjust the transparency of one or both eyepieces based on the ALS signal, and combine the dimming adjustment with a previously calibrated display setting or an adjusted display setting. In addition to factory calibration, in some embodiments smart glasses can also be configured for field recalibration under controlled lighting conditions (eg, inside a charging case and the like). In some embodiments, the user of the smart glasses can manually adjust the brightness and color settings in the smart glasses according to her/his own preferences independently of the automatic adjustment of the smart glasses. To adjust smart glasses settings, user input can include voice commands, touch sliders, input from a paired mobile device (e.g., smartphone), or physical buttons to manually override automatic settings when the user prefers to manually override the automatic settings. Change between states and brightness levels. Thus, some embodiments adjust display settings based on lens tinting and lens tint (stock keeping unit (SKU)) for normal, daylight, and dimming lenses with different lens tinting.

1 illustrates smart glasses 100 including an ambient light sensor 125 for augmented reality applications, according to some embodiments. The smart glasses 100 include a frame 111 holding left eyepieces ( 105L ) and right eyepieces ( 105R ) (hereinafter collectively referred to as “eyepieces 105 ”), a processor 112 , a memory 120 , a communication module 118 and a forward-looking camera 123 . Additionally, and as part of the user interaction system, the smart glasses 100 may include a speaker/microphone 121 so that the user can provide voice commands and receive audio feedback. To assess environmental conditions, smart glasses 100 may include one or more sensors 125 configured as ambient light sensors, acoustic wave detectors, and the like, such as inertial motion units (IMUs, such as acceleration meter or gyroscope) to help determine whether the smart glasses are in use or idle. In some embodiments, the sensor 125 may include a touch-sensitive controller and sensor. In some embodiments, input from touch sensors can be used in machine learning algorithms for gesture recognition. The ambient light sensor 125 can be configured to detect visible light (visible light; VIS, 450 nm to 750 nm), ultraviolet light (ultraviolet light; UV, 200 nm to 450 nm wavelength), infrared light (infra-red light ; IR, 750 nm to 10 µm wavelengths) or any other desired wavelength range. For example, in some embodiments, a UV detector may indicate the presence of direct sunlight (eg, the user is outside and/or on a bright sunny day). In some embodiments, eyepiece 105 may include active components, such as a liquid crystal layer configured to provide variable coloring or dimming of eyepiece 105 . Therefore, the transparency of the smart glasses 100 can be adjusted automatically or under user control according to environmental conditions or user needs.

Memory circuitry 120 stores instructions that, when executed by processor 112 , cause smart glasses 100 to perform at least some of the steps and operations disclosed herein. For example, the instructions stored in the memory 120 may be part of an application program installed in the mobile device 110 and hosted by the remote server 130 . An application can be configured to pair the mobile device 110 with the smart glasses 100 , retrieve data from it, and provide instructions and updates to the smart glasses 100 . For example, the mobile application may include a user assistant to control and adjust settings in the smart glasses 100 and even provide instructions and set configuration modes for the smart glasses 100 . In addition, the camera 123 can capture images or videos of the user's forward field of view. This image or video can be used by the processor 112 or an application in the mobile device 110 to review, examine and analyze the user's environment and draw decisions about steps to take based on the environment. Additionally, in some embodiments, camera 123 may include a shutter configured to collect light from the user at a preselected time rate of change or aperture based on the amount of ambient light measured by the ALS sensor. Light for forward vision.

The communication module 118 generates electromagnetic (EM) signals to communicate with the mobile device 110 (eg, the mobile device of the user of the smart glasses). The mobile device 110 can communicate with the remote server 130 via the network 150 . The remote server 130 can host an application program installed in the mobile device 110 , and the user can use the application program to control, adjust settings, provide, collect and process the data collected by the smart glasses 100 . Accordingly, the communication module 118 may include radio and antenna hardware and software to provide and receive wireless signals 115 from the mobile device 110 and/or the remote server 130 .

2 illustrates several configurations 20A, 20B, and 20C of smart glasses 200 with electronic controls for transparency adjustment (hereinafter collectively referred to as "configurations 20"), according to some embodiments. In the first configuration (20A), the user is outside in bright light (eg, 12:00 PM), and the transparency of the eyepiece in the smart glasses is turned down. In the second configuration ( 20B ), the user is outside at night (eg, 12:00 AM) in a poorly lit environment. Therefore, in configuration B, higher transparency of the smart glasses is desirable. To achieve either of the outcomes in configuration 20A or configuration 20B, the smart glasses may include an ambient light sensor to assess the level of available ambient lighting.

The third configuration (20C) is slightly more complicated. The user is driving on a dark road 220 at night with low ambient lighting. When a car approaches in the opposite direction with the headlights 230 on, the smart glasses can be configured to maintain the level of transparency (or at least not reduce the transparency) so that the user can clearly see the road. For this purpose, in addition to the ambient light sensor, the electronic control of the smart glasses can also apply artificial intelligence algorithms to correctly read the situation and apply appropriate actions.

More generally, embodiments disclosed herein include any of the configurations 20, supplemented with IMU sensor data and touch sensing data to account for the specific environment in which the user is placed, and to better assess usage The user's preferences and needs (eg, user driving at night, walking outside in dark areas such as forests, cloudy or stormy skies, and the like). In addition, a mobile device communicatively coupled to the smart glasses can identify the time of day and the position of the sun relative to the user's head and orientation (which can be captured via the IMU sensor) via GPS and other geolocation strategies. Thus, the machine learning algorithm disclosed herein can use geographic location information and the orientation of the user's head to assess the user profile of smart glasses and better provide adjustments to their transparency levels.

3 illustrates a plurality of scene-aware adaptable gamma curves 310-1, 310-2, and 310-3 for different ambient light configurations, including in smart glasses (see smart glasses 100 and 200), according to some embodiments. (hereinafter collectively referred to as “gamma curve 310 ”) is a graph 300 . Gamma curve 310 provides a relative brightness value 302 (vertical axis in graph 300 ), or power, associated with a given gray value 301 (horizontal axis in graph 300 ). In some embodiments, the grayscale value 301 is in the range of 0 to 256 (8-bit digitization). The gamma curve 310 can be used as a calibration curve for the digitized grayscale values 301 under different ambient light configurations in a pixelated display. For example, gamma curve 310-1 follows standard lighting conditions. Gamma curve 310-2 follows bright scene conditions (see configuration 20A). And the gamma curve 310-3 can follow high contrast conditions (see configuration 20C, or full moon scene at night). The calibration curve 310 may be slightly different for each of the pixels in the display and for each display in the smart glasses. Furthermore, the calibration curve 310 may change over time and use of the smart glasses.

In some embodiments, the gamma curve of the display is dynamically adjusted with ambient brightness to optimize the additive contrast and visibility of a virtual scene or component relative to the real world.

4 illustrates a color gamut and chromaticity point diagram 400 for adjusting red, green, and blue (RGB) displays in smart glasses (see smart glasses 100 or 200) for a given ambient light configuration, according to some specific examples. . Correlated color temperature curve 410 indicates different colors achieved by a perfect black body radiator at different temperatures (eg, 1000K, 2000K, 3000K, 4000K, 6000K, 8000K, and 10,000K). The color gamut 400 indicates chromaticity coordinates 401 (u', the horizontal axis) and 402 (v', the vertical axis) for each of the plurality of colors. For each color point, the (u',v') value can be converted to a specific intensity value for each of the RGB pixels according to a calibration curve (see graph 300). Positions of different wavelengths (eg, spanning the range of 420 nm to 680 nm) are illustrated along edge 420 of gamut 400 .

In some embodiments, white point 450 may be used not only to power the display of the smart glasses, but also to assess the color environment from images captured by a camera in the smart glasses (see camera 123). For example, when the scene color temperature changes (warmer=redder, cooler=bluer), based on the image collected by the camera, the white balance of the display can be changed to better match the scene. In some embodiments, the color temperature curve 410 is indicative of different feasible white points for the display that can be selected to match the surrounding environment based on scene perception. To achieve different white points, the display will adjust the mixing ratio of red, green and blue pixels. The precise amount of change can be indicated by a vector in (u',v') coordinates, which is then converted to RGB pixel values according to the calibration curve.

In some embodiments, and in a similar manner, a display may limit color temperature variation with time of day, eliciting warmer colors at night (eg, by shifting neutral color points in the direction of higher temperatures in a correlated color temperature curve). Thus, some specific examples may include a bedtime setting where the display is turned off or dimmed significantly so as not to stimulate the user to stay awake.

Based on ambient brightness, some instances enable a "dark" mode in which the display has a lower brightness with warmer colors depending on the time of day for enhanced comfort. Additionally, the "dark" mode can be manually enabled by the user one-time, or pre-programmed on a desired schedule, or can be set automatically by the smart glasses.

In some embodiments, color gamut 400 may be used to display on world colors to assist users suffering from color blindness. For example, a certain hue may be more discernible to a user with a given physiological condition, and thus the display may adjust to the mapping in gamut 400 in the direction of that hue. The mapping may be an offset of a point in gamut 400 (real world color) to a new point (user adapted color) in a given direction, and offset by a given distance. In some embodiments, the mapping may include different offsets (in length and direction) of points in gamut 400 . Thus, in some specific examples, the mapping may be a two-dimensional (2D) vector field associated with color gamut 400 . In some embodiments, mapping in color gamut 400 as described herein can be used to adapt colors from real world representations to virtual world representations, attributes derived by user choice or context (e.g., dreamy, evocative, evocative). fantastical, surreal, otherworldly, pompous attributes, and the like) define its characteristics.

5 illustrates a flow diagram of steps in a method 500 for adjusting an RGB display in smart glasses according to an ambient light configuration, as indicated by some embodiments. In some embodiments, at least one or more of the steps in method 500 may be implemented by processing in any of the devices disclosed herein (see FIG. 1 ) for executing instructions stored in memory device to execute. For example, any of the processor and memory may be part of any of smart glasses, mobile device, or remote server. Additionally, the instructions in memory may include any of the following: machine learning algorithms or artificial intelligence algorithms or linear or nonlinear regression algorithms, including neural networks and the like. A smart glasses device may include an ALS sensor and camera as disclosed herein (see FIG. 1 ). Furthermore, in some embodiments, methods consistent with the present invention may include at least one of the steps in method 500 performed alone, or in combination with any other steps to be simultaneous, quasi-simultaneous, or overlapping in time to execute.

Step 502 includes retrieving a new ALS signal from the ALS sensor. In some embodiments, step 502 includes determining average scene brightness and color.

Step 504 includes comparing the new ALS signal with the old ALS signal. In some embodiments, the old ALS signal may be the latest ALS signal in memory.

When the new ALS signal is different from the old ALS signal (see step 504), step 506 includes determining whether the change in the ALS signal is greater than a preselected threshold. Method 500 repeats from step 502 when the change in the ALS signal is not greater than a preselected threshold.

When the change in the ALS signal is greater than a preselected threshold, step 508 includes collecting images of additional scene-aware inputs with the camera.

Step 510 includes adjusting the display profile for brightness, color, temperature, and grayscale (see FIGS. 3-4 ) based on desired values corresponding to the new ALS signal.

Step 512 includes determining whether the display in the smart glasses includes a controllable active dimming feature. When the display does not include a controllable active dimming feature, method 500 repeats from step 502 .

When the display includes a controllable dimming feature according to step 512, step 514 includes adjusting the transparency of an eyepiece including the display according to a desired intensity of an image in the display based on the desired value corresponding to the ALS signal.

Method 500 is repeated for new measurements of the ALS signal. In some embodiments, method 500 is repeated by polling the ALS sensor at a preselected frequency. In some embodiments, method 500 may include continuously receiving an ALS signal from an ALS sensor, and triggering steps 510-514 only when a change above a preselected threshold is observed. In some embodiments, the method 500 may also include updating the pre-selected thresholds according to user configurations and requirements. In yet other embodiments, the method 500 can include receiving an updated value of a preselected threshold from a remote sensor.

6 is a flow diagram illustrating steps in a method 600 for controlling a display in a head-mounted kit based on ambient light measurements, according to some embodiments. In some embodiments, at least one or more of the steps in method 600 may be executed by the processor and stored in smart glasses or body parts of the user (e.g., head, arm, wrist, leg, ankle, finger, toes, knees, shoulders, chest, back, and the like) in the memory of any of the other wearable devices to execute. In some embodiments, at least one or more of the steps in method 600 may be performed by a processor executing instructions stored in memory, wherein either or both of the processor or memory are used to As part of the mobile device, remote server or database, the processor or memory is communicatively coupled to each other via a network. In addition, mobile devices, smart glasses, and wearable devices can be communicatively coupled to each other via wireless communication systems and protocols such as radio, Wi-Fi, Bluetooth, near-field communication (NFC), and the like. In some embodiments, methods consistent with this disclosure may include one or more steps from method 600 performed in any order simultaneously, quasi-simultaneously, or overlapping in time. Accordingly, the head-mounted kit in method 600 may include: a left eyepiece and a right eyepiece mounted on a frame; a display in at least one of the left eyepiece or the right eyepiece, the display including a plurality of arrays of light-emitting pixels ; an ambient light sensor for measuring an amount of ambient light; and a processor configured to control the light intensity of the light-emitting pixels based on the amount of ambient light.

Step 602 includes receiving a signal from an ambient light sensor indicative of an amount of ambient light in an environment of the headset.

Step 604 includes determining characteristics of a virtual image provided to a user based on the amount of ambient light in the environment of the headset.

Step 606 includes controlling light intensity of light-emitting pixels in a display of the headset based on the characteristic of the virtual image. In some embodiments, step 606 includes adjusting the light intensity of a plurality of light-emitting pixels based on the amount of ambient light and a calibration image stored in a memory circuit. In some embodiments, step 606 includes evaluating colorimetric values of images collected from a camera. In some embodiments, step 606 includes adjusting relative intensities of the plurality of red-emitting pixels, the plurality of green-emitting pixels, and the plurality of blue-emitting pixels based on chromaticity values associated with the amount of ambient light. In some embodiments, step 606 includes adjusting the relative brightness of the plurality of red-emitting pixels, the plurality of green-emitting pixels, and the plurality of blue-emitting pixels based on chromaticity values, the amount of ambient light, and color vision deficiencies in user perception. strength. In some embodiments, step 606 includes adjusting a transparency controller based on the amount of ambient light to adjust an amount of transmitted light through an eyepiece in the headset. In some embodiments, step 606 includes controlling the light intensity of the plurality of light-emitting pixels according to a thermal color gamut when the amount of ambient light indicates nighttime use. In some embodiments, step 606 includes controlling the light intensity of a plurality of light-emitting pixels based on the amount of ambient light and the tinting of eyepieces in the headset. hardware review

7 is a block diagram illustrating an exemplary computer system 700 upon which the smart glasses 100 and methods 500 and 600 of FIG. 1 may be implemented, according to some embodiments. In some aspects, computer system 700 may be implemented using hardware or a combination of software and hardware in a dedicated server or integrated into another entity or distributed across multiple entities. The computer system 700 may include a desktop computer, a laptop computer, a tablet computer, a phablet phone, a smart phone, a feature phone, a server computer, or others. The server computer can be remotely located in the data center or stored locally.

Computer system 700 includes a bus 708 or other communication mechanism for communicating information, and a processor 702 (eg, processor 112 ) coupled with bus 708 for processing information. As an example, computer system 700 may be implemented by one or more processors 702 . The processor 702 can be a general-purpose microprocessor, a microcontroller, a digital signal processor (Digital Signal Processor; DSP), a special application integrated circuit (Application Specific Integrated Circuit; ASIC), a field programmable gate array (Field Programmable Gate Array; FPGA), Programmable Logic Device (Programmable Logic Device; PLD), controller, state machine, gating logic, discrete hardware component, or any other suitable entity that can perform calculations or other manipulations of information.

In addition to hardware, computer system 700 may also include code that establishes an execution environment for the computer program in question, for example, making up processor firmware, a protocol stack, a database management system, an operating system, or stored in included memory 704 (e.g., memory 120) code for a combination of one or more of the foregoing, including memory such as random access memory (Random Access Memory; RAM), flash memory, read-only Memory (Read-Only Memory; ROM), Programmable Read-Only Memory (Programmable Read-Only Memory; PROM), Erasable PROM (Erasable PROM; EPROM), scratchpad, hard disk, removable disk A disk, CD-ROM, DVD, or any other suitable storage device coupled to the bus 708 for storing information and instructions to be executed by the processor 702. Processor 702 and memory 704 may be supplemented by or incorporated in special purpose logic circuitry.

Instructions may be stored in memory 704, and one or more computer programs of one or more modules of computer program instructions encoded, for example, on a computer readable medium according to any method known to those of ordinary skill in the art Implemented in the product for the computer system 700 to execute or control the operation of the computer system, such instructions include but are not limited to the following computer languages: such as data-oriented languages (such as SQL, dBase), system languages (such as C, Objective-C , C++, assembly), architectural languages (eg, Java, .NET) and application languages (eg, PHP, Ruby, Perl, Python). Instructions can also be implemented in computer languages such as array languages, feature-oriented languages, assembly languages, production languages, command-line interface languages, compiled languages, parallel languages, curly bracket languages, dataflow languages, data-structured languages, declarative languages, esoteric Languages, extended languages, fourth-generation languages, functional languages, interactive pattern languages, interpreted languages, iterative languages, list-based languages, small languages, logic-based languages, machine languages, numerical languages, metaprogramming Languages, multi-paradigm languages, numerical analysis, non-English languages, object-oriented taxonomic languages, object-oriented prototype-based languages, off-site rule languages, procedural languages, reflective languages, rule-based languages, script-processing languages, stack-based languages, synchronization languages, grammatical processing languages, visual languages, wirth languages and xml-based languages. Memory 704 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by processor 702 .

Computer programs as discussed herein do not necessarily correspond to files in a file system. A program may be stored in a section of a file that holds other programs or data (for example, one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple In a coordinated file (for example, a file that stores one or more modules, subroutines, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network. The programs and logic flows described in this specification can be executed by one or more programmable processors executing one or more computer programs to operate on input data and generate output to perform the function.

Computer system 700 further includes a data storage device 706, such as a magnetic or optical disk, coupled to bus 708 for storing information and instructions. The computer system 700 can be coupled to various devices via the input/output module 710 . The I/O module 710 can be any I/O module. Exemplary input/output modules 710 include data ports such as USB ports. The input/output module 710 is configured to connect to the communication module 712 . Exemplary communication modules 712 include network connection interface cards, such as Ethernet cards and modems. In some aspects, input/output module 710 is configured to connect to a plurality of devices, such as input device 714 and/or output device 716 . Exemplary input devices 714 include a keyboard and pointing devices, such as a mouse or trackball, by which a consumer can provide input to the computer system 700 . Other types of input devices 714 may also be used to provide interaction with consumers, such as tactile input devices, visual input devices, audio input devices, or brain-computer interface devices. For example, the feedback provided to the consumer can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and any form of input can be received from the consumer, including acoustic input, voice input, tactile input or brainwave input. Exemplary output devices 716 include display devices, such as liquid crystal display (LCD) monitors, for displaying information to a consumer.

According to an aspect of the invention, wearable device 100 may be implemented at least in part using computer system 700 in response to processor 702 executing one or more sequences of one or more instructions contained in memory 704 . Such instructions may be read into memory 704 from another machine-readable medium, such as data storage device 706 . Execution of the sequences of instructions contained in main memory 704 causes processor 702 to perform the program steps described herein. One or more processors in a multi-processing configuration may also be employed to execute the sequences of instructions contained in memory 704 . In alternative aspects, hard-wired circuitry may be used in place of or in combination with software instructions to implement various aspects of the invention. Thus, aspects of the invention are not limited to any specific combination of hardware circuitry and software.

Aspects of the subject matter described in this specification can be implemented in a computing system that includes back-end components, such as a data server, or that includes intermediary software components, such as an application server, or that includes front-end components, such as a A client computer through which a consumer may interact with an implementation of the subject matter described in this specification through a graphical consumer interface or web browser, or any combination of one or more such back-end components, middleware components, or front-end components combination. The components of the system can be interconnected by any form or medium of digital data communication (eg, a communication network). A communications network (eg, network 150 ) may include, for example, any one or more of a LAN, WAN, the Internet, and the like. In addition, the communication network may include, but is not limited to, any one or more of the following network topologies, including: bus network, star network, ring network, mesh network, star bus network road, tree or hierarchical network or similar. The communication module can be, for example, a modem or an Ethernet card.

The computer system 700 may include a client and a server. Clients and servers are generally remote from each other and typically interact via a communication network. The relationship of client and server arises by means of computer programs running on the respective computers and having a client-server relationship to each other. Computer system 700 may be, for example but not limited to, a desktop computer, a laptop computer, or a tablet computer. The computer system 700 may also be embedded in another device, such as but not limited to a mobile phone, PDA, mobile audio player, Global Positioning System (GPS) receiver, video game console, and/or television set-top box .

The term "machine-readable storage medium" or "computer-readable medium" as used herein refers to any medium or media that participate in providing instructions to processor 702 for execution. Such media may take many forms including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks, such as data storage device 706 . Volatile media includes dynamic memory, such as memory 704 . Transmission media includes coaxial cables, copper wire and fiber optics, including the wires forming bus 708 . Common forms of computer readable media include, for example, floppy disks, floppy disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, DVDs, any other optical media, punched cards, paper tape, any Other physical media, RAM, PROM, EPROM, FLASH EPROM, any other memory chips or cartridges, or any other media that can be read by a computer. The machine-readable storage medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter affecting a machine-readable propagating signal, or a combination of one or more of them.

To illustrate the interchangeability of hardware and software, items such as various illustrative blocks, modules, components, methods, operations, instructions and algorithms have been described generally in terms of their functionality. Implementing such functionality as hardware, software, or a combination of hardware and software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application.

As used herein, the phrase "at least one of" preceding a list of items by separating any of those items with the terms "and" or "or" modifies the list as a whole, not just one of the items in the list. Individual members (for example, individual projects). The phrase "at least one of" does not require selection of at least one of the items; rather, the phrase allows the inclusion of at least one of any of those items and/or at least one of any combination of those items and/or the meaning of at least one of each of these items. As examples, the phrases "at least one of A, B, and C" or "at least one of A, B, or C" each refer to only A, only B, or only C; any combination of A, B, and C and/or at least one of each of A, B and C.

The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any particular example described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other particular examples. Such as an aspect, the aspect, another aspect, some aspects, one or more aspects, an implementation, the implementation, another implementation, some implementations, one or more implementations, an embodiment, the embodiment example, another embodiment, some embodiments, one or more embodiments, a configuration, the configuration, another configuration, some configurations, one or more configurations, the present technology, this disclosure, Phrases of the present invention, its other variations, and the like are used for convenience and do not imply that disclosures related to such phrases are essential to the technology of the invention or that such disclosures are applicable to this invention All configurations of the inventive technology. A disclosure associated with such a phrase may apply to all configurations or one or more configurations. Disclosures related to such phrases may provide one or more examples. A phrase such as an aspect or aspects may refer to one or more aspects and vice versa, and this applies analogously to the other aforementioned phrases.

Reference to an element in the singular is not intended to mean "one and only one," but rather "one or more," unless specifically stated otherwise. Masculine pronouns (eg, his) include the feminine and neuter genders (eg, her and its), and vice versa. The term "some" means one or more. Underlined and/or italicized headings and subheadings are for convenience only, do not limit the present technology, and are not referenced in conjunction with the interpretation of the description of the present technology. Relational terms, such as first and second and the like, may be used to distinguish one entity or action from another without necessarily requiring or implying any actual such relationship or order between such entities or actions . All structural and functional equivalents to the elements of the various configurations described throughout this disclosure that are known or later come to be known by those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be incorporated by reference herein. Subject technology covered. Furthermore, nothing disclosed herein is intended to be dedicated to the public, whether or not such disclosure is explicitly recited in the above description. Claimed elements should not be construed under the provisions of 35 U.S.C. §112, sixth paragraph, unless the element is specifically described using the phrase "means for" or, in the case of a method claim, the element is Use the phrase "steps for" to describe.

While this specification contains many specificities, these should not be construed as limitations on the scope of what may be described, but rather as descriptions of particular implementations of the subject matter. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features above may be described as functioning in certain combinations and even initially described as such, one or more features from a described combination may in some cases be deleted from that combination and the described Combinations may be for subcombinations or variations of subcombinations.

The subject matter of this specification has been described in relation to certain aspects, but other aspects can be implemented and are within the scope of the following claims. For example, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. The actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the procedures depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the aspects described above should not be understood as requiring such separation in all aspects, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

The title, prior art, brief description of the figures, abstract and figures are incorporated herein into this disclosure and are provided as illustrative examples rather than limiting descriptions of the disclosure. It should be understood that it will not be used to limit the scope or meaning of the patent claims. Additionally, it can be seen in implementations that this specification provides illustrative examples and groups various features together in various implementations for the purpose of streamlining the disclosure. This method of disclosure, however, should not be interpreted as reflecting an intention that the described subject matter requires more features than are expressly stated in each claim. Rather, inventive subject matter lies in less than all features of a single disclosed configuration or operation, as reflected in the claims. The claims of claim are hereby incorporated into the Practice, with each claim in its own right being a separately described subject matter.

Claims are not intended to be limited to the aspects described herein, but are to be accorded the full scope consistent with the language claim and cover all legal equivalents. Nonetheless, nothing in the Claims Claims is intended to cover, nor should such subject matter be so construed, that fails to satisfy the requirements of applicable patent law.

20A: Configuration 20B: Configuration 20C: Configuration 100: Smart Glasses 105L: left eyepiece 105R: Right eyepiece 110:Mobile device 111: frame 112: Processor 115: wireless signal 118: Communication module 120: memory 121:Speaker/Microphone 123: camera 125: Ambient light sensor 130: remote server 150: Network 200: Smart Glasses 220: Dark Road 230: headlight 300:Charts 301: Given gray value 302: relative brightness value 310-1: Gamma Curve 310-2: Gamma Curve 310-3: Gamma Curve 400: Color gamut and chromaticity point diagram/color gamut 401: Chromaticity coordinates 402: Chromaticity coordinates 410: Correlated color temperature curve 420: edge 450: white dot 500: method 502: Step 504: step 506: Step 508: Step 510: step 512: Step 514: step 600: method 602: Step 604: Step 606: Step 700: Computer system 702: Processor 704: Memory 706: data storage device 708: Bus 710: Input/Output Module 712:Communication module 714: input device 716: output device

[ FIG. 1 ] Illustrates smart glasses including ambient light sensors for augmented reality applications, according to some embodiments.

[ FIG. 2 ] Illustrates several configurations of smart glasses with electronic controls for transparency adjustment, according to some embodiments.

[ FIG. 3 ] Illustrates scene-aware adaptability gamma curves for different ambient light configurations in smart glasses according to some embodiments.

[FIG. 4] Illustrates color gamut and chromaticity point diagrams for tuning a Red, Green, and Blue (RGB) display in smart glasses for a given ambient light configuration, according to some specific examples .

[ FIG. 5 ] A flowchart illustrating steps in a method for adjusting an RGB display in smart glasses according to an ambient light configuration according to some embodiments.

[ FIG. 6 ] is a flowchart illustrating steps in a method for controlling a display in a head-mounted kit based on ambient light measurements, according to some embodiments.

[ FIG. 7 ] is a block diagram illustrating an exemplary computer system by which the headset as in FIG. 1 and the methods in FIGS. 5 and 6 may be implemented according to some embodiments.

In the figures, elements with the same or similar reference numbers have the same or similar attributes and characteristics, unless explicitly stated otherwise.

100: Smart Glasses

105L: left eyepiece

105R: Right eyepiece

110:Mobile device

111: frame

112: Processor

115: wireless signal

118: Communication module

120: memory

121:Speaker/Microphone

123: camera

125: Ambient light sensor

130: remote server

150: Network

Claims (20)

A device comprising: left eyepiece and right eyepiece mounted on the frame; a display in at least one of the left eyepiece or the right eyepiece, the display comprising an array of light emitting pixels; an ambient light sensor for measuring the amount of ambient light; and A processor configured to control light intensity of the plurality of light-emitting pixels based on the amount of ambient light. The device according to claim 1, wherein the ambient light sensor comprises one or more photodiodes. The device according to claim 1, further comprising a memory storing a gamma curve for calibrating the light intensity of the plurality of light-emitting pixels to provide required brightness for the amount of ambient light. The device of claim 1, further comprising a memory storing a calibration image, wherein the processor is configured to adjust the light intensity of the plurality of light-emitting pixels based on the amount of ambient light and the calibration image. The device of claim 1, wherein the ambient light sensor includes a camera configured to collect images of a front view, and the processor is configured to evaluate colorimetric values from images collected with the camera, and based on the The amount of ambient light and the chromaticity value are used to control the light intensity of the plurality of light-emitting pixels. The device of claim 1, wherein the plurality of light-emitting pixels includes a plurality of red light-emitting pixels, a plurality of green light-emitting pixels, and a plurality of blue light-emitting pixels, wherein the processor is configured to be based on an amount associated with the ambient light The relative intensity of the plurality of red light-emitting pixels, the plurality of green light-emitting pixels and the plurality of blue light-emitting pixels is adjusted. The device of claim 1, wherein the plurality of light-emitting pixels includes a plurality of red light-emitting pixels, a plurality of green light-emitting pixels, and a plurality of blue light-emitting pixels, wherein the processor is configured to based on chromaticity values, the ambient light The relative intensities of the plurality of red light-emitting pixels, the plurality of green light-emitting pixels and the plurality of blue light-emitting pixels are adjusted in accordance with color vision defects in quantity and user's perception ability. The device of claim 1, wherein the left eyepiece and the right eyepiece further comprise a transparency controller for adjusting the amount of transmitted light passing through the left eyepiece and the right eyepiece, wherein the processor is configured based on the environment The amount of light to adjust with this transparency controller. The device according to claim 1, wherein when the amount of ambient light indicates use at night, the processor further controls the light intensity of the plurality of light-emitting pixels according to a thermal color gamut. The device of claim 1, wherein at least one of the left eyepiece and the right eyepiece is tinted, and the processor is configured based on the amount of ambient light and the tinting of the left eyepiece or the right eyepiece The light intensity of the plurality of light-emitting pixels is controlled. A computer-implemented method comprising: receiving a signal indicative of an amount of ambient light in the environment of the headset from the ambient light sensor; determining characteristics of a virtual image provided to a user based on the amount of ambient light in the environment of the headset; and Light intensity of light-emitting pixels in a display of the headset is controlled based on the characteristic of the virtual image. The computer-implemented method of claim 11, wherein controlling the light intensity of the plurality of light emitting pixels in the display comprises adjusting the light intensity of the plurality of light emitting pixels based on the amount of ambient light and a calibration image stored in a memory circuit brightness. The computer-implemented method of claim 11, wherein controlling the light intensity of the plurality of light-emitting pixels in the display comprises evaluating chromaticity values of images collected from a camera. The computer-implemented method of claim 11, wherein controlling the light intensity of the plurality of light emitting pixels in the display comprises adjusting a plurality of red light emitting pixels, a plurality of green light emitting pixels based on a chromaticity value associated with the amount of ambient light pixel and the relative intensity of a plurality of blue-emitting pixels. The computer-implemented method of claim 11, wherein controlling the light intensity of the plurality of light-emitting pixels in the display includes adjusting a plurality of reds based on a chromaticity value, an amount of the ambient light, and color vision deficiencies in user perception Relative intensities of the luminescent pixel, the plurality of green luminescent pixels, and the plurality of blue luminescent pixels. The computer-implemented method of claim 11, wherein controlling the light intensity of the plurality of light-emitting pixels in the display comprises adjusting a transparency controller based on the amount of ambient light to adjust light passing through an eyepiece in the head-mounted kit The amount of transmitted light. The computer-implemented method of claim 11, wherein controlling the light intensity of the plurality of light-emitting pixels in the display comprises controlling the light-emitting pixels of the plurality of light-emitting pixels according to a thermal color gamut when the amount of ambient light indicates nighttime use brightness. The computer-implemented method of claim 11, wherein controlling the light intensity of the plurality of light-emitting pixels in the display comprises adjusting the plurality of light-emitting pixels based on an amount of the ambient light in the headset and a tinting of an eyepiece The light intensity of the pixel. The computer-implemented method of claim 11, further comprising selecting a white point of the display based on a correlated color temperature to match an ambient environment based on scene awareness and the amount of ambient light in the environment of the headset. The computer-implemented method of claim 11, further comprising adjusting a white point of the display based on the color temperature and temperature values constrained by time of day and the amount of ambient light in the environment of the headset.