CN117765275A - On-head detection based on analog proximity sensor using IR camera - Google Patents

Abstract

The present disclosure relates to a system for detecting a proximity object, the system comprising one or more cameras and one or more luminaires. The proximity object is detected by: obtaining image data captured by the one or more cameras, wherein the image data is captured while at least one of the one or more illuminators is illuminated; determining a luminance statistic from the image data; and determining whether the luminance statistic meets a predetermined threshold. In accordance with a determination that the luminance statistic meets the predetermined threshold, detection of the approaching object is determined.

Description

On-head detection based on analog proximity sensor using IR camera

Technical Field

The present disclosure relates generally to image processing. More particularly, but not by way of limitation, the present disclosure relates to techniques and systems for monitoring a physical environment using an Infrared (IR) camera to detect proximate objects.

Background

In modern times, electronic devices may allow users to interact with the user's environment in a new manner. As an example, in an augmented reality or mixed reality environment, a user may interact with an environment by viewing a representation of the environment with both physical objects and virtual objects. Devices that provide such augmented reality capabilities are typically equipped with cameras and other sensors that allow the device to provide such representations to a user. However, these devices often lack a proximity detector. There is a need for a technique for detecting proximity without the use of a proximity sensor.

Drawings

FIG. 1 illustrates an exemplary diagram of an electronic device modifying operation in response to detecting that a user is proximate to the device in accordance with one or more embodiments.

FIG. 2 illustrates a flow diagram of a technique for determining whether a proximity object is detected in accordance with one or more embodiments.

FIG. 3 illustrates, in flow diagram form, an exemplary process for performing image data analysis to determine luminance statistics in accordance with one or more embodiments.

Fig. 4 illustrates an example luminance statistic for a Head Mounted Device (HMD) in accordance with one or more embodiments.

FIG. 5 illustrates a flow diagram of a technique for detecting a proximate object using luminance statistics in accordance with one or more embodiments.

FIG. 6 illustrates a flow diagram of a technique for time detection of a proximity object in accordance with one or more embodiments.

FIG. 7 illustrates, in block diagram form, an exemplary device diagram in accordance with one or more embodiments.

Fig. 8 illustrates, in block diagram form, a mobile device in accordance with one or more embodiments.

Detailed Description

In general, embodiments described herein relate to a technique for adjusting operation of a device in response to detecting a proximity object. In some implementations, luminance statistics of image data captured by an Infrared (IR) camera may be analyzed to determine whether a proximity threshold is met and, thus, whether a proximity object is detected.

In some implementations, the techniques include obtaining image data captured by one or more cameras. The camera may be an IR camera, or other camera that captures image data from which brightness may be analyzed. The image data may be captured in a single frame, or in a series of frames (such as a video sequence). The luminance from the image may be analyzed to determine luminance statistics. The luminance statistic may indicate, for example, peak luminance, average luminance, etc. In some implementations, the amount of the image that is a particular luminance can be determined. That is, the brightness of each pixel may be determined and compared to determine the amount of image within a particular brightness range. In some implementations, if it is determined that the predetermined amount of image data is within the luminance range, it may be determined that the proximity object is detected. If the overall brightness of the image data is determined to be within a predetermined range, the device may take the generated action. For example, in some embodiments, when the device detects a proximity object, one or more systems of the device (such as a display or other system) may be powered on. This may be preferable, for example, where the system is a wearable device and detecting a proximity object indicates that the user is wearing the wearable device.

Then, in response to determining that the proximity object is no longer detected, operation of the system may be suppressed or the system may be powered down. For example, it may be determined that the proximity object is no longer present based on the luminance statistics of the additional frames. Thus, one or more systems (such as the display of the device, etc.) may be powered down. Returning to the example of a wearable device, this may be preferable so that when a proximity object is no longer detected, it may be inferred that the user is no longer wearing the device, and thus certain systems of the device (such as the display) may be powered down.

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed concepts. As part of this description, some of the drawings of the present disclosure represent structures and devices in block diagram form in order to avoid obscuring novel aspects of the disclosed concepts. In the interest of clarity, not all features of an actual implementation may be described. Additionally, as part of this specification, some of the figures of the present disclosure may be provided in the form of a flow chart. Blocks in any particular flowchart may be presented in a particular order. However, it should be understood that the particular order of any given flowchart is merely illustrative of one embodiment. In other embodiments, any of the various elements depicted in the flowcharts may be deleted, or the illustrated sequence of operations may be performed in a different order, or even concurrently. Further, other embodiments may include additional steps not shown as part of the flowchart. Furthermore, the language used in the present disclosure has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter, resort to the claims being necessary to determine such inventive subject matter. Reference in the present disclosure to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosed subject matter, and multiple references to "one embodiment" or "an embodiment" should not be understood to all refer to the same embodiment.

It will be appreciated that in the development of any such actual implementation (as in any software and/or hardware development project), numerous decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another. It will be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the art having the benefit of this disclosure, of designing and implementing a multi-modal processing system.

Various examples of electronic systems related to various technologies and technologies for using such systems are described.

A physical environment, as used herein, refers to a physical world that people can sense and/or interact with without the assistance of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and/or interact with a physical environment, such as by visual, tactile, auditory, gustatory, and olfactory. Conversely, an augmented reality (XR) environment refers to a fully or partially simulated environment in which people sense and/or interact via electronic devices. For example, the XR environment may include Augmented Reality (AR) content, mixed Reality (MR) content, virtual Reality (VR) content, and the like. In the case of an XR system, a subset of the physical movements of a person, or a representation thereof, are tracked and in response one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with at least one physical law. As one example, the XR system may detect head movements and, in response, adjust the graphical content and sound field presented to the person in a manner similar to the manner in which such views and sounds change in the physical environment. As another example, the XR system may detect movement of an electronic device (e.g., mobile phone, tablet, laptop, etc.) presenting the XR environment, and in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds would change in the physical environment. In some cases (e.g., for reachability reasons), the XR system may adjust characteristics of graphical content in the XR environment in response to representations of physical movements (e.g., voice commands).

There are many different types of electronic systems that enable a person to sense and/or interact with various XR environments. Examples include: head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields integrated with display capabilities, windows integrated with display capabilities, displays formed as lenses designed for placement on a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablet computers, and desktop/laptop computers. The head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, the head-mounted system may be configured to accept an external opaque display (e.g., a smart phone). The head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment, and/or one or more microphones for capturing audio of the physical environment. The head-mounted system may have a transparent or translucent display instead of an opaque display. The transparent or translucent display may have a medium through which light representing an image is directed to the eyes of a person. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light sources, or any combination of these techniques. The medium may be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, the transparent or translucent display may be configured to selectively become opaque. Projection-based systems may employ retinal projection techniques that project a graphical image onto a person's retina. The projection system may also be configured to project the virtual object into the physical environment, for example as a hologram or on a physical surface.

Turning to FIG. 1, an exemplary environment diagram in accordance with one or more embodiments is presented. It should be understood that the various components and descriptions are provided for illustrative purposes and are not necessarily intended to limit the present disclosure.

As shown in fig. 100A, a user 102A is processing an electronic device 106A. In particular, the electronic device 106 is depicted as a head-mounted device. However, any kind of mobile electronic device may be used. The electronic device may include, for example, a display 120A that is powered down as presented. Thus, display 120A is depicted as black. Additionally, the electronic device 106A may include one or more sensors that may capture image data and other sensor data related to the environment surrounding the electronic device 106. For example, the device 106 may include a back sensor 122 that captures data of the environment facing away from the user. The rearward facing sensor may include, for example, one or more cameras (i.e., a camera system that captures image data of the environment), a depth sensor, an ambient light sensor, and the like.

The electronic device may include one or more forward sensors 124. The forward facing sensor may include, for example, a user facing camera, a depth sensor, a light sensor, and the like. Thus, the user-oriented sensor 124 may capture image data of the user while the user is utilizing the electronic device 106. Further, the user-oriented sensor 124 may include additional data to facilitate user interaction with the environment via the electronic device 106. For example, the user-oriented sensors may include gaze tracking sensors, and the like. The sensor may be used in association with other components to capture data related to the user's eyes from which gaze information, such as gaze vectors, may be determined. Additionally, the gaze tracking sensor may capture information about the user through the eyes, such as the position of the person, the distance of the eyes from the device, the pose relative to the device, and so forth.

In some implementations, the user-facing cameras may include, for example, one or more RGB cameras, depth cameras, IR cameras, and the like. In some implementations, the IR camera may be used in conjunction with other components of the electronic device 106 to obtain useful sensor data related to the user 102A. For example, the electronic device 106 may be equipped with one or more Light Emitting Diodes (LEDs) or other lighting components that, when illuminated, allow a user-facing camera to capture better lighted images of the user or other objects near the device.

Turning to fig. 100B, the user 102B may begin interacting with the electronic device 106 by lifting the electronic device 106 to the user's face. As another example, the electronic device 106 may be a wearable device and the user 102B may mount the device 106B to the user. For purposes of the example in fig. 100B, user 102B has lifted device 106B so that the device is close to the user. In some embodiments, the device may perform an analysis of the image data captured by the forward sensor 124 to confirm or determine such proximity. In some embodiments, when one of the illumination components is activated, image data and analysis may be captured by a camera, such as an IR camera. For example, in one embodiment, a forward IR camera may capture data when a user-facing LED is illuminated. Image data analyzed for brightness statistics may be collected from such illuminated image data. In response to detecting the proximity, the electronic device 106B may power up one or more device systems (such as the display 120B). Thus, as depicted in fig. 100B, display 120B is depicted as on, with content presented to the user.

Turning now to fig. 2, a flow diagram of a technique for determining whether a proximity object is detected in accordance with one or more embodiments is shown. For purposes of explanation, the following steps will be described in the context of fig. 1. However, it should be understood that various actions may be performed by alternative components. Further, the various actions may be performed in a different order. In addition, some acts may be performed concurrently, and some acts may not be required, or other acts may be added, according to various embodiments.

Flowchart 200 begins at block 205, where image data is obtained from a user-oriented camera. In some implementations, the device may simulate a proximity sensor in various environments, and thus the camera need not be user-oriented. For example, the techniques described herein may be used to simulate a proximity sensor based on image data captured from an alternative camera on a device. Optionally, as shown at block 210, image frames captured by the camera may be selected for analysis, as the frames are illuminated by illumination components on the device. For example, the LED may be attached to the device in the same direction as the camera captures the image data. In some implementations, the frame captured by the camera may be selected based on determining that the frame was captured when the LED or other lighting component was illuminated. Thus, an image for analysis can be selected from the lit images.

Flowchart 200 continues at block 215 where a luminance statistic for the image data is determined. These luminance statistics will be described in more detail below with reference to fig. 3. Generally, however, the luminance statistic includes a luminance or intensity value distribution of the image. In some implementations, the luminance or intensity values may be determined for pixels in an image frame or a sample set of pixels in image data. In accordance with some implementations, in the absence of a proximate object, the pixel will be associated with a low intensity value. However, since the illumination component may be illuminated when capturing an image, the image may include pixels of saturated intensity, which may be caused by the illuminated illumination component in the camera capturing the image. Thus, additionally, determining the luminance statistic may include detecting pixels having saturated intensities, which should be excluded when determining whether a proximity object is present.

The flow chart continues to block 220 where it is determined whether the luminance statistic meets a threshold. According to some implementations, the threshold may be defined based on a portion of the image that the pixel is associated with a predefined range of luminance values. In other words, as described above, pixels having saturated intensities should be suppressed to exclude image noise. Furthermore, low brightness pixels may occur when light is not reflected on any object in the vicinity of the camera, and thus these pixels may be associated with very low brightness intensity values. Thus, a sufficient portion of the pixels of a frame having a luminance or intensity value in the mid-range may indicate that a proximity object is detected.

If it is determined at block 220 that the luminance statistic meets the threshold, the flow chart proceeds to block 225. At block 225, it is determined that a proximity object is detected. Optionally, as indicated at block 230, the fact that a proximity object is detected may cause the device to determine that the device is being used. Certain embodiments described herein may be particularly useful for wearable devices. Thus, the determination may indicate that the wearable device is currently being worn based on detecting the proximity object. Then, at block 235, the operation of the device may be modified based on the determination that the device is being used. In some embodiments, detecting the proximity object may cause a change in operation of the device. For example, a system of electronic devices (such as a display device or other system of the device) may be powered on.

If it is determined at block 220 that the luminance statistic fails to meet the threshold, the flow chart proceeds to block 240. At block 240, it is determined that no proximity object is detected. Optionally, as indicated at block 240, the fact that the proximity object is not detected may cause the device to determine that the device is not in use. Because some embodiments described herein may be particularly useful for wearable devices, the determination may be based on not detecting a proximity object to indicate that the wearable device is not currently being worn. Then, at block 250, the operation of the device may be modified based on determining that the device is being used. In some embodiments, detecting the proximity object may cause a change in operation of the device. For example, a system of an electronic device (such as a display device or other system of the device) may be powered down.

FIG. 3 illustrates, in flow diagram form, an exemplary process for performing image data analysis to determine luminance statistics in accordance with one or more embodiments. For purposes of explanation, the following steps will be described in the context of fig. 1. However, it should be understood that various actions may be performed by alternative components. Further, the various actions may be performed in a different order. In addition, some acts may be performed concurrently, and some acts may not be required, or other acts may be added, according to various embodiments.

Flowchart 300 begins at block 305, where image data is obtained from a user-oriented camera. In some implementations, the device may simulate a proximity sensor in various environments, and thus the camera need not be user-oriented. For example, the techniques described herein may be used to simulate a proximity sensor based on image data captured from an alternative camera on a device. The cameras may include, for example, RGB cameras, depth cameras, IR cameras, and the like.

At block 310, a total number of pixels is determined from the image data. The pixels may comprise all pixels of a particular frame or set of frames, or may comprise samples of pixels of one or more frames. In some embodiments, one or more frames considered for determining the luminance statistic may be selected from image frames captured by the camera and limited to image frames captured when the lighting component actively lights up the environment captured by the camera. For example, image frames may be selected from a series of frames captured by an IR camera, as they are captured during the limits of one or more LEDs on a device facing the environment captured by the camera.

The flowchart 300 continues at block 315 in which a luminance histogram is generated based on the luminance of each of the total pixels determined at block 310. The luminance histogram may indicate a count of pixels identified at each luminance value. Thus, for a given frame or set of frames, a histogram may indicate how many pixels are associated with a particular luminance or intensity value. In some implementations, pixels within a predetermined brightness or intensity range are determined. Thus, a histogram may be clipped between two luminance values. At block 320, pixels above a predetermined brightness threshold are excluded. For example, pixels associated with luminance values above a threshold may indicate that the pixels are associated with stray light from the lighting device. Therefore, omitting pixels associated with very high luminance values reduces image noise. Further, as shown at block 325, pixels below the predetermined brightness threshold are also excluded when clipping the histogram. According to some implementations, pixels below a predetermined brightness threshold may indicate that light does not reach any approaching object providing a sufficient intensity value.

The flowchart 300 continues at block 330 where the number of remaining pixels is determined. For example, by determining how many pixels are considered to be within a predefined range (i.e., pixels that are not excluded at block 320 or block 325), the number of remaining pixels may be determined based on the clipped histogram. According to some embodiments, the threshold may be based on a minimum percentage of pixels for which smaller luminance values are detected. Thus, in some implementations, a minimum percentage of pixels of the luminance value associated with the intermediate luminance or intensity is indicative of a portion of the one or more images associated with the intermediate luminance or intensity value. At block 335, a determination is made as to whether a sufficient number of remaining pixels meet the threshold. If a sufficient number of remaining pixels are determined to meet the threshold, the flowchart ends at block 340 and the system determines that a proximity object is detected. In contrast, returning to block 335, if it is determined that the luminance statistic does not meet the threshold, the flowchart ends at block 345 and the system determines that a proximity object is not detected.

In some embodiments, additional considerations may be made to address unique lighting conditions that may lead to additional noise in the data. For example, in the presence of externally strong stray light, frames may be captured in an interlaced fashion, where an illumination component, such as an LED, is illuminated from one image without letting the illumination component enter the next image. The unlit image may be subtracted from the lit image before the number of pixels is determined at block 310. Thus, the technique described in fig. 3 may be applied to differential images.

Fig. 4 illustrates an example luminance statistic for a Head Mounted Device (HMD) in accordance with one or more embodiments. Thus, fig. 4 depicts an exemplary dataset for the techniques described above with respect to fig. 3.

Fig. 4 shows four image frames captured by a camera on the device. Image frame 402A depicts an exemplary image frame captured by a head-mounted device, where the device is being worn by a user and one or more associated LEDs are off and thus do not illuminate the environment. In contrast, image frame 402B depicts a frame in which the device is being worn by a user and one or more associated LEDs are on and thus illuminate the environment captured in the frame. Thus, image frame 402A depicts a uniformly dark frame, while image frame 402B depicts a frame with intermediate luminance values visible in the frame.

Similarly, differences in the views of the frames are also apparent when the device is not being worn by the user. Thus, image frame 404A depicts an image frame captured by the camera when the wearable device is not being worn and the LED is off and thus does not illuminate the environment. In contrast, frame 404B depicts an image frame captured when the device is not worn by the user but the LEDs are therefore illuminated, the stray illumination may be depicted in the form of right-hand pixels in the frame, while the rest of the frame remains substantially dark.

As described above, when selecting a frame for analysis, the frame under consideration may be limited to only the frame of the lighting component lighting environment. Thus, histograms 406 and 408 for frames 402B and 404B are provided, wherein the scene is illuminated by the LEDs. In particular, histogram 406 relates to a cumulative histogram over a video sequence of image frames (such as frame 402B) in which a near object is detected. In contrast, histogram 408 depicts a cumulative histogram over the video sequence when the head is removed. Further, image frames 402A and 404A are not considered. Histogram 406 corresponds to image frame 402B. In particular, histogram 406 depicts a pixel histogram based on luminance urgency values for each of the pixels (or sampled pixels from the frame). From this histogram, it is known that the image frame to be formed includes a considerable number of pixels having intermediate luminance values. This is clear when comparing histogram 406 with histogram 408, which is associated with a cumulative frame (such as image frame 404B) on the video sequence where no approaching object is detected. As shown in histogram 408, most pixels are depicted as having very low luminance values. Very bright pixels are an exception, which are associated with layer illumination from LEDs.

This distinction is further made clear when considering the chart at 410. This graph shows the first set of graph accumulated luminance in each of the frames. Line 412 shows a cumulative histogram of frames where no approaching object is detected, providing a very small luminance or intensity range. In contrast, line 414 shows a cumulative histogram of frames where approaching objects are detected, and there is a greater range of brightness in the pixels.

According to one or more embodiments, additional processing may be performed on the image data to detect a proximity object using the brightness. FIG. 5 illustrates a flow diagram of a technique for detecting a proximate object using luminance statistics in accordance with one or more embodiments. For purposes of explanation, the following steps will be described in the context of fig. 1. However, it should be understood that various actions may be performed by alternative components. Further, the various actions may be performed in a different order. In addition, some acts may be performed concurrently, and some acts may not be required, or other acts may be added, according to various embodiments.

Flowchart 500 begins at block 505, where a lit image and an unlit image are obtained from one or more user-facing cameras. In some embodiments, the illuminated and non-illuminated images are captured from a camera positioned in front of each eye. The illuminated images may be captured simultaneously, and the non-illuminated images may be captured simultaneously. In some embodiments, the head-mounted device may include an illuminator, such as an LED or the like, facing one or both eyes. The head-mounted device may additionally have one or more cameras facing the eyes. In some embodiments, the camera captures one or more illuminated images when the illuminator is active and captures one or more unlit images when the illuminator is inactive. To determine if a proximity object is detected, a first pair of frames is captured that includes capturing frames of each eye while the illuminator is active. A second pair of frames is also captured that includes capturing frames for each eye when the illuminator is inactive. Thus, in some embodiments, the illuminated and non-illuminated image data may include an illuminated frame for each eye and a non-illuminated frame for each eye.

For purposes of the example shown in fig. 5, a first example of a frame pair including an unlit frame 550 and an lit frame 552 is shown. These are examples of two frames captured when the user wears the HMD. The unlit frame 550 displays frames captured without the illuminator on the HMD being activated, while the lit frame 552 displays frames captured while the illuminator is active. For example, the two frames may be captured simultaneously within a predetermined period of time. Separately, a second exemplary frame pair is displayed, including an unlit frame 554 and an lit frame 556 captured when the user is not wearing the HMD. The unlit frame 554 is captured when the illuminator is inactive, and the lit frame 556 is captured when the illuminator is active. For example, the two frames may be captured simultaneously within a predetermined period of time.

Flowchart 500 continues to block 510 where ambient light is removed from the illuminated image based on the non-illuminated image. As described above, a lit image and an unlit image may be obtained for each eye. Thus, in some embodiments, ambient light is removed from the illuminated image of each eye. An unlit image may capture illumination in the environment that is not caused by the illuminator, thereby providing ambient light. Ambient light may exist, for example, based on windows, lights, overhead lights, etc. in the environment. In the first example shown, frame 558 shows a modified illuminated image 558 in which ambient light is removed. That is, light that appears in the unlit image 550 is subtracted from the lit frame 552. Similarly, in a second example, ambient light present in frame 554 is subtracted from illuminated image 556 to obtain modified illuminated image 560.

At block 515, the saturated region is occluded from the modified illuminated image. The saturation region occurs, for example, as reflected light emitted from a luminaire on the device. In some implementations, by occluding the saturation region, reflections from the illuminator are less likely to affect the overall luminance statistic of the frame. In some embodiments, the occluded region may include a detected portion of the image data having a brightness above a predetermined threshold. A mask may be applied to one or more portions of the region that meet a predetermined threshold. Thus, in the first example, 562 presents an occluded image in which the saturated region of the modified illuminated image 558 is occluded. Similarly, in a second example, 564 presents an occluded image in which the saturated region of the modified illuminated image 560 is occluded. In some embodiments, edges may be applied such that the occluded region includes a saturated portion and a predetermined amount of additional image data surrounding the saturated portion.

The flow chart proceeds to block 520 where pixels within the region of interest are identified. The region of interest may comprise a predefined portion of a frame. In some embodiments, the region of interest may differ between the left eye image and the right eye image. The region of interest may be any shape, such as elliptical, rectangular, etc. In some embodiments, the region of interest may include a predefined portion of the frame, regardless of the content presented in the frame. In a first example, a region of interest of the occluded image 562 is identified at 566. Similarly, in a second example, a region of interest of the occluded image 564 is identified at 568.

At block 525, a luminance histogram is generated based on the luminance of each pixel in the region of interest. The luminance histogram may indicate a count of pixels identified at each luminance value or set of luminance values within the region of interest. Thus, for a given frame or set of frames, a histogram may indicate how many pixels are associated with a particular luminance or intensity value. Thus, in the first example, the histogram 570 presents a histogram indicating mid-range luminance values in the region of interest 566. In contrast, in the second example, histogram 572 presents a histogram indicating lower luminance values consistent with darker regions of interest 568.

The flow chart proceeds to block 530 where the luminance data is filtered. In some implementations, pixels within a predetermined brightness or intensity range are determined. Thus, a histogram may be clipped between two luminance values. At block 320, pixels above a predetermined brightness threshold are excluded. Doing so may reduce noise in the luminance data.

At block 535, it is determined whether the luminance statistic meets a predetermined threshold. In some implementations, multiple thresholds may be used, and the particular threshold used may be based on the current state of the device. For example, if a proximity object is detected such that the device is considered to be on the user's head, a higher brightness value may be used than if the device is currently determined not to be on the user's head. The luminance statistics may include determining whether a particular luminance, such as an average luminance or a median luminance, meets one or more predetermined thresholds.

If it is determined that the luminance statistic meets the threshold, the flowchart ends at block 540 and it is determined that a proximity object is detected. In contrast, if at block 535, if it is determined that the luminance statistic does not meet the threshold, the flowchart ends at block 545 and it is determined that a proximity object is not detected. As described above, in some embodiments, the luminance statistic that does not satisfy the threshold may be a lower threshold than if the luminance statistic were determined to satisfy the threshold. In this way, a determination may be made based on whether the luminance statistic exceeds or falls below a predetermined range.

In some embodiments, determining whether a proximity object is detected may be part of a time determination as to whether the device is on-head or off-head. For example, to determine whether the device is on-head or off-head, a series of detections of approaching objects may be based on. Further, some embodiments include considering one or both eyes when detecting a proximity object and/or determining whether the device is overhead. FIG. 6 illustrates a flow diagram of a technique for time detection of a proximity object in accordance with one or more embodiments. In particular, fig. 6 illustrates an exemplary flow for determining whether a device is in an active mode or an inactive mode, and may occur in connection with detection of a proximity object as described above with respect to fig. 5. For purposes of explanation, the following steps will be described in the context of fig. 5. However, it should be understood that various actions may be performed by alternative components. Further, the various actions may be performed in a different order. In addition, some acts may be performed concurrently, and some acts may not be required, or other acts may be added, according to various embodiments.

The flow chart begins at block 605, where a pair of illuminated and non-illuminated images are obtained from a user-facing camera. For example, the illuminator may be active when the first set of images is captured by the user-facing camera. Then, another set of images is captured while the illuminator is inactive. These cameras may be located, for example, in front of the eyes of the user. In some embodiments, a single camera may be used, and the camera may or may not be aligned with the user's eyes. Additionally or alternatively, a camera may be located in the headset to face the eyes or eye regions of each user.

At block 610, a determination is made as to whether the device is in an active mode. According to one or more embodiments, a device may be considered to be in an active mode if the device is being worn by a user. For example, if a proximity object is detected based on image data captured by a user-facing camera, the device may be considered to be worn by the user, as will be described below.

If a determination is made that the device is currently in an active mode, the flow chart proceeds to block 615 where a determination is made as to whether an off-head threshold is met. That is, luminance statistics of pixels in an image are used to determine whether a threshold is met, for example as described above with respect to fig. 5. As described above, in some embodiments, the threshold may be associated with a brightness threshold from which it is determined that a proximity object is not detected. In this way, the threshold value used to determine that the device is no longer on-head may be lower than the brightness value used to determine whether the device is on-head. If the off-head threshold is not met, the flowchart returns to block 605 and the device continues to capture both illuminated and unlit images.

Returning to block 615, if a determination is made that the off-head threshold is met, the flow chart proceeds to block 620 where a determination is made whether the time threshold is met. For example, to switch from active mode to inactive mode, a determination may be made that an off-head threshold is met for a predetermined number of frames or for a predetermined period of time. That is, if a single frame indicates that the off-head threshold is met, the result is ignored as an anomaly if the frame is in a set of temporarily captured frames that did not otherwise meet the off-head threshold. Further, in some implementations, the time threshold may need to take into account image data captured from one or more cameras. As an example, in some embodiments, to switch from active mode to inactive mode, image data from multiple cameras on the device (such as cameras configured to point to each eye) may indicate that an out-of-head threshold is met. That is, separate determinations may be made for image data captured from each of one or more cameras on the headset. Thus, if the time threshold is met, the flow chart proceeds to block 625 and the device is switched to an inactive mode, and the flow chart proceeds to block 605, where the device continues to capture frame pairs of illuminated and non-illuminated images.

Returning to block 610, if a determination is made that the device is not in an active mode (i.e., the device is deemed to be in an inactive mode), the flow chart proceeds to block 630. At block 630, a determination is made as to whether an overhead threshold is met. That is, luminance statistics of pixels in an image are used to determine whether a threshold is met, for example as described above with respect to fig. 5. As described above, in some embodiments, the threshold may be associated with a brightness threshold from which detection of a proximity object is determined. In this way, the threshold value used to determine that the device is on-head may be higher than the brightness value used to determine whether the device is off-head. If the overhead threshold is not met, the flowchart returns to block 605 and the device continues to capture both lighted and unlit images.

Returning to block 630, if a determination is made that the overhead threshold is met, the flow chart proceeds to block 635 where a determination is made whether the time threshold is met. For example, to switch from active mode to inactive mode, a determination may be made that an overhead threshold is met for a predetermined number of frames or for a predetermined period of time. That is, if a single frame indicates that the overhead threshold is met, the result is ignored as an anomaly if the frame is in a set of temporarily captured frames that did not otherwise meet the overhead threshold. Further, in some implementations, the time threshold may need to take into account image data captured from one or more cameras. As an example, in some embodiments, to switch from active mode to inactive mode, image data from multiple cameras on the device (such as cameras configured to point to each eye) may indicate that an overhead threshold is met. That is, separate determinations may be made for image data captured from each of one or more cameras on the headset. Thus, if the time threshold is met, the flow chart proceeds to block 640 and the device is switched to active mode, and the flow chart proceeds to block 605, where the device continues to capture frame pairs of illuminated and non-illuminated images.

Fig. 7 illustrates a system diagram of a system by which various embodiments of the present disclosure may be practiced. In particular, fig. 7 illustrates an electronic device 700 that is a computer system. The electronic device 700 may be part of a multi-function device such as a mobile phone, tablet computer, personal digital assistant, portable music/video player, wearable device, head-mounted system, projection-based system, base station, laptop computer, desktop computer, network device, or any other electronic system such as those described herein. The electronic device 700 may connect to other devices across a network using a network interface 745. Additionally or alternatively, the electronic device 700 may be communicatively connected to one or more additional devices (such as server devices, base stations, accessory devices, etc.), in which various functions may be housed or distributed across the additional devices. It should be appreciated that the various components and functions within electronic device 700 may be distributed differently across devices or may be distributed across additional devices.

The electronic device 700 includes a processor 720. Processor 720 may be a system-on-chip, such as those found in mobile devices, and include one or more Central Processing Units (CPUs), special purpose Graphics Processing Units (GPUs), or both. Further, processor 720 may include multiple processors of the same or different types. The electronic device 700 may also include memory 750. Memory 750 may include one or more different types of memory that may be used to perform device functions in conjunction with processor 720. For example, memory 750 may include a cache, a ROM, a RAM, or any kind of transitory or non-transitory computer-readable storage medium capable of storing computer-readable code. Memory 750 may store various programming modules (such as proximity detection module 752) during execution that are configured to determine whether a proximity object is detected based on image data captured by camera 725. In some embodiments, the proximity detection module may use data from the eye tracking module 752. Eye tracking module 752 may use user-oriented sensors, such as one or more cameras 725 or other sensors 740, to determine data about the user's eyes. For example, the eye tracking module may determine data regarding eye state based on image data captured from camera 725 (such as an IR camera configured to face the user's eye when the device is worn). In addition, image data may be captured when one or more illumination components (such as LEDs 730) are illuminated. In some embodiments, cameras and other components for eye tracking may be used for proximity detection. In addition, memory 750 may include one or more additional applications 758.

The electronic device 700 may include a set of sensors 740. In this example, the set of sensors 740 may include one or more image capture sensors, ambient light sensors, motion sensors, eye tracking sensors, and the like. In other implementations, the set of sensors 740 also includes accelerometers, global Positioning Systems (GPS), pressure sensors, inertial Measurement Units (IMUs), and the like.

Electronic device 700 may allow a user to interact with an XR environment. Many electronic systems enable individuals to interact with and/or perceive various XR scenes. One example includes a head-mounted system. The head-mounted system may have an opaque display and one or more speakers. Alternatively, the head-mounted system may be designed to receive an external display (e.g., a smart phone). The head-mounted system may have one or more imaging sensors and/or microphones for capturing images/video of the physical scenery and/or capturing audio of the physical scenery, respectively. The head-mounted system may also have a transparent or translucent see-through display 760. A transparent or translucent display may incorporate a substrate through which light representing an image is directed to an individual's eye. The display may incorporate LEDs, OLEDs, digital light projectors, laser scanning light sources, liquid crystals on silicon, or any combination of these technologies. The light-transmitting substrate may be an optical waveguide, an optical combiner, a light reflector, a holographic substrate, or any combination of these substrates. In one embodiment, the transparent or translucent display is selectively switchable between an opaque state and a transparent or translucent state. As another example, the electronic system may be a projection-based system. Projection-based systems may use retinal projections to project images onto the retina of an individual. Alternatively, the projection system may also project the virtual object into the physical set (e.g., onto a physical surface or as a hologram). Other examples of XR systems include heads-up displays, automotive windshields capable of displaying graphics, windows capable of displaying graphics, lenses capable of displaying graphics, headphones or earplugs, speaker arrangements, input mechanisms (e.g., controllers with or without haptic feedback), tablet computers, smart phones, and desktop or laptop computers.

Referring now to fig. 8, a simplified functional block diagram of an exemplary multi-function electronic device 800 is shown, according to one embodiment. Each of the electronic devices may be a multi-function electronic device or may have some or all of the components of a multi-function electronic device described herein. The multi-function electronic device 800 may include a processor 805, a display 810, a user interface 815, graphics hardware 820, a device sensor 825 (e.g., a proximity sensor/ambient light sensor, an accelerometer, and/or a gyroscope), a microphone 830, an audio codec 835, a speaker 840, communication circuitry 845, digital image capture circuitry 850 (e.g., including a camera system), memory 860, a storage device 865, and a communication bus 870. The multi-function electronic device 800 may be, for example, a mobile phone, a personal music player, a wearable device, a tablet computer, or the like.

The processor 805 may execute the necessary instructions to implement or control the operation of the various functions performed by the device 800. The processor 805 may, for example, drive a display 810 and may receive user input from a user interface 815. The user interface 815 may allow a user to interact with the device 800. For example, the user interface 815 may take a variety of forms, such as buttons, a keypad, a dial, a click wheel, a keyboard, a display screen, a touch screen, and the like. The processor 805 may also be, for example, a system-on-chip such as those found in mobile devices and include a dedicated Graphics Processing Unit (GPU). The processor 805 may be based on a Reduced Instruction Set Computer (RISC) or Complex Instruction Set Computer (CISC) architecture or any other suitable architecture and may include one or more processing cores. Graphics hardware 820 may be special purpose computing hardware for processing graphics and/or auxiliary processor 805 to process graphics information. In one implementation, graphics hardware 820 may include a programmable GPU.

Image capture circuit 850 may include one or more lens components, such as 880A and 880B. The lens assembly may have a combination of various characteristics, such as different focal lengths, etc. For example, lens assembly 880A may have a short focal length relative to the focal length of lens assembly 880B. Each lens assembly may have a separate associated sensor element 890. Alternatively, two or more lens assemblies may share a common sensor element. The image capturing circuit 850 may capture still images, video images, enhanced images, and the like. The output from the image capture circuit 850 may be processed, at least in part, by: video codec 855, processor 805, graphics hardware 820, and/or a dedicated image processing unit or pipeline incorporated within circuitry 845. The images so captured may be stored in memory 860 and/or storage 865.

The memory 860 may include one or more different types of media used by the processor 805 and the graphics hardware 820 to perform device functions. For example, memory 860 may include a memory cache, read Only Memory (ROM), and/or Random Access Memory (RAM). Storage 865 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. Storage 865 may include one or more non-transitory computer-readable storage media including, for example, magnetic disks (fixed, floppy, and removable disks) and tapes, optical media such as CD-ROMs and Digital Video Disks (DVDs), and semiconductor memory devices such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs). Memory 860 and storage 865 may be used to tangibly retain computer program instructions or computer readable code organized into one or more modules and written in any desired computer programming language. Such computer program code, when executed by the processor 805, for example, may implement one or more of the methods described herein.

It is to be understood that the above description is intended to be illustrative, and not restrictive. Materials have been presented to enable any person skilled in the art to make and use the disclosed subject matter as claimed and to provide the material in the context of particular embodiments, variations of which will be apparent to a person skilled in the art (e.g. some of the disclosed embodiments may be used in combination with one another). Accordingly, the particular arrangement of steps or acts illustrated in fig. 2-3 and 5-6 or the arrangement of elements illustrated in fig. 1, 4 and 6-8 should not be construed as limiting the scope of the disclosed subject matter. The scope of the invention should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the terms "including" and "in which" are used as the plain-Chinese equivalents of the respective terms "comprising" and "characterized by.

The techniques defined herein allow for the option of obtaining and utilizing personal information of a user. Such personal information may be utilized, for example, to provide multi-user communication sessions on electronic devices. However, in terms of collecting such personal information, such information should be obtained with the user's informed consent so that the user knows and controls the use of his personal information.

Parties having access to personal information will utilize the information for legal and reasonable purposes only and will comply with privacy policies and practices in accordance with at least the appropriate laws and regulations. Moreover, such policies should be perfect, user accessible, and considered to meet or exceed government/industry standards. Furthermore, personal information will not be distributed, sold, or otherwise shared, except for any reasonable and legal purpose.

However, the user may limit the extent to which personal information is available to the parties. The processes and devices described herein may allow for changing settings or other preferences such that a user controls access to their personal information. Furthermore, while some of the features defined herein are described in the context of using personal information, aspects of these features may be implemented without the need to use such information. As an example, the personal information of the user may be masked or otherwise generalized such that the information does not identify the particular user from which the information was obtained.

Claims (20)

1. A system, comprising:

one or more cameras;

one or more illuminators;

one or more processors; and

one or more computer-readable media comprising computer-readable code executable by one or more processors to:

Obtaining image data captured by the one or more cameras, wherein the image data is captured while at least one of the one or more illuminators is illuminated;

determining a brightness statistic value according to the image data;

determining whether the luminance statistic meets a predetermined threshold; and

in accordance with a determination that the luminance statistic meets the predetermined threshold, detection of a proximate object is determined.

2. The system of claim 1, wherein the system comprises a wearable device, and wherein the one or more cameras are located in front of the eyes when the wearable device is worn.

3. The system of claim 2, further comprising computer readable code for:

and powering on a display of the device according to detecting that the object is close to the wearable device.

4. The system of claim 2, further comprising computer readable code for:

in response to determining that the proximity object is detected, determining that the wearable device is being worn; and

in response to determining that the wearable device is being worn, one or more systems of the device are activated.

5. The system of claim 1, wherein the computer readable code for determining whether the brightness statistic meets a predetermined threshold further comprises computer readable code for:

Identifying a number of pixels in the image data that satisfy a luminance range; and

it is determined whether the number of identified pixels meets a proximity threshold.

6. The system of claim 1, wherein the one or more cameras comprise one or more infrared cameras, and wherein the one or more illuminators comprise one or more Light Emitting Diodes (LEDs).

7. The system of claim 1, further comprising computer readable code for:

obtaining additional image data captured by the one or more cameras;

determining whether a luminance statistic of the additional image data meets a predetermined threshold;

determining a lack of a proximate object in accordance with a determination that the luminance statistic of the additional image data fails to meet the predetermined threshold; and

powering down the display in accordance with the determination that the proximity object is absent.

8. The system of claim 1, wherein the computer readable code for determining luminance statistics from the image data further comprises computer readable code for:

identifying a lit image and an unlit image in the image data;

removing ambient light from the illuminated image based on the unlit image to obtain a modified illuminated image; and

A mask is applied to the illuminated image to obtain an occluded image,

wherein the luminance statistic is determined from a region of interest in the occluded image.

9. A non-transitory computer-readable medium comprising computer-readable code executable by one or more processors to:

obtaining image data captured by one or more cameras of a device, wherein the image data is captured while one or more illuminators are illuminated;

determining a brightness statistic value according to the image data;

determining whether the luminance statistic meets a predetermined threshold; and

in accordance with a determination that the luminance statistic meets the predetermined threshold, detection of a proximate object is determined.

10. The non-transitory computer readable medium of claim 9, wherein the device comprises a wearable device, and wherein the one or more cameras are located in front of the eyes when the wearable device is worn.

11. The non-transitory computer-readable medium of claim 10, further comprising computer-readable code for:

and powering on a display of the device according to detecting that the object is close to the wearable device.

12. The non-transitory computer-readable medium of claim 10, further comprising computer-readable code for:

in response to determining that the proximity object is detected, determining that the wearable device is being worn; and

in response to determining that the wearable device is being worn, one or more systems of the device are activated.

13. The non-transitory computer-readable medium of claim 9, wherein the computer-readable code for determining whether the brightness statistic meets a predetermined threshold further comprises computer-readable code for:

identifying a number of pixels in the image data that satisfy a luminance range; and

it is determined whether the number of identified pixels meets a proximity threshold.

14. The non-transitory computer-readable medium of claim 9, wherein the one or more cameras comprise one or more infrared cameras, and wherein the one or more illuminators comprise one or more Light Emitting Diodes (LEDs).

15. The non-transitory computer-readable medium of claim 9, further comprising computer-readable code for:

obtaining additional image data captured by the one or more cameras;

Determining whether a luminance statistic of the additional image data meets a predetermined threshold;

determining a lack of a proximate object in accordance with a determination that the luminance statistic of the additional image data fails to meet the predetermined threshold; and

powering down the display in accordance with the determination that the proximity object is absent.

16. The non-transitory computer-readable medium of claim 9, wherein the computer-readable code for determining luminance statistics from the image data further comprises computer-readable code for:

identifying a lit image and an unlit image in the image data;

removing ambient light from the illuminated image based on the unlit image to obtain a modified illuminated image; and

a mask is applied to the illuminated image to obtain an occluded image,

wherein the luminance statistic is determined from a region of interest in the occluded image.

17. A method, comprising:

obtaining image data captured by one or more cameras of a device, wherein the image data is captured while one or more illuminators are illuminated;

determining a brightness statistic value according to the image data;

Determining whether the luminance statistic meets a predetermined threshold; and

in accordance with a determination that the luminance statistic meets the predetermined threshold, detection of a proximate object is determined.

18. The method of claim 17, wherein the device comprises a wearable device, and wherein the one or more cameras are located in front of the eyes when the wearable device is worn.

19. The method of claim 18, further comprising:

and powering on a display of the device according to detecting that the object is close to the wearable device.

20. The method of claim 17, wherein determining the luminance statistic from the image data further comprises:

identifying a lit image and an unlit image in the image data;

removing ambient light from the illuminated image based on the unlit image to obtain a modified illuminated image; and

a mask is applied to the illuminated image to obtain an occluded image,

wherein the luminance statistic is determined from a region of interest in the occluded image.