KR20140002034A - Presence sensing - Google Patents

Abstract

One embodiment may take the form of a method of operating a computing device to provide presence-based functionality. The method may include operating the computing device in a reduced power state and collecting a first data set from at least one sensor. Based on the first data set, the computing device determines the probability that the object is within the threshold distance of the computing device, and if the object is within the threshold distance, the device activates at least one second sensor to collect the second data set. do. Based on the second data set, the device determines whether the object is a person. If the object is a person, the device determines a person's position relative to the computing device and executes a state change of the computing device based on the person's position relative to the computing device. If the object is not a human, the computing device remains in a reduced power state.

Description

Presence detection {PRESENCE SENSING}

Cross-reference to related application

This Patent Cooperation Treaty patent application claims the priority of US Patent Application No. 13 / 469,996, filed May 11, 2012, entitled "PRESENCE SENSING," which is filed with the Agent Patent No. P11391USX1 (P222617. US Pat. No. 13,219,573, filed Jul. 26, 2011, " PRESENCE SENSING ", filed Jul. 26, 2011, " PRESENCE SENSING. &Quot; US Patent Application No. 61 / 504,026, and US Provisional Patent Application No. 61 / 485,610, filed May 12, 2011, entitled " PRESENCE SENSING, " all of which are incorporated herein by reference. It is incorporated by reference in its entirety and for all purposes.

TECHNICAL FIELD The present invention generally relates to devices with computing capabilities, and more particularly, to apparatus and systems for sensing the presence of a user locally in proximity to the device.

Many computing devices have power saving features / modes intended to reduce power consumption when the user is not using such a device. Often, this power saving function is implemented through a timer that counts down the amount of time set since the user last provided input to the device. For example, a particular device may be configured to enter a sleep mode or other mode that consumes less power than a fully operational mode if the user does not provide input for five minutes.

However, from time to time, the device may enter a power saving function / mode while the user is still using the device. For example, a user may enter a power saving function because the user did not provide input within the time period set in the timer while reading content, watching a movie, or listening to music on the device. In addition, restoring from a power saving function / mode can be time consuming and even requires a user to enter credentials, which can generally be annoying for the user.

One embodiment may take the form of a method of operating a computing device to provide presence based functionality. The method may include operating the computing device in a reduced power state and collecting a first data set from the first sensor or group of sensors. Based on the first data set, the computing device determines the probability or likelihood of the object being located in proximity to the device. Additionally or alternatively, the computing device may, for example, make a hard decision on proximity of the object, the computing device may determine whether the object is within a threshold distance of the computing device, and if the object Is within a threshold distance and / or is likely to be located in proximity to the device, the device activates the second sensor to collect a second data set. Based on the second data set, the device determines whether the object is a person. If the object is a person, the device determines a person's position relative to the computing device and executes a state change of the computing device based on the person's position relative to the computing device. If the object is not a human, the computing device remains in a reduced power state.

Another embodiment may take the form of a method for determining if a user is in proximity to a computing device. The method comprises capturing an image using an image sensor and from the captured image, a skin tone detection parameter, a face detection parameter, a body detection parameter and a movement detection parameter. Calculating at least one. The method also includes determining whether the user exists using at least one of a skin color detection parameter, a face detection parameter, and a motion detection parameter, and if the user determines that the user exists, changing the state of the computing device.

In yet another embodiment, a computing system having a main processor and a presence sensor coupled to the main processor is provided. The presence sensor includes an image sensor and a processor coupled to the image sensor and configured to process the captured image to determine whether the user exists within the image. The image sensor may be configured to capture 3-D images, depth images, RGB images, and / or grayscale images, and one or more of the images may be used for probability determination. Other inputs may also contribute information that is useful for determining existence. For example, input devices such as keyboards, mice, and / or microphones may each provide input useful for determining presence probabilities. If the processor determines that the user exists in the image, an indication that the user is determined to be present is sent from the processor to the main processor and the main processor changes the state of the computing system based on the indication.

While many embodiments are disclosed, other embodiments of the invention will become apparent to those skilled in the art from the following detailed description. As will be appreciated, these embodiments are capable of modification in various respects, all without departing from the spirit and scope of such embodiments. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not as restrictive.

1 illustrates an example computing device having a user presence sensing function.
2 is a block diagram of the computing device of FIG. 1.
FIG. 3 is a plot showing the presence detection result when an object of interest is located at a different distance from a presence sensor.
4 is another plot showing the presence detection result when the object of interest is offset by a certain angle from the sensor.
5 is a flowchart illustrating an example method of operating a tiered presence sensor system.
6 is a flowchart illustrating a method of determining the presence of a user.
7 is a flowchart illustrating a skin tone detection routine for use in presence detection.
8 is a flowchart illustrating a face recognition routine for use in presence detection.
9 is a flowchart illustrating a motion detection routine for use in presence detection.
10 illustrates a frame divided into windows for single frame motion detection.
11 is a flowchart illustrating an example method for single frame motion detection.
12 is a flowchart illustrating an example method for multiple concurrent asynchronous sensors used to determine the probability that a user will be located in proximity to the device.

In general, the embodiments described herein relate to user presence determination and related computing device functionality. It should be appreciated that the user's experience of interacting with such functional computing devices may be enhanced. In addition, in some embodiments, power savings and / or power efficiency may be realized through implementation of the embodiments described herein. As used herein, a "user" can generally refer to a person or people. However, in some embodiments, the system can be configured to detect an object other than a person and change the state based on the presence of the object.

One embodiment may take the form of a computing device configured to sense the presence and / or absence of a user and provide an operating state based on the presence and / or absence of the user. In another embodiment, the computing device may calculate and provide a likelihood or probability score that the user may or may not be present. In some embodiments, a plurality of parameters may be determined together, weighted, and used when making a presence determination. Such weighted detection can be used in a more informed high level decision making algorithm, or when fusing data from several sensors.

For example, in some embodiments, the computing device may be configured to determine based on sensor input the time at which the user reaches or is in proximity to the computing device and / or the probability that the user is present. As the probability is calculated, false positives and / or false negatives may be reduced. Bayesian optimal thresholds may be implemented that achieve the desired performance. That is, the triggering threshold may be set in such a way as to achieve a desired balance between the number of acceptable false positives to the number of acceptable false negatives. Depending on the affirmation that the user is present or when achieving a threshold probability that the user is present, the device powers up, exits sleep mode, and / or provides some feedback to the user. Can provide.

In general, one purpose of a presence sensing system may be to use technology more intelligently. For example, in some embodiments, system awake may begin if it is determined that the user is approaching. System awake may include a reduced set of routines that allow the system to enter an operating mode faster than a normal power up sequence. For example, the system can power up in 1/2 seconds instead of 6-8 seconds due to the reduced set of routines. In some embodiments, the computing device may be configured to determine a time for the user to move away from or in proximity of the device. In response, the device may enter a power saving mode, such as display sleep mode, system sleep mode, activation of a screen saver, and the like. The system can also partially exit sleep mode to speed up the computer wake up time based on sensing the presence of the user. In yet another embodiment, the displayed content may be based on the relative location of the user. For example, if the user is close to the device, the device may provide product details, while if the user is far away the device may display nothing.

In some embodiments, the device also tracks movements (eg, position and velocity) of the user, provides feedback in response to certain movements and / or enters or changes operating states. It can be configured to. For example, movement to the device may activate more features, such as providing more options / menus in the user interface, while movement away from the device reduces the number of menus / options and / or displays It is possible to reduce the number of functions available to the user, such as reducing or increasing the size of the options that are made. Additionally or alternatively, the display can zoom in or zoom out based on movement to or away from the device. In some embodiments, the lateral (eg, left to right) movement of the user may change the background and / or screensaver images displayed on the device. Also, the change in image may generally correspond to the sensed motion. For example, movement from left to right may cause an image to be replaced with another image in motion from left to right. Alternatively, as the user moves from left to right, leaves or curtains may reflect that movement. That is, the leaves may fly from left to right, or the curtain may sway in a manner corresponding to the detected movement.

In addition, in some embodiments, the presence of the user may be used to provide certain functionality along with the user's location with respect to the device. In some embodiments, inputs and / or outputs may be based on location. For example, the device may be configured to provide audio stereo panning (eg, audio steering) towards the user's location. Also, in some embodiments, microphone steering may be implemented based on the location of the user.

In addition, multiple sensors and / or operating states may be implemented in a tiered manner. That is, the first sensor may operate in the first operation mode. The second sensor or the like may be activated when detecting the movement or the presence of the user. In some embodiments, activation of the second sensor may occur simultaneously with the device entering the second mode of operation, while in other embodiments only based on data retrieved from the second sensor or with data from the first sensor. In combination, the second operation mode may not be entered until a determination is made.

Presence determination can be made by data collected by one or more sensors. In one embodiment, data from one or more sensors is used to determine if a user is present. For example, neural nets, support vector machines or other suitable classifiers or probabilistic determiners can be implemented. In some cases, large sets of data points can be collected, sorted and stored for use in presence determination. In addition, data obtained later may be added and used for future judgment.

Turning to the drawings and referring first to FIG. 1, a computing device 100 is illustrated. Computing device 100 may generally include one or more sensors 102 that may be used for presence detection. For example, one or more cameras and / or photosensors may be used for presence detection. Although cameras and light sensors are generally described herein with respect to presence detection, it should be appreciated that other sensor types, such as ultrasonic sensors, microwave radar (RADAR), and the like, may also be implemented. Some of these techniques can be used to determine physiological parameters when the human is in proximity to the device. For example, radar can be used to detect and / or locate a heartbeat in a room. In addition, various techniques and light wavelengths may be implemented. For example, proximity may be determined using depth sensors. Some example depth sensor technologies include focusing and defocusing, active IR reflected power, active IR structured light; Active IR time of flight (2D + depth), active IR flight time (single pixel sensor), passive IR (motion detector), passive IR thermal imaging (2D), stereo vision, polarization (polarization) technology and the like. In some embodiments, active IR can detect certain intrinsic material properties such as human skin reflectivity or the release of carbon dioxide in the breath using multiple specific IR wavelengths. As such, the specific embodiments described herein are presented by way of example only and not limitation.

2 is a block diagram of the computing device 100 of FIG. 1. In general, a computing device includes a microprocessor / microcontroller 104 to which other components (eg, sensors) are connected. The microprocessor / microcontroller 104 may be implemented with data from one or more low power microcontrollers and sensors (eg, cameras, proximity, etc.), as well as data fusion points for high level user presence or non-existent decision making. . In some embodiments, the user presence determination and associated data may be externalized and separated from the main operation of the device. That is, the user presence system separates presence sensing data from the main computer processing unit (CPU) 105, operating system, and the like to provide security and privacy.

Various suitable sensors may provide input / data to the microprocessor / microcontroller 104. In particular, the camera based sensor 106 may be communicatively coupled with the microprocessor 104. Any suitable camera based sensor may be implemented and various other techniques may be used. For example, camera sensors available from ST Microelectronics may be used. The camera-based sensor may include a full image camera 108 that provides face detection functionality to the integrated processor 110. That is, the sensor may have an embedded microprocessor 110 and may estimate face position and distance. This sensor can also be used to determine the distance of an object. Camera 108 also provides windowed histogram information that may be useful for motion detection from an AGC system.

The camera 108 may also have a horizontal field of view up to or greater than 120 degrees and a vertical field of view up to or greater than 120 degrees. In some embodiments, lenses such as fish eye lenses may be used to obtain a field of view with an angle greater than 120 degrees. In one embodiment, the horizontal field of view may be between 75-95 degrees (eg, approximately 85 degrees) and the vertical field of view may be between 40-80 degrees (eg, approximately 60 degrees). Faces can be detected at distances up to 20 feet or more. In one embodiment, the face can be detected at approximately 6-14 feet. Facial position data may be available between approximately 0.6-1 Hz and AGC data may be available at full frame rate, approximately 10-30 Hz.

In general, images captured by camera-based sensor 106 and related raw information may not be available outside of the camera-based sensor. Rather, information about whether a face is detected within the functional range of the sensor, the position of the face and / or the movement of the face within that range may be provided. In some embodiments, the camera sensor may provide a binary output that indicates the presence of a user. In addition, if the user is present, the user's position relative to the device may be output by the camera-based sensor, eg in x-y coordinates. The sensor can also be configured to indicate, among other things, the number of faces present (eg, the number of people present).

In some embodiments, camera-based sensor 106 may be implemented independent of other sensors to achieve desired operating characteristics for the device. In some embodiments, the camera based sensor may be configured to operate in a stepwise manner and provide output. For example, in the first state, the camera-based sensor may detect the presence of the user. If there is a user, the camera-based sensor may enter the second state and determine how many people are present. Then, or simultaneously, camera-based sensors can determine the location of people present. As the camera moves from the operating state, the camera provides an output that can be used by the device to change the state of the device as described in more detail below.

Some embodiments may capture an image using the main camera 103. Main camera 103 may be a system camera used to capture video and still images by a user of the device. In some embodiments, the main camera may be separate and different from the camera used for presence detection (eg, there are many cameras in the system), while in other embodiments the main camera output is dedicated by a camera based sensor. ) May be used in place of a camera based sensor 106 with a camera. For example, in some embodiments, in addition to the main camera, a depth based camera may be implemented. Depth-based cameras can take any suitable form. In one embodiment, the main camera output may be provided not only to the image processor 107 for user use but also to the microcontroller of the camera based sensor 106 for user presence detection. There may be several options as to how the image processor and user detection co-processor can communicate data with and make data from the main camera available to the user. For example, if there is no user, the output from the main camera can be processed primarily by the microcontroller for presence detection determination. In this state, data from the camera may not generally be available to other components of the system. If the user is present, output from the main camera can be provided to the image processor 107. However, in order to make image data / information available, a user may need to access a camera based application (eg, a video chat application, or an image capture program, etc.). Otherwise, you may generally not be able to access image data from the camera.

It should be appreciated that in addition to one or more cameras, there may be many other configurations capable of detecting the desired presence using conventional camera and image processing functions. For example, in one embodiment, the main camera output may be routed to a single chip that combines conventional image processing and user presence detection. In another embodiment, the video output from the camera can be streamed to the host for processing by the central processing unit.

A second sensor, such as proximity sensor 112, may also be connected to microprocessor 104. In some embodiments, controller 114, multiplexer 116, and light emitting diode array 118 may operate in conjunction with proximity sensor 112. In particular, the controller 114 may be configured to control the operation of the multiplexer 116 and the LEDs 118 in a time division multiplexed (TDM) manner. Depending on the TDM alternating of the LEDs, appropriate filters can be implemented to obtain the desired response. In other embodiments, mechanical devices (eg, micro-electrical devices) may be used to multiplex one or more LEDs to cover different views.

LED 118 may operate in any suitable range of wavelengths and, in one example, may operate in the near infrared region of the electromagnetic spectrum. Each of the LEDs LED1-LEDN can be directed to a particular field of view. In some embodiments, each LED 118 may target a separate field of view, while in other embodiments the fields of view of adjacent LEDs may overlap. In some embodiments, the LED array 118 can distribute the LEDs around the bezel of the computing device. In another embodiment, the LED array 118 may be configured in rows (eg, over a curved portion of the display screen bezel) where the LEDs are arranged directionally to cover different views.

In one embodiment, an average value of the field of view (eg, a value representing proximity) may be obtained and used to determine whether the object is in proximity to the device 100. If the average value exceeds the threshold, it may indicate that the object is in proximity of the device 100. Using the LED array 118, the proximity sensor can detect proximity more accurately over a wide field of view. Because each LED points to a different field of view, the position of the object can also be determined using proximity sensor 112. As such, in other embodiments, a change in proximity value from the estimated empty scene may be determined. The largest change (or any rank) can be reviewed across various sensors and this value can determine proximity and / or location compared to a threshold.

In some embodiments, camera-based sensor 106 and proximity sensor 112 may be used with microprocessor 104 to determine whether a user is in proximity to computing device 100. Among other purposes and / or functions, a phased sensing system may be implemented to provide power savings, enhance the user experience, or provide a particular user experience desired. In particular, the staged sensing system may operate the first sensor to first determine the presence of the user within the threshold distance of the computing device to provide power savings. In some embodiments, the threshold distance may be within 2-10 feet (eg, 5 feet) of the device 100. Also, in some embodiments, data collected at the first sensor can be used to determine the relative location of the user.

In this staged system, if a user is present, the second sensor can be activated. The data from the second sensor may be used alone or in combination with data from the first sensor to further identify the user / person and / or the location of the user. The data from both the first and second sensors can be used together to determine what function to perform and / or what the user is doing. For example, it may be determined how close the user is to the device, ie whether the user is facing the device, whether the user moves away from / toward the device, and so on. Such data can also be used to identify a user (eg, as a credentialed user).

The state of the computing device 100 may change based on determining that a user exists. For example, if the user accesses the device, the display can be awake, the system can be awake and the like. If the user is moving from left to right, the displayed image can be changed and generally move corresponding to the user's movement. In addition, if a large number of users are present (as determined based on identifying the presence of many faces), the device 100 may be powered in a secure state and require input of user credentials to fully access the device. can do.

Presence determination can be based on many factors used in neural networks, support vector machines (SVMs) or other machine learning based classifiers or probability determination systems. For example, skin color / color, presence and movement can be used in a weighted manner with neural networks making a presence determination. As mentioned above, based on the presence determination, the device 100 may enter and / or change the operating state.

It should be recognized that the choice of a particular sensor to use will depend on a variety of factors, including, for example, desired functionality and power consumption limitations. As such, in some embodiments, camera-based sensor 106 may be implemented as a first tier sensor, while in other embodiments, proximity sensor 112 may be used as a first stage sensor. . A more detailed description of implementing a proximity sensor such as proximity sensor 112 is provided below.

The sensor 112 may chop the light at any suitable frequency and measure the phase shift of the returned reflected optical signal. The output of the LED 118 may be square waves or other waveforms, and the sensor 112 uses an I / Q demodulation scheme. Light coming from sensor 112 is mixed with sine waves and cosine waves to provide I (in-phase) components and Q (quadrature) components. The sine / cosine wave is synchronized with the LED modulation. These correspond to the 'raw' output from the sensor, and if other internal measurement methods exist, it can be converted in this way. Without loss of generality, it can be assumed that the period is 2π and integration occurs throughout the period. In practice, a fixed period may be used and will integrate over any large multiple of that period. This difference results in a fixed scale factor, which can be ignored. Basic measurement components are as follows.

이고 크기가 A인 동일 주파수의 구형파 입력 신호는 I 및 Q 성분이 다음과 같다.If you are measuring an object at a distance (radiation) from the sensor 112 that covers the entire field of view, the phase offset

A square wave input signal of the same frequency of magnitude A has the following I and Q components.

And,

이다.

to be.

값은 다음과 같이 구할 수 있다.Next,

The value can be obtained as follows.

Then M can be reconstructed as

및

로 나타낼 수 있다. 반사 신호의 크기는 A 및 B로 규정될 수 있다. 이 경우, 입력 광신호가 추가되므로, 적분이 있고, 따라서 I는, Assuming there are two objects A and B within the field of view of the sensor (each of which is at a certain distance), the phase shift associated with these distances is

And

. The magnitude of the reflected signal can be defined as A and B. In this case, since the input optical signal is added, there is integration, so I is

이다.

to be.

Similarly, for Q,

이다.

to be.

A light source whose intensity does not change over time will provide zero contribution to the I and Q components. This property provides good ambient light rejection. This feature can also be canceled due to the phase offset from objects at different distances. With one / zero square wave demodulation, this information can be maintained but the ambient light exclusion can be worsened. This demodulation yields slightly different formulas, but the final result will be similar. In the following, the factor 2 in front of I / Q will be dropped because it will be absorbed in another scale factor.

)을 갖고, 고정 방사 거리(r_i)에 있다. 또한, LED로부터의 출력은 스테라디안(steradian)(I_i) 당 방출 세기(emitted intensity)를 갖는 소정의 입체각 전체에 걸쳐 일정하다. 소정의 거리에 대한 위상 시프트는

로 규정된다. Some simplification can be made and a basic model of sensor output as a function of objects in the scene is proposed. Separate cases will also be developed because the other cases are better with the implementation, but other cases can also be implemented. The LED / sensor field of view may be divided into N sections indexed from 1 to N. Each of these sections has a solid angle of Ω _i . In addition, each of these solid angles has a fixed reflectance (

) And is at a fixed spinning distance r _i . In addition, the output from the LED is constant throughout a given solid angle having an emitted intensity per steradian (I _i ). Phase shift for a given distance

.

는 이미 규정되었다. From this model, the contribution (I _i , Q _i ) from a given solid angle at the sensor can be obtained. It is also useful to define a polar coordinate system in the I / Q space. The magnitude of the IQ vector is defined by M _i , the angle

Has already been prescribed.

(I _m , Q _m ) can both be defined as the measured (raw) I and Q values. One or more terms (I _c , Q _c ) can be added to represent some constant crosstalk (electric or light). Finally,

이다.

to be.

In general, it may be advantageous to understand the environment in which the device is placed in order to determine whether the user is located in proximity to the device. This may help to reduce false positives and more accurately determine the time at which the user enters or exits the proximity of the device 100. However, creating a background model presents many challenges due to the lack of relative information provided by the sensor 112. In order to define a useful model, some simplified assumptions can be made. First, the calculation of the model of a single sensor will be processed and then the case of several sensors.

In essence, two types of objects affect the distance measurement provided by a particular proximity sensor, such as sensor 112. There are objects that can't be hidden by people, and there are objects that can be hidden by people. The former will be referred to as the 'foreground' object and the latter will be referred to as the 'background' object. Of course, an object can fall into two classes depending on how it is located with respect to a person. First, the scene can be divided into these two types of objects. In general, the challenge is to measure the distance to dynamic objects, such as those who enter and leave the scene. In order to measure these objects successfully, accurate models of static objects in the scene are created and their relationship to dynamic objects is modeled.

First, (I _p , Q _p ) is defined as the signal associated with the object being measured. (I _m , Q _m ) and (I _c , Q _c ) can continue to be used as measured (raw) and crosstalk values, respectively.

Empty Scene

One model assumes no foreground or background objects, and the signals are all due to the people in the scene. In its purest form, factory calibration / crosstalk values can be used as follows.

This model can be used to generate a distance output. For scenes without foreground objects, this model will always overestimate distance. Note that this model relies on accurate factory calibration values for the life of the device. This model may not account for the added crosstalk due to smudge / etc.

Once a static offset is observed, that static offset is modeled with any combination of foreground and background objects. The choice of how to distribute these static offsets has a strong impact on the estimates of I _p and Q _p .

Foreground Only

One way to describe a static offset is to assume that the static offset is all due to the foreground object. Effects such as crosstalk changes due to temperature or smudges fall into this class. Foreground objects, by definition, contribute to a signal constantly, regardless of the person's presence. In a pure foreground model, the spatial distribution of the foreground object is irrelevant and assumes that anything other than the foreground is the object of interest. Let's define the signal from the foreground as (I _fg , Q _fg ). This model implies:

Note that (I _fg + I _c , Q _fg + Q _c ) is the measured sensor reading with no object of interest in the scene. This is a standard 'baseline subtraction' model.

Uniform Background with Partial Occlusion

For this model, it is assumed that the background is at a uniform distance and has a uniform reflectance. It is also assumed that the object obscures the field of view vertically. The LED falloff with angle is defined as l (θ). It is assumed that a single object having a fixed width w corresponds to the angular section Δθ _p at the fixed position. The central position of the object is defined by the angle term θ _p .

The general model has been described above. For this model, the area is purely a function of the width, the incident light is defined as l (θ), and the distance / reflectivity is constant but unknown.

For convenience, it is prescribed as follows.

, 및

, And

L (θ _p ; Δθ _p ) represents the portion of light from the LED aimed at the solid angle defined by the object of interest, L _total represents the total light output, and R (θ _p ; Δθ _p ) is reflected in the background ( cast) represents a portion of the total light.

로 규정할 것이다. 이는 다음과 같이 주어진다. The magnitude of light reaching the sensor from the object of interest is proportional to L (θ _p ; Δθ _p ). Set the proportional constant θ _p , and the phase offset associated with the distance to the object of interest

Will be prescribed. This is given by

, 그리고 배경 거리와 연관된 위상은

로 규정된다. 이는 다음과 같이 주어진다. Similarly, the magnitude of light reaching the sensor from the background is proportional to R (θ _p ; Δθ _p ). The proportional constant is

, And the phase associated with the background distance

. This is given by

to summarize,

Assume the following measurements:

, 및

, And

If the angle θ _p and the width w can be known or assumed, this simultaneous equation can be solved.

Uniform Background and Uniform Foreground with Partial Occlusion

및

으로 규정한다. 이제, 전경의 경우,For this model, we start with a 'uniform background with partial occlusion' model, build it, and add a foreground component that is uniform and does not spatially change its effect on the object of interest. Because the foreground component does not change spatially and is not affected by the presence of the object of interest, it is possible to change the size and phase of the foreground object.

And

It is prescribed by. Now, in the foreground,

, 및

, And

이다.

to be.

Simply add it to the previous model to get:

, 및

, And

Suppose we can measure the following in an empty scene:

, 및

, And

Compared with the previous case, we can add two more estimated variables.

Sectioned Background, Uniform Foreground

The model divides the horizontal field of view into a series of sections 1 ... S, each modeled with a uniform foreground / uniform background. The subscript s is added to indicate the section to which the variable belongs. Starting from the background section, assume that an object exists in the scene whose width w corresponds to the angle section Δθ _p , and the angular position θ _p . Locally redefine the R function to represent a portion of the light reflected in the background after occlusion by the object of interest. This may be referred to as R ^s .

Now,

, 및

, And

Because the foreground signal is not changed by objects in the scene, there is no need to model it locally. However, the foreground may occlude the object of interest at varying angles across the sections. It can be modeled in many different ways, the cleanest of which is to associate the 'occlusion factor' F ^s for each foreground section. L ^s is also defined as the portion of the total light output from the LED illuminating the object of interest in section s. Now it looks like this:

, 및

, And

For a uniform foreground, Fs is equal to 1 for all sections and the formula is shortened back to the case of non-compartmented foreground. In summary.

, 및

, And

Here, two variables are added per section for the background and one variable per section for the foreground occlusion. Occlusion effects from foreground objects can be ignored and only extra background variables are added.

Two Sensors with Overlapping Fields of View

및 고정 폭 w를 갖는다고 가정한다.Two sensors with overlapping fields of view may be used. Consider only what kind of information can be obtained in this area, taking into account only the overlapping part of the field of view, where each sensor has its own L (θ _p ; Δθ _p ) and θ _p indicates the global coordinate system do. These are L ¹ And L ² , and subscripts may be used to represent the sensor. In addition, it is assumed that the two sensors have different sensitivity and LED outputs, which gives a scale factor error α for the measurement of the same object in the overlapping field of view. Also, 1 / d ² for distance and signal size Suppose you have a relationship. In addition, the object has a fixed reflectance

And assume a fixed width w.

,

θ_p 및 d는 두 센서의 측정 사이의 공통 값이고 관심 객체에 특정하다는 것에 유의한다. d와

사이의 관계가 잘 규정되어 있다 - 본 명세서의 일례의 섹션을 참조바람. 여기서, α는 두 센서/LED 사이의 일정한 감도차이며, 이는 그 센서들의 수명 동안 저속으로 변화되어야 한다. 이러한 규정은 다음과 같다.

,

Note that θ _p and d are common values between the measurements of the two sensors and are specific to the object of interest. d and

The relationship between is well defined-see the example section of this specification. Where α is a constant sensitivity difference between the two sensors / LEDs, which should change at low speeds over the lifetime of those sensors. These regulations are as follows.

, 및

, And

및

에 대한 수식을 생성할 수 있다. 다음과 같은 다섯 가지 알려지지 않은 것이 있다.These formulas can be replaced with a background only partial occlusion model for Ip and Qp

And

You can create a formula for. There are five unknowns:

Α

·

·

Θ _p

Δθ _p

는 바로 측정될 수 있다. 불행하게도, 이들 수식은 비선형이므로, 이러한 제약 내에서 유일해(unique solution)가 존재한다는 것을 증명하는 어떤 작업이 여전히 수행될 수 있다. 이러한 추정 프로세스를 성취하기 위해, 많은 추정 방식 중 어떤 추정 방식이 이용될 수 있다. 그 예는 확장 칼만(Kalman) 필터, 시그마-포인트(sigma-point) 칼만 필터, 또는 직접 추정을 이용하는 것을 포함할 수 있다.Also, there are four equations, so one of these values is known If you do (or can only assume), the rest can potentially be calculated. It is reasonable to assume that first, α and Δθ _p can be estimated well. For example, once another sensor is provided, such as camera based sensor 106, θ _p and

Can be measured immediately. Unfortunately, since these equations are nonlinear, some work can still be done to prove that there is a unique solution within this constraint. In order to achieve this estimation process, any of a number of estimation methods may be used. Examples may include using an extended Kalman filter, a sigma-point Kalman filter, or direct estimation.

Example Implementation of Background Only Partial Occlusion Model

The falloff cast by the 10 degree LED sensor 112 was imaged against a white wall. Its horizontally projected falloff is a Gaussian with a standard deviation of approximately 12 degrees. The prototype was placed about 3.5 feet above the floor of a relatively dark empty room with a backdrop of 12 feet.

Crosstalk was measured with a black felt baffle covering the sensor 112. Zero phase offset was measured with reflective baffle. A nominal 'open' background was captured. Sensor data was collected from a person standing in 1-foot increments, leaving sensor 112 with an offset of 0 degrees from an LED 10 feet away. Sensor data was collected from -15 degrees to +15 degrees in 5 degree increments at 5 feet of radiation. Felt measurements can be referred to as (I _c ; Q _c ) because they measure crosstalk in essence. Reflective baffle measurements may be referred to as (I _o ; Q _o ) and open measurements as (I _open ; Q _open ). Finally, the primitive measurement with the object of interest in the scene may be referred to as (I _m ; Q _m ) and the object of interest signal to be estimated is (I _p ; Q _p ). L (θ _p ; Δθ _p ) was modeled on the premise of the Gaussian distribution described above in which the specific form is as follows.

Where "erf" is an error function. It is also prescribed as follows.

및

And

는 위상 델타(phase delta)에서 거리(distance)로의 변환이고, △θ_p는 폭이 2피트인 사람을 전제로 하여 산출된다. 이제, 연립 방정식은 다음과 같이 설정될 수 있다. here

Is the conversion from phase delta to distance, and Δθ _p is calculated assuming a person 2 feet wide. Now, the simultaneous equations can be set as follows.

, 및

, And

Here, L (θ _p ; Δθ _p ) is expressed using the above formulas for Δθ _p and L (θ _p ; Δθ _p ). Consider θ _p as a known value and solve the nonlinear simultaneous equations numerically. The plots of FIGS. 3 and 4 show the actual data results. In FIG. 3, line 120 indicates no correction and line 122 indicates the corrected data. In FIG. 4, line 130 indicates no correction, line 132 indicates corrected data, and line 134 indicates the actual distance.

By calculating a single sensor 112 with various background models, multiple sensors can be combined into an integrated location model. As described above, in one embodiment multiple proximity sensors may be implemented. In another embodiment, multiple LEDs can be used in a TDM fashion to provide the desired field of view. Integrating camera-based sensors requires estimation of all parameters of interest.

5 illustrates a method 300 of changing the state of an apparatus using multiple sensors in a stepwise manner. First, the device may be in a power consumption reduction mode, such as a sleep mode, and the controller may receive data from the first sensor (block 302). The received data is processed (block 304) and compared with a threshold (block 306). In comparison to the threshold, a determination may be made whether the user exists or is likely to exist (block 308). If the user does not exist, data may continue to be received from the first sensor (block 302). However, if it is determined that the user exists or is likely to be present, then the second sensor may be activated (block 310) and data is received from the second sensor (block 312). Data from the second sensor is processed (block 314) and combined with data from the first sensor (block 316). Processing of data from the first and second sensors is not limited to the following to be useful for determining presence, but may be performed by performing digital signal processing on the data, such as data filtering, data scaling, and / or data conditioning in general. It may include. In addition, combining data from the first and second sensors may include storing the data together and / or combining the data logically or mathematically.

Data from the first and second sensors are used to calculate user presence values and / or user presence probability scores (block 318). The user presence value and / or user presence probability score are compared to the threshold to determine if the user exists (block 322). Also, if it is determined that the user exists, other parameters such as the user's distance and location to the device can be determined (block 324) and the state of the device can be changed (block 326). Such state change may include moving the device from sleep mode to awake mode or other appropriate state change.

In addition, the determination of the state change of the device, as well as other parameters (e.g., distance, location, etc.) may only be made after a positive determination of user presence based on the first sensor data, as indicated by the dashed line from block 308. Be aware that it can happen.

In addition, the second determination of user presence (block 320) may be more accurate than the first determination (block 308) based on the additional information provided from the second sensor. In addition, as discussed above, additional parameters may be determined based on the combination of data from both the first and second sensors.

It should be appreciated that other embodiments may implement more or fewer steps than the method 300. 6-11 illustrate more specific flowcharts of the presence determination method.

Referring to FIG. 6, a flowchart 200 is illustrated that illustrates presence detection. First, an image is acquired using a camera (block 202). Light level determination may be performed (block 204) and provided to the skin color detection routine (block 206). Optionally, in some embodiments, roughness determination may be provided to other routines as indicated by arrow 203. In addition, the captured image may be pre-processed (block 208). In some cases, the preprocessing may include, for example, downscaling the image, changing the color space of the image, and / or enhancing the image. Other detector specific pretreatments may also be performed (blocks 214, 215 and 217). For example, the image may optionally be blurred by preprocessing at block 214 before being provided to the skin color detection routine (block 206). Further, at block 215 the preprocessing changes the color to grayscale before providing the image to face detection routine (block 210) and / or before providing the image to motion detection routine (block 212). Performing edge detection in the preprocessing. The depth image may be preprocessed by segmenting it and the segmented portion may be provided to the detector as input. The skin color detection routine, the face detection routine, and the motion detection routine are described in more detail with reference to FIGS. 7 to 11 below.

The results of the skin color detection routine, the face detection routine and the motion detection routine can be weighted and combined using fusion and detection logic (block 216), and a user presence classification is determined (block 218). The fusion and detection logic may include reaching a determination of whether a user exists using neural networks, support vector machines, and / or some other form of probabilistic machine learning based algorithm. 7 illustrates a skin color detection routine (block 206) with a flowchart starting at low light determination (block 204). It should be appreciated that low light judgment can be used in a variety of different ways that affect the processing of an image. For example, in some embodiments, low light judgment may be provided to the neural network as a vector, while in other embodiments low light judgment may be used to select a particular type of classifier to be used. That is, if it is determined that the image is not taken at low light, feature vectors may be generated (block 220) and the first pixel classifier is applied (block 222). If the image was captured at low light, another set of feature vectors can be generated (block 224) and a second per pixel classifier can be applied (block 226). Feature shapes such as color conversion and the like may optionally be provided to achieve a desired result and may vary depending on low light judgment. Also, the first and second pixel-specific classifiers may be different based on the low light determination. For example, the first classifier may be a pixel-by-pixel 7-5-2 multilayer perceptron (MLP) feed forward neural network classifier, while the second classifier is a pixel-by-pixel 2-12-2 MLP feed forward. It may be a neural network classifier. In some embodiments, such a classifier may be implemented with an open kernel with a GPU to help speed up the process.

The output from this classifier may be a probability (eg, a value between 0 and 1) indicating the probability that the image will contain skin color. A morphology filter can optionally be applied to the image (block 228) and an average grayscale level can be calculated (block 230). Also, nonlinear scaling (block 232), temporal queue filter (block 234) and clamp (block 236) may be applied before determining the probability of user presence according to skin color detection (block 238).

8 illustrates a face detection routine (block 210) with a flowchart that begins with applying a face detector (block 240). For example, any suitable face detector may be implemented that provides a probability score indicating the likelihood of a face, such as a Viola-Jones cascade face detector. The face presence score can then be scaled (block 242) and an intermittent detection flicker filter can be selectively applied (block 244) to smooth the image. It should be appreciated that in some embodiments, such as where the camera has relatively good quality, planarization may be omitted in the process. The flicker filter may include a time queue filter (block 246), a determination as to whether the normalized score deviation from the mean is less than the threshold (block 248), and then a multiplication of the output value with the scaled score (block 250). The time cue filter (block 252) and the clamp (block 254) are applied before determining the probability of user presence according to face detection (block 256). The body detector or body sensor may be configured to follow the same flow as or similar to the face detector. In addition, the body detector may be implemented using an image sensor having a lower resolution than that used in the face detector.

9 illustrates the motion detection routine 212 as a flowchart that begins by collecting many frames, and as such, the memory may be implemented to store many frames. For example, three frames may be used, the current frame and two other frames. In the embodiment illustrated in FIG. 9, the current frame and two subsequent frames are used. First, the input frame is sequentially delayed by k frames (blocks 260 and 262) and feedforwarded and added with the output of the second delay (block 262) (block 264). The output of block 260 is multiplied by 2 (block 266) and the difference (block 268) between the adder (block 264) and the output of multiplier (block 266) is determined. Next, an inner product per pixel is determined (block 270), scaled (block 272), and the pixel is clamped (block 274). The average gray level is calculated (block 276), nonlinear scaling is performed (block 278), a time cue filter is applied (block 280), and then clamped to [0, 1] (block 282). Finally, a probability of user presence over the movement is determined (block 284).

Some parameters that can be useful in motion detection routines are auto focus window statistics or horizontal edge or Sobel / Sharr edges, 2D color histogram data, component histogram data, and true color. Automatic white balance / auto exposure (AWB / AE) window statistics on color content, and the like. Some preprocessing steps include the Y channel (intensity), the slope magnitude computed by the Sobel or Shar filter (which accumulates the gradient for proper normalization), and the critical gradient magnitude (normalized by count of edge pixels). ), Skin-probability in chrominance (Cr, Cb) space, sub-images of any of the foregoing, and the like, for motion detection. In some embodiments, motion detection may include the ability to calculate image centroids. The distance from the center of the current frame to the center of the previous frame is used as a measure of the amount of motion, and the hard threshold is applied to generate binary detection or motion. Thus, for example, a change in the Y 'intensity, the edge slope magnitude, the binary edge, or the center position of the skin probability image may mean motion. Tradeoffs in sensitivity and robustness can affect the specific combination of parameters used. For example, skin probability images can be used for edge gradients and binary edges can be used to provide robustness to lighting changes. Skin probabilities may be performed by neural networks as described above or alternatively using an automatic white balance color space filter to approximate this function.

Some embodiments of sensing motion may refer to window statistics of skin detection of gradient images. One embodiment may consider changing the global sum. In particular, the images are summed over the frame to produce a scalar value s [i], where i is the current frame index. The queue of previous N values is kept at S = {s [i-1], s [i-2], ..., s [iN]}. S _{L , N} is represented by a sequence from s [iL] to s [iN], and the extremes of these values are calculated as u = max (S _{L, N} ) and v = min (S _{L , N} ). . The amount of motion is determined by the excursion outside this range, e = max (s [i] −u, vs [i]). Motion is detected when e exceeds a predetermined threshold.

In some embodiments, the motion detection routine may be implemented in a single frame, so less memory may be used or no memory may be used because the entire frame will not be stored. In some embodiments, the image (single frame) may be divided into windows capable of statistical calculation. The statistical change of the window can also be used to determine the motion and the position of the user.

10 illustrates a possible set of windows in which an image can be partitioned for single frame motion detection. Specifically, FIG. 10 shows a single frame divided into non-overlapping windows and concentric windows for statistical purposes. In each case, the luminance of the image (frame at the top of the page) and the intensity gradient magnitude of the image (frame at the bottom of the page) are taken into account. For example, image 300 may be divided into multiple non-overlapping exposure statistics windows 302. Alternatively, image 300 may be divided into multiple concentric overlapping exposure statistics windows 304. Statistics of each window may be determined based on a luminance image (such as in image 300) or based on intensity gradient size image 306.

Using windows provides more robust motion capture when calculating the sum of the gradient magnitudes. For overlapping rectangular windows, one embodiment includes eight rectangles arranged concentrically, with the largest covering the entire frame and the smallest at the center of the image. Thus, in frame i, the sum of the frames is s _j [i] for j∈ [1,2, ..., 8]. The sum of the strips pixels lying between the rectangles is the difference of these sums d _j [i] = s _j [i] -s except for the special case d ₈ [i] = s ₈ [i] / h _{8 It} is calculated as _j +1 [i] / h _j . The difference is normalized to the height of the strip h _j roughly proportional to its area.

Next, the extreme values u, v (maximum value and minimum value) of the difference d _j for N previous frames are calculated using the cue, and the deviation is e _j = max (d _j [i] -u, vd _j [i ])to be. Each deviation e _j is compared with a threshold to provide a motion indicator of the area j of the frame. Minor illumination changes can result in false positive detection. Real motion is generally associated with the detection of two or three of eight areas. Thus, in some embodiments, at least two motion detection regions are needed to determine that motion has been detected in the frame. In addition, large lighting variations, such as turning on or off the lights in a room, often show motion detection in many areas. Therefore, when more than three regions detect the motion, the detection can be suppressed. These design parameters can be adjusted based on experience or to provide the desired sensitivity and toughness levels.

11 is a flowchart illustrating a motion detection routine 212B using a single frame 300 with a non-concentric window 302. In some embodiments, each statistics window may be provided with a unique analysis pipe. That is, each window can be processed at the same time. In another embodiment, one or more windows may be processed sequentially in a common analysis pipe. As used herein, an "analysis pipe" can refer to a processing step associated with statistical analysis of a window and can include a time queue filter.

As described above, image 300 may be divided into statistics windows 302 and average statistics may be calculated for each window (block 310). A time queue filter may be applied to the average statistics (block 312) and a deviation value “e” may be calculated from the short term past behavior (block 314). This deviation value may be compared with a threshold to determine if the deviation value exceeds the threshold (block 316). A normalized coefficient is maintained for each excursion above the threshold (block 318) and if the normalized coefficient exceeds the voting threshold (block 320), it is determined that motion has been detected (block 322). If it is determined that no motion was detected, additional data may be collected from the second sensor (s). In addition, after some determined amount of time has passed, the device may return to a reduced power state and the first sensor (s) may collect data.

In addition, the deviation value may be used to generate summary statistics (block 324). The simplified statistics can be scaled nonlinearly (block 326) and a probability motion score can be provided (block 328). In general, block 322 will provide a binary one or zero output indicating that motion was detected, while block 328 will provide a value between 0 and 1 indicating the likelihood that motion was detected in image 300. .

As can be appreciated, neural networks, support vector machines (SVMs), or other classification systems can be used in each of the aforementioned routines to determine the presence of a user. In addition, the probability value from each routine may be sufficient alone to determine that the user is present, for example, if a value is above a predetermined threshold. Further, in some embodiments, a combination of probabilities from each routine may also be used to determine if a user is present. In some embodiments, the output of certain routines may not be available because their validity may be an issue. For example, the skin color detection routine may be unreliable due to illumination. In addition, the probabilities output from such routines can be combined in a weighted fashion (eg, one probability is given a greater weight based on the probability that the probability is more accurate than the other). Can be.

Embodiments described herein may be implemented to reduce power consumption of computing devices such as notebook computers, desktop computers, and the like. In particular, since such computing devices can provide presence detection even when the device is in a low power state such as hibernate or sleep state, the device can power up when a user exists and power up when the user has left. It may enter a power down or reduced power state. In addition, embodiments may be provided to enhance user experience in handling computing devices by providing, among other things, intuitive power up and power down operations, as well as security functions.

A staged system can be implemented that prevents the use of the main processor and RAM in low power operation. For example, in one embodiment, only a camera, an image signal processing (ISP) device, and an embedded processor capable of calculating the presence value in real time may be implemented in the lowest power state. In the next step, the face detector chip and RAM can be turned on. In subsequent steps, the system processor and other resources may be powered on.

Thus, in the above example for motion detection, since the RAM may not be available, the memory of the previous frame is limited to the statistics calculated by the ISP, and also in the embedded processor register and cache (e.g. 32k). Limited to available space. It should also be appreciated that presence detection information (eg, statistics, images, etc.) is not available outside of the presence detection routine. That is, for example, an image captured for presence detection may not be visible to the user.

In some embodiments, body detection parameters may be used. The body detection parameter may be similar to the face detection parameter but may be based on macro features of the human body rather than a smaller feature that can prove the face. For example, the body detection parameters may relate to arms, legs, torso, heads and the like. Since the body detection parameters are based on larger features, lower resolution cameras can be implemented than those used for face detection and a reasonably good result can be obtained like a body sensor. Further, in some embodiments, the camera can operate in low resolution mode when used to obtain body detection parameters and in high resolution mode when used for other purposes. In practice, the same camera can be used for the purposes of both body detection and face detection parameters. It should be appreciated that the body detection parameter can be used to determine the presence probability in the same manner as the face detection parameter. In addition, the body detection parameter may be a parameter representing a probability that the body is detected. The data used in the body detector may be a depth image, a grayscale image, an RGB image, or any one of the above-described types of images or combined with other types of images.

In some embodiments, many other sensors and inputs may be implemented. Many other sensors and inputs can help reduce false positives and false positives by increasing the accuracy and capabilities of existing sensors. In one embodiment, a Bayesian filter may be implemented. In particular, iterative Bayesian estimation may be performed based on probability data fusion. 12 is an example flowchart illustrating a method 350 of determining the probability that a user is present using multiple simultaneous asynchronous sensors. Near the top of the flowchart is a multiple sensor 352. In particular, such a sensor may include a video camera 354, a 3-D camera 356, a microphone 358, and a keyboard 360. This is just one example and it should be appreciated that more or less, as well as other sensors, may be used in practical implementations. For example, a touch screen, trackpad and mouse can each be used. Each of the sensors 352 may operate simultaneously.

The other sensors 352 may each be configured to capture certain features. That is, each sensor has the ability to provide some useful information to the presence judgment. Combining sensor information provides synergy of information to be more useful and / or more accurate than information from each sensor when the combination of information is used alone. Specifically, data from sensor 352 can be used to determine the user's actual location and speed of the user in 3-D space, in addition to determining the likelihood that the user is present. The obtained data is combined with a priori prediction of the motion model in a robust manner to provide synergy of sensor information. Data fusion optionally fuses model predictions with observed sensor data, weighting each component's contribution to the estimated state (user's position and velocity) in proportion to the quality (quality) of that information. . Thus, if the motion model is accurate and there is noise in the sensor data, then the weighted model can be weighted more than the sensor data, and vice versa. The combination of this model and the observed data is constantly adapted based on the evolving probabilities of the system (eg, the weights produced by likelihood models are constantly updated).

The flowchart is provided with some example features that can be extracted from the data provided by the sensors. In particular, the video camera may extract features such as 3-D blobs, face & body detection, and motion vectors (block 370). The 3-D blob can represent an object within the field of view of the camera. The 3-D camera 356 may be configured to extract face detection, body detection, and motion vectors as well (block 372). The microphone 358 may be configured to extract audio acceleration and novelty detection (block 374). Keyboard 360 may be configured to extract keypress detection (block 376). The audio acceleration feature may use a Doppler effect to determine whether an object is approaching or retreating. In addition, the new detection feature may attempt to find a change in background noise that may indicate the presence of a user. For example, an increase in sound volume may indicate the presence of a user.

Once these features are extracted, the observability is determined (block 178). That is, each of the extracted features may be used to determine the possibility that an object of interest (eg, a user) exists. The observability is then used to update the state representation (block 182). State representations use fusion of data. A state representation (block 184) may be generated. In one embodiment, the state representation may take the form of a histogram. Other embodiments include parametric Gaussian distribution (Kalman filter), Monte-Carlo sample based distribution (sequential Monte-Carlo method / particle filter), hybrid asynchronous multiplexing. Observation particle filters and the like. Status representations can present information about user presence calculations. For example, the status representation can present information regarding the 3-D location of the user (s) and the speed of the user (s). The motion model (block 188) can enter information used in the state representation and can also receive information from the state representation. The motion model is a state evolution expression that models the user's movement. The state representation can be used first of all in statistical calculations (block 186) that can determine the probability of a user's presence. In addition, the statistical calculation may estimate the variance of the user's location and estimate.

The state representation also provides information to the measurement model (block 180) and the measurement model can be provided to the observability judgment (block 178). The measurement model can generally scale each observation of the sensor. In particular, since the relative observation of the sensor may be given some weight, a model may be applied that helps to give greater weight to the observation that is worthy of greater weight. For example, observation of a keypress may be weighted more than observation of a 3-D blob. Also, if the sensors are noisy, they can be given a smaller weight. Each measurement model is used in the observability phase and again used to update the state representation (block 182).

Implementation of presence detection may increase the intelligent operation of the computing device. In other words, the computing device can better predict user interaction and enhance the user experience. For example, the display may not be dimmed while the user is present and may freeze upon departure of the user. In addition, the display output may be scaled and / or changed based on the user's distance from the device. Other intuitive and empirical features, such as tree leaves swaying across the screen to mimic perceived movement, can add to the intelligent and interactive feel for the user.

The foregoing has described some example embodiments of sensing the presence of a user. Although the foregoing description has set forth specific embodiments, those skilled in the art will recognize that changes may be made in form and detail without departing from the spirit and scope of the embodiments. For example, variations on one or more of the algorithms for presence detection may be implemented. In one example, hardware limitations may seek to change the algorithm. Accordingly, it is to be understood that the specific embodiments described herein are examples only and are not intended to limit the scope thereof.

Claims (28)

A method of operating a computing device to provide presence based functionality, the method comprising:
Operating the computing device in a power saving state;
Collecting a first data set from at least one sensor;
Based on the first data set, determining a probability that an object of interest is in proximity to the computing device;
Selectively executing a state change at the computing device based on the portion of the object relative to the computing device;
If the probability exceeds a threshold level, activating at least one second sensor to collect a second data set;
Determining a probability that the object is a person based on the first and second data sets;
If it is determined that the object is a person,
Selectively determining at least one of a location and a distance of the person with respect to the computing device; And
Executing a second state change at the computing device based on at least one of the location and presence of the person with respect to the computing device.
≪ / RTI > The method of claim 1, wherein the at least one sensor and the at least one second sensor comprise the same sensor. 3. The method of claim 2, wherein the sensor operates in a first mode as the at least one sensor and in a second mode as the at least one second sensor. 4. The method of claim 3, wherein the second sensor comprises an image based sensor, the first mode comprises a low resolution mode, and the second mode comprises a high resolution mode. The method of claim 1, wherein changing the state,
Changing a display background / screen saver shift, the shift corresponding to a movement of the user;
Bringing the display into an awake state;
Placing the device in an awake state;
Reducing the number of processes when bringing the device into an awake state;
Maintaining the device in an active state by modifying conventional timeouts when a person is detected;
Making audio steering based on the position of the person;
Making microphone steering based on user location; And
Modifying the user interface to provide a smaller set of user options if the user is far away
≪ / RTI > The device of claim 5, wherein when the device is already awake and detects that the user moves away from the device,
Change the display state to a sleep state, or
To change the device state to sleep
How it is constructed. The method of claim 1, further comprising using the combined sensor data to determine presence using a neural net or a support vector machine. 2. The method of claim 1, further comprising making a hard decision as to whether the object is within a threshold distance of the device. 2. The sensor data of claim 1, wherein the sensor data coupled to the determination of the presence is determined by using skin tone determination, presence determination, and movement determination in a weighted manner to determine the existence. Further comprising using. The method of claim 1, wherein the at least one sensor comprises at least one of a keyboard, a mouse, or a microphone. The method of claim 1, further comprising using combined sensor data to determine presence using the determined distance of the user from the device. The method of claim 1, further comprising determining a number of faces near the device for security, partially powering up or powering up to a secure state and credentials for further access. How security is provided by requesting credentials. The method of claim 1, wherein the at least one sensor comprises a human body sensor. A method of determining if a user is near a computing device,
Capturing an image using an image sensor;
Calculating at least one of a human body detection parameter and a motion detection parameter from the captured image using a processor;
Using at least one of the human body detection parameter and the motion detection parameter, determining a probability of the user's presence;
If the probability indicates the presence of a user, operating a depth estimator to determine a likelihood that the distance of the user from the device is less than a threshold; And
Changing the state of the device if the probability exceeds a threshold
≪ / RTI > 15. The method of claim 14, further comprising calculating a skin color detection parameter and a face detection parameter, wherein both the skin color detection parameter and the face detection parameter are used to determine if a user is present. 16. The method of claim 15,
Calculating each of the skin color detection parameter, the face detection parameter, the human body detection parameter, and the motion parameter; And
Weighting the parameters to each other, wherein the skin color detection parameter is weighted less than the other parameters.
≪ / RTI > The method of claim 14, wherein the motion detection parameter is calculated using a single frame. 18. The method of claim 17,
Dividing the image into windows; And
Calculating statistics for the windows concentrically, wherein changes in at least two windows indicate movement in the image
≪ / RTI > 13. A computing device,
Main processor; And
Presence sensor system connected to the main processor
/ RTI >
The presence sensor system,
At least one image sensor configured to capture an image; And
A processor coupled to the image sensor, the processor configured to process the image to determine a probability that the user is in the vicinity of the device by calculating at least one of a human presence parameter, a motion parameter, a skin color parameter, and a face detection parameter;
Lt; / RTI >
If the probability exceeds a likelihood threshold, an indication is sent from the processor to the main processor that the user has been determined to be present, and the main processor changes the state of the computing system based on the indication. Computing device. 20. The computing device of claim 19, wherein the presence sensor is configured to receive input from at least one additional input device, wherein data received from the at least one additional input device is used in determining the probability. The method of claim 20, wherein the at least one additional input device,
An additional image sensor;
keyboard;
touch screen;
Trackpads;
microphone; And
mouse
Computing device comprising one of the. The computing device of claim 21, wherein one or more of the sensors and input devices operate simultaneously. The computing device of claim 21, wherein at least one of the image sensor and the additional image sensor comprises a depth sensor. The computing device of claim 23, wherein both the image sensor and the phase-addition image sensor operate simultaneously. The computing device of claim 21, wherein the state of the device is changed based only on data from one of the sensors. The computing device of claim 25, wherein the changing of the state comprises entering a warm-up state. 20. The computing device of claim 19, wherein changing the state of the device includes maintaining the device in an active mode when a user is detected. 28. The method of claim 27, wherein changing the state comprises disabling one of a timeout for sleeping or screen dimming of the device when no other input is detected. Computing device.

Images