KR20100098726A - Apparatus and methods for a touch user interface using an image sensor - Google Patents

Abstract

An apparatus and method for a touch user interface using an image sensor is presented. A method for processing image-based input commands for a user interface includes receiving image frames from a sensor; Determining when the sensor enters a cover state; Determining from the subsequent image frames when the sensor enters a de-cover state; Analyzing information based on subsequent image frames to interpret a user command; And issuing a user command to the user interface. A device having an image-based user interface includes a processor coupled to an image sensor and a memory, the processor receiving image frames from the image sensor, determining when the image sensor enters a cover state, and subsequent image frames. Determine when the image sensor enters the de-cover state, analyze information based on subsequent image frames to interpret the user command, and have logic to issue the user command to the user interface. It is configured to.

Description

Apparatus and method for a touch user interface using an image sensor {APPARATUS AND METHODS FOR A TOUCH USER INTERFACE USING AN IMAGE SENSOR}

Embodiments of the present invention generally relate to image sensors based on interfaces, in particular mobile devices having interfaces that use an image sensor to receive user commands.

As mobile devices become more powerful and sophisticated, user interface developers face the challenge of expanding the capabilities of mobile devices while improving the ease of use of the mobile devices in developing the interface.

Touch screens are becoming increasingly popular as user interfaces for mobile devices due to the recent evolution of multi-touch functionality and their intuitive approach that simplifies complex user interface navigation. Touch screens can also have the advantage of maximizing the screen size of the mobile device because the actual keyboards and / or other physical cursor control interfaces can be omitted. However, touch screens have a number of limitations, such as tactile feedback and other controls of virtual keyboards, screen occlusion by the user's finger and / or smudging of the display surface during use. May be associated with operational drawbacks. Moreover, touch screen displays are typically more expensive to develop and manufacture than non-touch screens.

Because of the aforementioned disadvantages of touch-screen displays, some users prefer to use a physical keypad with a smaller display of their mobile devices. In connection with these physical user interfaces, other conventional methods have proposed introducing an intuitive nature of touch screen capabilities for use in existing and future mobile devices. These methods can affect integrated digital cameras that are common to many mobile devices.

Some of these conventional methods have suggested using MPEG motion vector algorithms to determine how the user should move his hand in front of the camera. Other systems may use the integrated camera to estimate the direction of the mobile device (eg, tilt) to determine user input. These methods may include algorithms that operate in real time so that the user interface is fully responsive. Thus, these methods may be computationally intensive, overload the on-board processor (s) of the mobile device, and / or may use specialized hardware. Thus, conventional methods can adversely affect the cost and increase the power consumption of the mobile device.

Moreover, these conventional methods may cause the user to move the arm and / or hand too far in front of the camera, which may cause the user to inappropriately attract attention and / or cause fatigue over time. have. In addition, these algorithms determine how to specify selection points and / or target objects in the user interface at a distance exceeding a single user movement (eg, to perform a relative navigation task). Problems such as resetting selection points when sliding / dragging or the like. Moreover, these techniques may require a user to keep his or her hands stable or stationary to properly make selections and / or prevent unintentional selection of an item.

Therefore, it would be desirable to provide a touch user interface navigation technique for existing and future camera phones, which avoids the aforementioned disadvantages and can be implemented in a cost-effective manner.

Exemplary embodiments of the invention relate to apparatus and methods for a touch user interface using an image sensor.

In one embodiment, a method for processing image-based input commands for a user interface is presented. The method includes receiving image frames from a sensor; Determining when the sensor enters a cover state; Determining from the subsequent image frames when the sensor enters a de-cover state; Analyzing information based on subsequent image frames to interpret a user command; And issuing a user command to the user interface.

In another embodiment, an apparatus with an image-based user interface is presented. The apparatus may include an image sensor and a processor coupled to the memory; The processor receives image frames from an image sensor, determines when the image sensor enters the cover state, determines when the image sensor enters the de-cover state from subsequent image frames, and interprets user commands. Analyze information based on subsequent image frames, and have logic to issue a user command to the user interface.

Yet another embodiment of the present invention may include a mobile device having an image-based touch user interface, which includes a camera and a processor coupled to a memory. The processor receives image frames from the camera, subdivides the image frame into tiles, calculates a metric for each tile, performs a count of tiles with a predetermined value for the metric, Determine the de-cover map based on the trail values from subsequent image files, calculate the slope of the de-cover map, determine the direction of movement based on the slope, and command the user interface based on the direction. Contains logic configured to issue.

Yet another embodiment of the present invention may include an apparatus for processing image-based input commands for a user interface, the apparatus comprising: means for receiving image frames from a sensor; Means for determining when the sensor enters a cover state; Means for determining from the subsequent image frames when the sensor enters the de-cover state; Means for analyzing information based on subsequent image frames to interpret a user command; And means for issuing a user command to the user interface.

Another embodiment of the present invention may include a computer-readable medium having stored thereon program code that when executed by a machine causes the machine to perform operations for processing image-based input instructions for a user interface. . The computer-readable medium may include program code for receiving image frames from a sensor; Program code for determining when the sensor enters a cover state; Program code for determining from the subsequent image frames when the sensor enters the de-cover state; Program code for analyzing information based on subsequent image frames to interpret a user command; And program code for issuing a user command to the user interface.

The accompanying drawings are presented to help explain the embodiments of the invention, and are provided merely for the purpose of illustration and do not limit the invention.

1A-1D are schematic diagrams illustrating the operation of an exemplary mobile device having an image-based touch user interface.
2 is a block diagram illustrating an exemplary configuration of a mobile device with an image-based touch user interface.
3 is a flow diagram illustrating an example high-level process associated with an image-based touch user interface.
4 is a flow diagram illustrating an example process for determining a cover state associated with an image-based touch user interface.
5 is a flow diagram illustrating an example process for determining a de-cover state associated with an image-based touch user interface.
6 is a flow diagram illustrating an example process for determining user commands associated with an image-based touch user interface.

Aspects of the present invention are disclosed in the following detailed description and associated drawings of specific embodiments of the present invention. Alternative embodiments may be invented without departing from the scope of the present invention. In addition, well-known elements of the invention will not be described in detail or will be omitted so as not to obscure the relevant details of the invention.

The term " exemplary " is used herein to mean " serving as an example, example, or illustration. &Quot; Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments. Likewise, the term "embodiments of the invention" does not require that all embodiments of the invention include all of the features, advantages or operations discussed.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of embodiments of the invention. As used herein, the singular is intended to include the plural unless the context clearly dictates otherwise. The terms “comprise” and “include”, when used herein, specify the presence of one or more of the features, integers, steps, actions, elements and / or components mentioned. It will be further understood that it does not exclude the presence or addition of other features, integers, steps, actions, elements, components, and / or groups thereof.

In addition, many embodiments are described by sequences of operations to be performed by, for example, elements of a computing device. It will be appreciated that the various operations described herein may be performed by specific circuits (eg, application specific integrated circuits (ASICs)), by program instructions executed by one or more processors, or by a combination thereof. will be. In addition, the sequences of these operations described herein may be fully implemented in any form of computer readable storage media having stored thereon a corresponding set of computer instructions that, when executed, cause the associated processor to perform the functions described herein. May be considered. Accordingly, various aspects of the invention may be embodied in many different forms, all of which are considered to be within the scope of the claimed subject matter. Moreover, for each of the embodiments described herein, the corresponding form of any such embodiments may be described herein as, for example, logic configured to perform the described operation.

1A-1D are diagrams illustrating an overview of the operation of an exemplary mobile device 100 with an image-based touch user interface (IBTUI). 1A shows an exemplary mobile device 100 as a flip-phone (shown as the top of the phone is cut off). The back side 105 of the device includes an output of an image sensor 110 capable of continuously collecting image frames 115. In operation, the IBTUI may track the path of an object exiting the field of view of the sensor 110 after the sensor is covered. Once tracking is completed, the command can be interpreted by the IBTUI based on the nature of the tracked movement.

As shown in FIG. 1B, a user may initiate a command entry by first placing finger 120 over image sensor 110. This operation may substantially or completely cover image sensor 110 to produce one or more image frames 125 with low luminance values. This makes the IBTUI "cover state" and signals the IBTUI to track the movement of the finger 120 when the finger 120 leaves the field of view of the image sensor 110. As shown in FIG. 1C, the finger 120 is leaving the field of view of the image sensor 110 by moving to the left side of the page. A series of image frames 130 may be generated with luminance variations corresponding to this movement. Image frames may be processed to interpret movement as a command. In this example, the movement of the finger may be interpreted as a command to generate a corresponding movement of the cursor in the graphical user interface (GUI) of the mobile device. FIG. 1D shows a series of image frames 135 created with a user's finger 120 moving toward the bottom of the page and corresponding luminance variations. This movement may generate a command to move the cursor downward in the GUI of the mobile device. As described below, other movements and / or gestures may be interpreted as different commands.

While mobile device 100 is shown as a camera flip-phone, other embodiments of the present invention may be associated with any type of devices, as described in more detail below.

2 is a block diagram illustrating an exemplary configuration 210 of a mobile device 200 with an image-based touch user interface (IBTUI). Mobile device 200 may have a platform 210 that can exchange data and / or commands over a network. The platform 210 may further include a transceiver 215 (transmitter and and receiver) operably connected to the processor 220, or other controller, microprocessor, ASIC, logic circuit, or any other data processing device. , They are not explicitly shown). The processor 220 may execute programs stored in the memory 225 of the mobile device 200. One program executable on the processor 220 may be associated with an image-based touch user interface that may provide inputs to the graphical user interface of the mobile device 200. The memory 225 may include executable modules (eg, IBTUI, GUI, etc.), image frames, and other data structures including those associated with the operation of the IBTUI. Memory 225 may include read-only and / or random-access memory (RAM and ROM), EEPROM, flash cards, or any memory common to these platforms. The image sensor 230 may be functionally connected to the processor 220, and typically may react sensitively to visible light. Other embodiments of the present invention can feature an image sensor 230 that can use other wavelengths, so that the IBTUI can operate when there is no visible light. The optical components of the image sensor associated with the outer surface of the mobile device 200 (eg, a clear cover protecting the camera lens) may be mounted in a recessed manner. In such an arrangement, the user's finger may not actually be in physical contact with the image sensor, thus preventing the user's finger from introducing foreign objects (eg, dust, oil, etc.) into the optical path of the image sensor, or Avoid damaging the image sensor (eg, scratching). Thus, the image-based touch user interface does not require actual contact of the image sensor.

Image sensor 230 may be a camera that records image frames at a periodic rate (eg, 30 frames / second). When accepting user input via the IBTUI, the image sensor 230 may continuously provide image frames for IBTUI processing. For example, image sensor 230 may inform processor 220 when a user displays a “contact” screen to accept input from a user's finger for use in movement of the cursor and / or selection within the screen. Image frames may be provided. When the image sensor 230 does not provide image frames to the IBTUI, the image sensor can be used to provide pictures and / or videos. In addition, the image sensor 230 can collect any conventional unprocessed image frames associated with improving the aesthetic qualities of the image frames and the IBTUI can utilize these image frames. For example, when an image sensor is used for IBTUI, image frames may not be subjected to any white balance, color balance, auto-focus, image sharpening, etc. Can be. If this process is omitted, the computational burden imposed on the mobile device 100 when using the IBTUI will be reduced, further improving the life of the battery.

Various logical elements for providing instructions may be implemented in individual elements, software modules executed on a processor, or any combination of software and hardware for executing the functions described herein. For example, the processor 220 and the memory 225 may both be used interacting to load, store, and execute the various functions described herein, such that logic to perform these functions may be distributed on the various elements. Can be. In the alternative, the functionality may be integrated into one individual component (eg, internal memory of the processor 220). Thus, the features of the mobile device 200 of FIG. 2 will be considered merely illustrative, and embodiments of the invention are not limited to the illustrated features or arrangement.

Moreover, embodiments of the present invention may be used in connection with any device, and are not limited to the illustrated embodiments. For example, the devices may include cell phones, access terminals, personal digital assistants, music players, radios, GPS receivers, laptop computers, kiosks, and the like.

3 is a flow diagram illustrating an example top-level process 300 associated with an image-based touch user interface (IBTUI). Process 300 may begin when mobile device 200 is initially turned on or power cycled, and processor 220 initiates initialization of various processes for device operation. (Block 310). This may include initialization of the hardware and software / firmware / logic components and graphical user interface associated with receiving and processing image frames from image sensor 230. Image frames may be provided in a conventional video format (e.g., 30 frames per second, where each frame has 240 x 320 pixels), and may employ a luminance-chrominance color space (YCrCb). Can be used. Frames may also be provided in a quasi-video format with reduced frame rate and / or low spatial sampling within each image frame. Additionally, image frames may not be pre-processed to enhance color, white balance, sharpness and / or improve other aesthetic qualities.

Next, the process may begin analyzing the images generated by the image sensor 230 to determine if the image sensor is in a cover state. As defined herein, the cover state occurs when the image sensor 230 is covered by an object (typically a user's finger). This analysis may be performed on the luminance channel of the image frames, and may include calculating one or more metrics based on average brightness and detail (block 315). The metrics may be statistical in nature and will be described in more detail below. Next, the process may perform a determination as to whether the image sensor is in the cover state by performing threshold comparison using the metrics calculated at block 315 (block 320). If it is determined that the image sensor 230 is in the cover state, the process proceeds to block 325, otherwise the analysis at block 315 continues until the cover state is reached. Details of blocks 315 and 320 are presented in the description of FIG. 4 below.

When it is determined that the image sensor has entered the cover state, process 300 begins analyzing subsequent image frames to determine when the image sensor transitions to a "de-cover state" (blocks 325 and 330). . As used herein, the de-cover state is defined as the state when the user's finger leaves the image sensor to the extent that the movement of the user's finger can be reliably tracked. This can be determined by calculating luminance and / or detail metrics and their changes over time. During this process, the de-cover map can be generated to store the calculated metrics and their temporal changes. Once the de-cover map is complete, the process may proceed to block 335 where the de-cover map is analyzed. Details of blocks 325 and 330 are presented in the description of FIG. 5 below.

At block 335, the de-cover map is analyzed to determine how the finger left from the image sensor 230. By analyzing the spatial changes in the de-cover map, the direction of finger movement can be determined (block 335). This information may be used to interpret commands that may be provided to the graphical user interface of the mobile device. Details of blocks 335 and 340 are presented in the description of FIG. 6 below.

Thus, embodiments of the present invention may include a method 300 for processing image-based input commands for a user interface. The method receives image frames from a sensor and determines when the sensor enters a cover state (blocks 315, 320), and determines from when subsequent sensors enter the de-cover state (blocks 315, 320). 325, 330, analyzing information based on subsequent image frames to interpret the user command (blocks 335, 330), and issuing the user command to the user interface (block 340).

Moreover, another embodiment of the present invention may include a device 200 having an image-based user interface. The device may include a processor 220 coupled to the image sensor 230 and the memory 225. The processor receives image frames from the sensor and determines when the sensor enters the cover state (blocks 315, 320) and determines when the sensor enters the de-cover state from subsequent image frames (blocks 325, 330). ) May be configured to parse information based on subsequent image frames to interpret the user command (blocks 335, 330), and to have logic to issue the user command to the user interface (block 340).

4 is a flow diagram illustrating an example process 400 for determining a cover state associated with an image-based touch user interface. Process 400 may begin by receiving image frame i from image sensor 200 (block 410). The image frame may be subdivided into n × m tiles (eg, for a 240 × 320 longitudinal preview frame, each tile may include approximately 60 × 80 pixels, where n = m = 4). ). Next, the luminance and / or detail metrics for each tile for the pixels from the luminance channel of the image may be calculated by the processor 220 (block 420). The luminance metric for each tile can be calculated by determining the average luminance in each tile. The detail metric can be calculated by determining the standard deviation of each tile. The standard deviation std may be approximated by the following equation for high speed execution by the processor 220.

Where val are intensity values that can be taken on 8-bit pixels;

hist (val) is a histogram of luminance values, that is, the number of pixels of a tile with luminance of val;

avg is a previously calculated average luminance value.

The above formula assumes that luminance pixels are stored using 8-bit integers, but it should be noted that embodiments of the present invention are not limited to the above-described formulas or data types. .

Once the luminance and / or detail metrics are calculated for all the tiles of image frame i, process 400 can proceed by counting the number of tiles that exceed a predetermined threshold (s) (block 425). For example, in one embodiment, the number of tiles with an average value lower than 30 and the std value lower than 100 may be used to set the count. This count number is then tested to determine if it exceeds the threshold (block 430). The threshold is predetermined and may be set to a portion of the total number of tiles of the image frame (eg, the predetermined threshold number may be set to .95 * n * m). If the count number does not exceed the threshold, the frame count is incremented and the next image frame is received for cover state determination processing (blocks 435, 410). Once it is determined that the count number exceeds a predetermined threshold number, the image sensor 230 is determined to be in a cover state. The process may then proceed to the de-cover process 500 as described below.

5 is a flow diagram illustrating an example process 500 for determining a de-cover state associated with an image-based touch user interface. Initially, metrics of tile values corresponding to an image frame when a cover condition is detected are stored (block 510). For example, the average brightness and std for each tile can be stored. These metrics can be stored in the data structure referred to herein as reference tile metrics, where the table can take the form of a multi-dimensional matrix.

The image frame is then received from image sensor 230 and subdivided into n × m tiles as described above (blocks 515 and 520). Next, process 500 may calculate luminance and / or detail metrics for each tile in the manner described below with respect to cover state determination (block 525). Once the processor 220 calculates the metric (s) for each tile, each tile is examined and the trail value exceeds the metric of the reference tile to which the metric of the current tile corresponds by a predetermined amount. Can be assigned to each tile. This comparison process may be performed between each tile of current image frame j and previously stored reference tile metrics associated with image frame i. This comparison operation may be based on predetermined thresholds. For example, the trail value for a tile may be such that when the average luminance of a given tile exceeds the average luminance of the corresponding reference tile by 30 levels and / or the std of the given tile exceeds the std of the reference tile by 90 levels. Can be calculated when

Each trail value is associated with one tile and thus can be stored in an n × m data structure. The trail value can be calculated using the following formula.

here,

j is the current number of frames corresponding to time,

avg (x, y) is the average luminance value of the tiles of position x, y in frame j,

refTile (x, y) is the average luminance value of the reference tile at position x, y,

T is a threshold (eg, 30).

In general, the trail value indicates when a particular tile is not covered. The larger the trail value, the later the time that a particular tile is not covered. However, the trail values include information about the amount of time and luminance obtained to break the ties about when the various tiles are not covered. The temporal component of the trail value may be encoded by the frame number j and may only have integer values. In order to provide greater granularity to the temporal components of the trail values, temporal information j can be modified by the difference between the average brightness of the current tile and its corresponding reference. If this difference is large, it means that the tile was not covered sooner and therefore an amount is inferred from the time information (this amount is a scaled difference). Each trail value may be stored in a two-dimensional structure called a de-cover map. The de-cover map may have n × m entries, each entry corresponding to a tile position in the current image frame j. For example, a tile with a trail value of 292 may not be covered after a tile with trail 192 (tiles in the range of 200 are de-covered after tiles in the range of 100). Ties between tiles not covered during the same image frame may be divided based on the obtained light level. For example, a tile with a trail of 299 (frame j = 3, still dark) is after a tile with trail 292 (much brighter than frame j = 3 or its stored covered luma value). Assume that it is de-covered.

Once the trail value is calculated for each tile position (each x, y on n, m tiles), the de-cover map is complete. A test can be performed to determine if a predetermined number of tiles have an associated trail value that can be associated with the de-cover state (block 535). The predetermined number of tiles may be, for example, the majority of tiles in the frame. If the decision is true, the process continues to process 600. Otherwise, a test is performed to determine if the threshold number of frames have been processed (blocks 540 and 545). If the number of frames exceeds the threshold, this means that the user is holding a finger on the image sensor and the command can be interpreted as "select" or "input" (eg, the number of frames passed). May be set to correspond to two second periods of time). The selection / entry commands may be similar to a key press or mouse click of an “input” key on the keyboard, which may be used to enter a value or select an object in the GUI of the mobile device 200.

6 is a flow diagram illustrating an example process 600 for determining user commands associated with an image-based touch user interface. The de-cover map can be thought of as a 3-D surface, where the x, y indices are tile positions, and the z values are tile values corresponding to the time of finger movement when the finger de-covers the lens. At block 610, the gradient of the de-cover map may be calculated to determine the steepest ascent of the de-cover map. The slope and tail values in the de-cover map may be compared to thresholds as a secondary or alternative way to determine if the user wants to enter a “select / enter” command in the GUI. (Blocks 615 and 630). In this example, if the trail values are high and the slope is low (which indicates that the trail values are constant), the user can keep the finger on the image sensor to indicate a "select / enter" command.

From the slope of the de-cover map, the direction of finger movement can be determined in a number of different ways (block 620). For example, processor 220 may find which n rows and m columns are strongest in the n × m de-cover map. For example, the following de-cover map is as follows.

The largest column is the last column and the largest row is the first row, so the algorithm will instruct the system to move the cursor to the upper right because its direction has the highest density of trails. Once the direction of movement is determined, a command can be issued to the user interface indicating the direction. Typically, these instructions will be made in a format that device drivers of the mobile device 200 can readily accommodate.

In other embodiments, unique user gestures may be provided to the IBTUI for other commands. For example, multiple finger movements may be used to control unique features of mobile device 200 or may be used as shortcuts to instructions that may take multiple steps. For example, two downward movements of a finger passing over the image sensor may be used to control the parameters of the image sensor for taking pictures in certain situations (eg, set short exposures for scanning barcodes) (ie, automatic -To control focus, auto-exposure, hand-jitter reduction, and the like.

As mentioned above, the image sensor 200 may be sensitive to wavelengths other than those corresponding to visible light. For example, the image sensor 200 can be sensitive to infrared radiation, so it can be used in low light situations. Such embodiments may utilize a sensor having the ability to disengage the IR blocking filter. Other sensors may utilize an IR radiation source (eg, an IR LED) that may be activated when the amount of visible light is below the usable threshold.

Moreover, the de-cover map can be extended to multi-dimensional structures with one dimension corresponding to time. These data structures can be visualized as three-dimensional rectangles, where the xy dimensions correspond to position and the z dimension corresponds to time. In this data "volume", each data element may correspond to a trail value at point x, y of the image frame at time t _j .

Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the preceding description may include voltages, currents, electromagnetic waves, magnetic fields or particles, light fields or particles, or any combination thereof. It can be expressed as.

In addition, those skilled in the art will recognize that various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments presented herein may be implemented as electronic hardware, computer software, or a combination thereof. will be. To clarify the interoperability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of embodiments of the present invention.

The methods, sequences and / or algorithms described in connection with the embodiments presented herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination thereof. Software modules include a RAM memory; Flash memory; ROM memory; EPROM memory; EEPROM memory; Registers; Hard disk; Portable disk; CD-ROM; Or may reside in any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, such that the processor can read information from and write information to the storage medium. In the alternative, the storage medium may be integral to the processor.

Thus, embodiments of the present invention may include a computer readable medium that implements a method for an image-based touch user interface in accordance with the functions, steps, and / or operations described herein. Accordingly, embodiments of the present invention may include a computer-readable medium having stored thereon program code that when executed by a machine causes the machine to perform operations for processing image-based input instructions for the user interface. The computer-readable medium may include program code for receiving image frames from a sensor; Program code for determining when the sensor enters a cover state; Program code for determining from the subsequent image frames when the sensor enters the de-cover state; Program code for analyzing information based on subsequent image frames to interpret a user command; And program code for issuing a user command to the user interface. Accordingly, the present invention is not limited to the illustrated examples, and any means for performing the functions described herein are included in the embodiments of the present invention.

While the foregoing description describes exemplary embodiments of the invention, it should be noted that various changes and modifications may be made herein without departing from the scope of the invention as defined by the appended claims. The functions, steps and / or actions of the method claims in accordance with the embodiments of the invention presented herein need not be performed in any particular order. In addition, although elements of the invention may be described or claimed in the singular, the plural should also be considered unless the limitations on the singular are expressly stated.

Claims (33)

A method for processing image-based input commands for a user interface, the method comprising:
Receiving image frames from a sensor;
Determining when the sensor enters a cover state;
Determining from the subsequent image frames when the sensor enters a de-cover state;
Analyzing information based on the subsequent image frames to interpret a user command; And
Issuing the user command to a user interface;
Method for processing image-based input commands. 2. The method of claim 1, further comprising: subdividing the image frame into tiles;
Calculating a metric for each tile; And
And performing a count of tiles having a predetermined value for the metric. 3. The method of claim 2, further comprising performing the method of claim 2 on subsequently received frames until the count exceeds a predetermined number. 4. The method of claim 3, wherein when the count exceeds a predetermined number,
Storing a reference metric for each tile;
Subdividing subsequent frames into tiles;
Calculating a metric for each tile; And
Calculating at least one trail value for tiles with metrics that exceed predetermined values. 5. The method of claim 4, further comprising performing the method of claim 4 on subsequently received frames until all tiles have a corresponding trail value. 6. The method of claim 5, further comprising interpreting the user command as "select" or "enter" if a predetermined number of image frames have been processed before all of the tiles have corresponding trail values. And a method for processing image-based input instructions. The method of claim 1,
Calculating a gradient of the de-cover map;
Determining a moving direction based on the slope; And
Issuing a command to the user interface based on the direction. 8. The method of claim 7, further comprising: determining whether the slope exceeds a predetermined value;
Determining whether a predetermined number of trail values exceed a predetermined value; And
Interpreting the user command as “select” or “input” based on the slope and trail determination. 3. The method of claim 2, wherein the metric comprises an average of luminance and a standard deviation of the luminance. The method of claim 1, wherein the sensor is a camera. The method of claim 10, wherein the user command is input by placing a finger over the camera. 12. The method of claim 11, wherein the series of gestures is interpreted as a command associated with control parameters of the camera. The method of claim 1, wherein the image frames received from the sensor are substantially based on infrared radiation. A device having an image-based user interface,
An image sensor, and
A processor coupled to the memory;
The processor comprising:
Receiving image frames from the image sensor,
Determine when the image sensor enters the cover state,
Determines when the image sensor enters the de-cover state from subsequent image frames,
Analyze information based on the subsequent image frames to interpret a user command, and
Configured to have logic to issue the user command to a user interface,
Device with an image-based user interface. The method of claim 14, wherein the processor,
Subdivide the image frame into tiles,
Calculate a metric for each tile, and
And further configured to have logic to perform a count of tiles with a predetermined value for the metric. 16. The image-based user interface of claim 15, wherein the processor is further configured to have logic to perform the logic of claim 15 on subsequent received frames until the count exceeds a predetermined number. Device. The method of claim 16, wherein the processor,
Save the baseline metrics for each tile,
Subdivide subsequent frames into tiles,
Calculate a metric for each tile, and
And further configured to have logic to calculate at least one trail value for tiles with metrics that exceed predetermined values. 18. The image-based user interface of claim 17, wherein the processor is further configured to have logic to perform the logic of claim 4 on subsequently received frames until all tiles have a corresponding trail value. Device. 19. The processor of claim 18, wherein the processor is further configured to have logic to interpret the user command as "select" or "input" if a predetermined number of image frames are processed before all of the tiles have a corresponding trail value. Apparatus having an image-based user interface configured. The method of claim 14, wherein the processor,
Calculates the slope of the de-cover map,
Determine a direction of movement based on the slope, and
And further have logic for issuing a command to the user interface based on the orientation. The processor of claim 20, wherein the processor comprises:
Determine whether the slope exceeds a predetermined value,
Determine whether a predetermined number of trail values exceed a predetermined value, and
And further configured to have logic to interpret the user command as “select” or “input” based on the slope and trail determination. 16. The apparatus of claim 15, wherein the metric comprises an average of luminance and a standard deviation of the luminance. 15. The apparatus of claim 14, wherein the sensor is a camera and the user command is input by placing a finger over the camera. 24. The device of claim 23, wherein the camera is recessed from the body of the device such that the finger is not in physical contact with the camera. A mobile device having an image-based touch user interface,
Camera, and
A processor coupled to the memory;
The processor comprising:
Receiving image frames from the camera,
Subdividing the image frame into tiles,
Calculate a metric for each tile,
Perform a count of tiles with a predetermined value for the metric,
Determine a de-cover map based on trail values from subsequent image files,
Calculates the slope of the de-cover map,
Determine a direction of movement based on the slope, and
Logic configured to issue a command to the user interface based on the direction;
Mobile device with image-based touch user interface. An apparatus for processing image-based input commands for a user interface, comprising:
Means for receiving image frames from a sensor;
Means for determining when the sensor enters a covered state;
Means for determining from the subsequent image frames when the sensor enters a de-cover state;
Means for analyzing information based on the subsequent image frames to interpret a user command; And
Means for issuing the user command to a user interface,
Apparatus for processing image-based input commands. 27. The apparatus of claim 26, further comprising: means for subdividing an image frame into tiles;
Means for calculating a metric for each tile; And
Means for performing a count of tiles with a predetermined value for the metric. 28. The apparatus of claim 27, further comprising means for processing subsequently received frames until the count exceeds a predetermined number. The method of claim 28, wherein when the count exceeds the predetermined number,
Means for storing a reference metric for each tile;
Means for subdividing the subsequent frames into tiles;
Means for calculating a metric for each tile; And
And means for calculating at least one trail value for tiles with metrics that exceed predetermined values. A computer-readable medium having stored thereon program code that, when executed by a machine, causes the machine to perform operations for processing image-based input instructions for a user interface.
The computer-readable medium may include
Program code for receiving image frames from a sensor;
Program code for determining when the sensor enters a cover state;
Program code for determining from the subsequent image frames when the sensor enters the de-cover state;
Program code for analyzing information based on the subsequent image frames to interpret a user command; And
Program code for issuing said user command to a user interface,
Computer-readable media. 31. The apparatus of claim 30, further comprising: program code for subdividing the image frame into tiles;
Program code for calculating a metric for each tile; And
And program code for performing a count of tiles having a predetermined value for the metric. 32. The computer-readable medium of claim 31, further comprising program code for processing subsequently received frames until the count exceeds a predetermined number. 33. The apparatus of claim 32, wherein when the count exceeds the predetermined number,
Program code for storing a reference metric for each tile;
Program code for subdividing the subsequent frames into tiles;
Program code for calculating a metric for each tile; And
And program code for calculating at least one trail value for tiles with metrics that exceed predetermined values.

Images