KR20140026898A - System and method for a virtual keyboard - Google Patents

Abstract

The present invention relates to a method for capturing an image of a work area including a display of a virtual keyboard having a plurality of keys and detecting a key selected among the plurality of the keys. Detecting the selected key includes analyzing a variation of the optical intensity in the captured image and detecting objects within the work area. Detecting the objects includes generating a first line histogram based on the optical intensity of each line of each captured image, generating a first row histogram based on the optical intensity of each row of each captured image and identifying a key selected on the basis of the first line histogram and the first row histogram. The method includes outputting an identifier of the selected key to a host device.

Description

SYSTEM AND METHOD FOR A VIRTUAL KEYBOARD}

The present disclosure is directed to a system and method for optically detecting the presence of an object or objects, such as one or more fingers, in a detection area for performing text input, such as a keyboard.

Currently, small, light handheld devices such as smartphones or devices with touch panels have become widespread. For these portable devices, there are some easy ways for entering text and characters. Some devices include a small built-in or onboard mechanical keyboard or a small printed virtual keyboard on the touch panel for text entry. Some of the touch panels are included in a smartphone, and thus are small in size. Other touch panels are included in larger devices, such as the iPad or Kindle, to use a printed keyboard on the larger touch panel. A touch panel is an electronic virtual display capable of detecting the presence and position of a touch within a display area or screen. Touch panels are commonly used in devices such as game consoles, integrated computers, tablet computers, and smartphones.

These keyboards are sufficient to enter a small number of characters, such as text messages, but not enough to enter long text, such as more than 200 words. In some cases, especially for older people, the virtual and manual keyboard of a smartphone is too small to type a single character. As a result, users of such keyboards generate many typos. On the other hand, a virtual keyboard for a touch panel device can occupy almost half of the screen just to achieve a standard key size.

Virtual keyboards have been proposed for detecting textual input. For example, US Pat. No. 7,215,327, issued to Liu et al. Under the name "Device and Method for Generating a Virtual Keyboard / Display," describes two methods for scanning the working surface to detect the position and movement of a user's finger. Use a scanning laser. The device detects the position of the finger by comparing the difference in time received from the two reflected scanning lasers.

Another example is U.S. Pat.No. 6,710,700 issued to Tomasi et al. Under the name "Quasi-three-dimensional Method and Apparatus to Detect and Localize Interaction of User-Object and Virtual Transfer Device". The virtual input using two optical systems is described. The first optical system OS1 projects light rays parallel to the working surface, and the second optical system OS2 receives and detects the light reflected from the first optical system OS1. Using homography, the actual position of an object in the real world can be calculated from the image pixel coordinates. In order to implement the virtual keyboard, the viewing angle of the second optical system OS2 must be 90 degrees or larger. At such large viewing angles, the lens distortion is so large that homography transformations can cause large errors in calculating the real world position of the object. As a result, problems may arise in selecting the correct key.

It is an object of the present invention to provide a method and system for capturing an image of a work area comprising an indication of a virtual keyboard having a plurality of keys and detecting a selected key among the plurality of keys.

The present disclosure relates to a full-size, virtually weightless virtual keyboard that does not occupy a screen or touch panel and thus does not obscure information on the screen or touch panel.

The virtual keyboard is part of a system that includes a processor, a light source, and an image sensor having a memory configured to generate a keyboard pattern on a work surface. The keyboard pattern can be printed onto a portable device or optically projected onto a work surface. The printed portable keyboard pattern may be a piece of paper, a pre-drawn on a work surface, or a keyboard image displayed on a screen, such as a separate touch panel device. Optically projecting keyboard patterns can be accomplished using various techniques such as diffeactive optical elements (DOE) or digital light processing (DLP). Alternatively, the keyboard may simply be a photo or other physical representation of the keyboard to help guide the user to the appropriate key location.

The present disclosure relates to a method of capturing an image of a work area, the work area comprising a display of a virtual keyboard having a plurality of keys, the image comprising rows and columns of pixels. The method includes detecting a key selected from the plurality of keys, the method comprising detecting a change in intensity of light in the captured image to detect an object in the work area. Detecting the object comprises generating a first row histogram based on the intensity of light in each row of each captured image, generating a first column histogram based on the intensity of light in each column of each captured image, Identifying the selected key based on the first row histogram and the first column histogram. In addition, the method includes outputting an identifier of the selected key to the host device.

According to the present invention, a method and system for capturing an image of a work area including a display of a virtual keyboard having a plurality of keys and detecting a selected key among the plurality of keys can be implemented.

1A is a side view of a virtual keyboard system in accordance with an aspect of the present disclosure.
FIG. 1B is an isometric view of the virtual keyboard system shown in FIG. 1A.
2A-2F, 3A-3D, 4A-4C are images and histograms of fingers in a region of interest according to one embodiment of the disclosure.
5 is a portion of a keyboard pattern related to the images of FIGS. 2A-2F, 3A-3D, 4A-4C.
6 is a schematic diagram of a virtual keyboard system having an embedded processor according to one embodiment of the disclosure.
7 is a block diagram of one method performed by a virtual keyboard system according to an embodiment of the present disclosure.
8 is a flowchart of a method of a virtual keyboard system for performing a finger tip detection routine in accordance with an embodiment of the present disclosure.
9 is a flow chart of a method of a virtual keyboard system for performing a touch detection routine according to one embodiment of the disclosure.
10 is a flow chart of a method of a virtual keyboard system for performing a touch position calculation routine according to another embodiment of the disclosure.
11 is a flow chart of a method of a virtual keyboard system for performing a key blocking detection routine according to one embodiment of the disclosure.
12 is a flowchart of a method of a virtual keyboard system for performing a key event generation routine according to one embodiment of the disclosure.
13 is a block diagram of an alternative embodiment of a virtual keyboard system.
14 is an X-axis calibration table according to one embodiment of the disclosure.
15 is a Y-side calibration table according to an embodiment of the present disclosure.
16 is a plan view of superimposed coordinates on an apparatus and a work surface according to the present disclosure.
17 is a graphical representation of subpixel values associated with the center of a finger tip.

In the following description, certain specific details are set forth in order to provide a thorough understanding of various embodiments of the disclosure. However, one of ordinary skill in the art appreciates that the present disclosure may be practiced without these specific details. In other instances well-known structures for sensor and image projection have not been described in detail so as not to unnecessarily obscure the description of embodiments of the present disclosure.

Unless the context otherwise requires, in the following detailed description and in the claims, the term "comprise" is used in an open, inclusive sense, that is, in the sense of "including, but not limited to," Used.

Also, throughout this specification, "an embodiment" means that a particular function, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Accordingly, "an embodiment" shown throughout this detail does not necessarily refer to the same embodiment. In addition, the specific functions, structures, and features may be combined in any way in one or more embodiments.

In the drawings, like reference numerals refer to similar functions or components. The size or relative position shown in the figures is not always drawn to scale. For example, the shapes of the various functions are not drawn to scale, but enlarged and positioned to increase the readability of the drawings.

FIG. 1A is a side view of a virtual keyboard device 9 having a housing 10 that includes a light source unit 12 positioned adjacent the bottom 14 and front 15 of the housing 10. In the virtual keyboard device 9, a light projection system 26 is located adjacent the top face 18 of the housing 10, and the light sensing system 16 is located between the light projection system 26 and the light source unit 12. Located. The projection system 26 projects the keyboard pattern 38 over the working surface 22 in the detection area 23 (see FIG. 1B).

The user's finger 24 is shown to intersect the fan shaped light beam 20 generated by the light source unit 12. The user finger may touch the working surface 22 but is detected only by breaking the sector light beam 20. The detection area 23 is a two-dimensional area defined as an area that can be detected by the light beam 20 and the sensor system 16. The working surface 22 does not include sensors or mechanical or electrical components, and may be a table surface, an airplane tray, a piece of paper, or any other suitable surface. The detection area 23 is any area where the image sensor 16 can capture an image of the tip of the user's finger 24 or other tip, such as the tip of a stylus or pen, therein.

The light source unit 12 emits a thin fan-shaped light beam 20 of a specific frequency band, such as an infrared (IR) band (700 nm to 1,400 nm). Ideally, the thin fan shaped light beam 20 is parallel and close to the working surface 22. The light sensing system 16 is angled or otherwise configured to look down the work surface 22, and continues to detect the user's activity on the work surface 22 in the detection area 23.

The housing 10 can include a light source 25 set to project light onto the work surface 22. The light source 25 provides the user with a visual indication of the boundary of the field of view 36, allowing the user's finger movements to be in the field of view 36. Light 25 may be an LED, an incandescent lamp, or the like. In addition, the intensity of the light source 25 can be increased or decreased to the intensity required by the user.

In one embodiment, the light source 25 may be positioned to illuminate a portion of the work surface 22. The light source 25 can only function to indicate to the user an approximate area of finger movement that can be detected by the sensor system 16. In one embodiment, the sensor system is set to light at a frequency corresponding to the light source unit 12 so as to improve reflection from the user finger 24. Although light source 25 is shown above sensor system 16 and projection system 26, light source 25 may be located below or next to sensor system 16. According to one embodiment, the light source 25 may comprise a plurality of light sources positioned on the device 9 in order to create aesthetically good design in close proximity to the periphery.

In one embodiment, the light beam 20 is projected parallel to the work surface 22 and is a distance 21 away from the work surface. In one embodiment, the distance 21 is between 3 and 10 mm. However, the light beam 20 may be projected at different angles to irregular work surfaces, such as irregular, irregular or angular work surfaces. The light source unit 12 may be an LED, laser or other light source capable of projecting light of a particular frequency into the thin fan shaped light beam 20. The light beam 20 has a thickness in the range of 0.1 mm to 0.5 mm, or in the range of 3 mm to 5 mm. The light beam 20 emits with sufficient intensity to illuminate the finger tip when in the beam of the light beam 20. The light may be of any frequency, but is preferably invisible. In other words, an infrared (IR) region with a frequency range of 1 to 430 THz is the preferred choice.

In one embodiment, the light source unit 12 flashes and projects the light beam 20. Flashing and projecting the light beam 20 has the advantage of saving power consumption as compared to continuously projecting the light beam 20. For example, if the light source unit 12 flashes at a frequency of 30 Hz and a duty cycle of 50% and projects light, the light source unit 12 is only turned on for half the time as compared to other cases. Additional power efficiency can be used to increase battery life or increase the intensity of the light beam 20 by the saved power. The greater the intensity of the light beam 20, the better the signal-to-noise ratio characteristic of the image captured by the sensor system 16. Increasing the signal-to-noise ratio increases the measurement accuracy and thus improves the accuracy and performance of the device 9.

FIG. 1B shows an isometric view of the virtual keyboard device 9 of FIG. 1A projecting the keyboard pattern 38 onto the virtual work surface 22. Initially, there are no fingers on the working surface 22 or in the detection area 23. The user can select a key using one or more finger tips, such as a finger 24 or a stylus, to destroy the plane created by the light beam 20. In one embodiment, the user's finger 24 must destroy the ray only in the detection area 23 to select the keys of the virtual keyboard 38. However, in another embodiment, the device 9 determines that the user's finger 24 has selected a key when the finger tip has touched the work surface 22.

Sensor system 16 detects the presence of a tip or tips in the plane of light beam 20. The sensor system 16 takes an image of the field of view 36 in a rectangular, circular, coincident with the shape of the light beam 20, or other suitable shape as shown in FIGS. 2A and 3A.

As a result of the thickness of the light beam, the closer the finger 24 is to the work surface 22, the greater the intensity of light in its image that represents its shape. That is, when the finger 24 first touches or penetrates the light beam 20, the corresponding image includes a small area of light representing the finger 24. As the finger penetrates the light beam 20 deeper, the area of the corresponding light in the image becomes larger. Finally, when the finger touches the work surface 22, the finger 24 returns the largest image.

The image sensor system 16 subsequently collects images of a user object (such as a finger) that is illuminated by the light source 12 within the detection region 23 of the keyboard. The images are then analyzed and a key press or event is reported as a result of the keys printed or projected by the user's finger 24 being touched. The sensor system 16 of the device 9 is configured to capture a plurality of images 100, 104 in order to detect the position of a finger tip or finger tips within the field of view 36 (see FIGS. 2A, 3A). Images 100 and 104 are examples of images of first finger 106 and second finger 108 that type various letters.

Before any object enters the field of view 36, the light sensor 16 captures dark images, ie no object or finger is detected. As the finger 24 descends to touch the keys of the projected keyboard pattern 38 on the work surface 22, as soon as the finger penetrates or touches the fan-shaped light beam 20, the lower part of the finger (finger) Tip) is illuminated by the light source 12. Echoes of light in the illuminated portion of the finger 24 are then captured or received by the light sensor system 16. After touching the key, the finger moves up to leave the detection area 23. If the finger 24 does not interact with the fan shaped light beam 20, the light sensor 16 does not detect the finger tip because it does not receive the echo of light.

The sensor system 16 has an optical lens and a corresponding optical filter, which blocks light other than the frequency band used by the light source and allows the band of desired light to the image sensor. More specifically, the corresponding filter is designed to allow the passage of incoming light with the frequency band of the light source and block the rest. The optical filter may block some of the significant light outside the frequency of the light beam 20 emitted from the light source unit 12. The optical filter may be a low frequency filter, a high frequency filter, or an intermediate frequency filter according to the frequency of the light beam 20. In one embodiment, the light beam 20 is emitted in the infrared frequency band and the optical filter blocks all frequencies other than the infrared frequency band.

Optical filters of the sensor system 16 may be manufactured in the sensor system 16. Many image sensors include a color filter array (CFA) applied prior to applying the micro lens layer in manufacturing. Otherwise, after manufacturing the image sensor, a film-like material may be covered on the lens to block all frequencies except the frequency of the light beam 20.

Because the optical filter blocks most incoming light and passes only a range of light, the pixel values of the image indicate the amount of reflected light or the intensity of the light. Therefore, R (red), G (green), B (blue) images do not exist. The images generated by the present disclosure are only intensity images or gray level images. To reduce the interference of ambient light, the image can first be subtracted from its previous background image and vice versa. This background image is collected when the device is turned on and the collected images are stable and updated when there is no user activity for a period of time (see FIG. 8). After background image differential processing, various known image segmentation techniques can be used to detect one or more fingers or objects. In particular, histogram-based methods are applied for computational efficiency reasons, which are described in more detail below. After image segmentation, candidate objects such as fingers or finger tips are defined, which are then further screened according to size.

In one embodiment, the device 9 includes a low power motion detection sensor device that can signal the optical system 12 and the sensor system 16 when a motion is detected. This is for a low power mode, such that the optical system 12 and sensor system 16 enter a sleep mode if there is no motion in the field of view 36 for a period of time. When motion is detected, light system 12 begins to emit a fan-shaped beam and sensor system 16 begins to capture an image.

The device 9 may comprise a power button 30 for controlling the operating state or mode of the device 9. In addition, the power button 30 can be used to connect the device 9 and the host device via a communication protocol such as Bluetooth. For example, pressing the power button 30 twice may indicate that the device 9 is about to activate internal Bluetooth propagation. Furthermore, the device 9 may use the power button 32 to enter a low power standby mode where the device 9 can recover from sleep mode more quickly. For example, a user may enter a low power standby or sleep mode by pressing the power button three times. Otherwise, the device 9 can be programmed to enter the low power standby mode by holding down the power button 30 for a few seconds.

The housing 10 may also include a communication port 32 to replace the connection of the device 9 with the host device. For example, the communication port 32 may be a USB port, a FireWire port, a mini USB port, or the like. The device 9 may be set such that when connected to the host via the communication port 32 the device 9 turns off all radio communication propagation or all the batteries contained in the device 9. The device 9 can also enter the charging mode automatically when connected directly to the host device via the communication port 32.

As mentioned above, FIGS. 2A and 3A are simplified examples of images 100 and 104 taken by sensor system 16 as the user presses a key over keyboard pattern 38. As shown in FIG. 5, for ease of explanation, the images are limited to portions 35 of the keyboard pattern 38. In this embodiment, the part 35 corresponds to the left side of the standard QWERTY keyboard seen by the device 9.

Images 100 and 104 are shown in an array 116 of 10 x 10 pixels for ease of illustration. In practice, each image can have a number of pixels 116. For example, FIG. 4A includes a portion 322 of an image of 200 pixels corresponding to 20 columns by 10 rows. The image from which the portion 322 is cut out may have 4,000,000 total pixels corresponding to 2,000 rows and 2,000 columns.

According to one embodiment of the present disclosure, sensor system 16 may be a CCD or CMOS image sensor with various resolutions. Such an image sensor may include hundreds of thousands to millions of pixels in each image, which are comprised of rows 135 and columns 134. In one embodiment, sensor system 16 is implemented with a relatively low resolution image sensor. For example, sensor system 16 may be a VGA image sensor with 640 rows and 480 columns of pixels. As another example, sensor system 16 may be a CIF image sensor having 352 rows and 288 columns of pixels. However, sensor system 16 may be implemented with an image sensor of any higher or lower resolution in addition to those mentioned herein.

Sensor system 16 may be configured to capture images of a particular frequency. In one embodiment, sensor system 16 captures an image at a rate of 30 frames per second. To reduce power consumption, you can configure the rate at which images are captured so that they are captured only when movement on the work surface is detected.

In one embodiment, the sensor system 16 synchronizes the image capture with the blinking of the light of the light source unit 12. For example, if the fan-shaped light beam 20 blinks to turn on for 1 millisecond, the shutter of sensor system 16 opens for 3 milliseconds. The shutter opens for 1 millisecond before the light blinks and then opens for 1 millisecond to capture the tip of the finger or stylus accurately. As mentioned above, flashing the light beam 20 makes more power available per beam resulting in a stronger light beam 20. More tips reflected by the stronger light beam 20, or otherwise reflected by the tips, appear in the image.

The frequency of the blinking light beam 20 can be determined in the details of the image sensor 16. For example, the light beam 20 blinks at 30 Hz when the image sensor 16 is set to capture 30 frames per second. Some image sensors 16 include pins, commonly referred to as strobe output pins, which can be used to synchronize the blinking of the light beam 20 with the image capture of the sensor system 16. This strobe output pin sends a signal to the light source unit 12 to begin ejection of the light beam 20, which causes the sensor system 16 to capture the image when the sensor system 16 captures the image. Wake up some time after the scheduled time to maximize the light available to illuminate the user's finger 24.

If sensor system 16 is incorrectly synchronized with light source system 12, device 9 may not accurately detect the presence of tip 24. To avoid this detection error, flickering from the light source unit 12 may be at least twice the capture frequency of the sensor system 16 according to the Nyquist theorem.

Images 100 and 104 are processed immediately in real time and then discarded if it is determined that there is no finger in detection area 23. The apparatus includes a processing and control system 218 to analyze the images; However, in alternative embodiments, a host or companion device may be used to process the images as described herein.

6 diagrammatically illustrates a virtual keyboard system 200 that includes a processing and control system 218 and a host 210. The processing and control system 218 receives images from the light sensing system 16, performs a series of routines for detecting the user's fingers, detects a finger tip in the images, and the finger may The time point for touching or entering the detection area 23 is found, and it is determined which key is touched. If a key touch is determined, a 'Key-down' event is reported to the host 210. If the finger is still touching the key in subsequent images, a 'Key-held' event is reported for each image frame while the finger is touching the work surface 22. If the finger moves away from the key, a 'Key-up' event is reported.

In addition, the processing and control system 218 determines whether one or more fingers touch the work surface 22 and generates an appropriate event for the operation of each finger. Each of these events will be described in more detail below with respect to FIGS. 6-12. In particular, one routine detects where the finger shines by calculating column and row histograms and determining the peaks of the histograms.

For example, referring to FIGS. 2B and 3B, respective thermal histograms 110, 114 of images 100, 104 are generated by processing and control system 218. 2C and 2E are images of the same objects as those of FIG. 2A and are processed to generate row histograms 610 and 612 of FIGS. 2D and 2F, respectively. FIG. 3C is an image of two objects from FIG. 3A and the corresponding row histogram 616. Each of the histograms will be described in more detail below.

The virtual keyboard system 200 includes a projection system 26, a sensor system 16, and a light source unit 12. The processing and control system 218 includes a central processing unit (CPU) 220, a memory 222, a communication unit 224, a projection control unit 225, a sensor control unit 227, and a light source driver 223. do. The communication links 217a, 217b, 217c connect the projection system 26 and the sensor system 16 light source unit 12 to the processing and control system 218, respectively.

The processing and control system 218 processes the images collected from the sensor system 16 and communicates with a host device 210, such as an electronic device that receives text input from a user computer, smartphone, laptop, or other user. System 200 is coupled with host device 210 via communication link 214. The communication link 214 may be wired or wireless. The host may control a communication protocol for data transmission between the host 210 and the device 200. The host 210 may provide a battery charging service via the communication link 214.

For example, FIG. 2A is a first image 100 taken by a sensor system 16 comprising a first finger 106, a second finger 108, illuminated by a fan shaped light beam 20. The first finger 106 is located close to the first edge 118 of the first image 106. As the first finger 106 is illuminated, a crescent or curved shape is captured. This crescent or curved shape is typically the shape of a human finger or other curved object such as a pen or cylindrical stylus. If a square object is used, the image will capture another rectangular shape.

The processing and control system 218 of FIG. 6 is configured to analyze these images to determine when the finger 24 touches the work surface 22 and to detect a selection of keys or key touch. The processing and control system 218 is configured to capture the images 100, 104 and to determine what event occurred if an event occurred.

The CPU 220 executes commands stored in the memory 222 such as image sensor correction, image correlation, digital signal processing, and the like, and determines which key is selected by the user. The processing and control system 218 receives an instruction from the host device 210, sends an instruction to the host device 210, and indicates text input from the user's finger or fingers in the work area 23. The communication unit 224 can be used to send data.

The sensor control unit 227 sets parameters of the sensor system 16 such as shutter time, frame rate, start row and column 134, end row and column 134 of the image sensor, and snaps the working surface 22. Fire the sensor to take a shot. Processing and control system 218 reads images from sensor system 16 and stores them in memory 222.

The projection control unit 225 controls the overall output of the projection system 26 and turns the projection system 26 on or off. The light source driver 223 may control the light source unit 12 and adjust the output to turn on and off the light source unit 12. The processing and control system 218 synchronizes the light source unit 12 with the sensor system 16. To take a snapshot of the work surface 22, the processing and control system 218 controls the light source driver 223 to turn on the light source and the sensor control unit 227 to release the shutter of the sensor system to collect each image. Control to open. After a period of time sufficient to expose the image, the embedded processing and control system 218 turns off the light source and closes the shutter. The image is then stored in memory.

The processing and control system 218 may synchronize the sensor system 16 with the light source unit 12 as described above. Embedded processing and control system 218 may control light source unit 12 to emit a continuous or flashing light beam 20. System 200 is configured to rely on CPU 220 and memory 222 of processing and control system 218 to communicate with host 210 via a wireless connection 214 to host device 10 and the system. Bandwidth between 200 may be preserved.

7 is a block diagram of one method performed by a virtual keyboard system according to an embodiment of the present disclosure. The image sensor system 16 collects images of the working surface 22 of the projected keyboard pattern 38. These images are transmitted from the image sensor system 16 to the processing and control system 218 via a communication link 217b.

Processing and control system 218 receives the images, performs a tip detection routine 228 that processes the images and identifies the location in x and y coordinates of the finger tips within the detection area. The tip detection routine 228 will be described in more detail below with respect to FIG. 8. The touch detection routine 230 then processes and blocks the detected finger tips to see if any of the finger tips touch the work surface, and whether even one finger tip is still on the work surface. The touch detection routine 230 will be described in more detail below with respect to FIG. 9.

The touch position calculation routine 232 receives the information about the finger tips touching the work surface 22 and accesses the calibration table 234 to determine the position (x, y) of the touching tips. The touch position routine 232 will be described in more detail below with respect to FIG. 10. Key identification routine 236 searches or otherwise accesses key map table 238 to determine which key is associated with the location (x, y). The detected key information is identified and sent to the key event generation routine 240. In addition, the detected key information is analyzed by the key block detection routine 242 to see if there are no keys blocked by the currently detected finger tips. The key breaking routine 242 and key identification routine 236 will be described in more detail below with respect to FIG.

The key lock detection routine 242 receives information about previously detected keys from the key event storage routine 244. If the key lock detection routine determines that some keys have been blocked, the blocked keys are reported to the key event generation routine 240. The key event generation routine generates a key event based on the detected touched key, blocked keys, and previously selected keys. The key event is then sent to host 210. The key event generation routine 240 will be described in more detail below with respect to FIG. 12.

8 is a flowchart of a series of steps executed by the tip detection routine 228. In general, once turned on, the device 9 continuously takes an image of the working surface 22. As mentioned above, the first images captured by the device 9 will not include the finger 24 and are considered background images. At 246, during initialization of the virtual keyboard the background image is acquired and updated if there is no user activity for a period of time. The background image may be stored in memory 222 or in other ways for comparison with future images.

Alternatively, the sensor system 16 can take images continuously and compare the current image with the immediately preceding image. The background image is stored or updated once a large number of very similar consecutive images are captured, such as 100 frames of image. Once it is identified that the current image is different from the background image, the current image is processed at step 250 to remove the background image data.

At 248, the images of FIG. 2A or 3A are received by tip detection routine 228. After the background image is extracted or removed, at 250, a column direction histogram is calculated from the difference image at step 252. The histograms are sums of rows and columns of pixels. For example, as shown in FIG. 2A, the crescent shape of the first finger 106 appears across three pixels 120, 122, and 124. The crescent shape of the second finger 108 appears across the four pixels 126, 128, 130, and 132. 2B is a histogram 110 showing the sum of each column 134 of the array of pixels 116. More specifically, the values of column 134 of each pixel 116 are summed to determine the intensity of light per column 134. The histogram 110 includes two spikes 136, 138 corresponding to the first and second fingers 106, 108, respectively. In alternative embodiments, an average of the intensity of light from histogram 110 may be used, which will be similar to the sum by other ratios.

The intensity of the first spike 136 is greater than the second spike 138 because the crescent shape of the first finger 106 is larger than the crescent shape of the second finger 108. The first spike 136 coincides with the center of the column that includes the pixel 122 that includes most of the image of the first finger 106. The pixel 122 has the largest intensity per pixel in the pixel array 116. The intensity of the second spike 138 coincides with the boundary of two adjacent columns comprising the pixel 128 and the pixel 130, corresponding to the central portion of the second finger 108. In addition, the first finger 106 is stronger because it is closer to the image sensor 16.

For example, in one embodiment, the image sensor will capture the intensity of photons incident on each pixel 116 of the pixel array. Objects closer to the image sensor 16 are more incident photons captured by the image sensor than objects located further away from the sensor, such as when each of the first finger 106 is compared to the second finger 108. Will provide. The intensity of the photons is measured from the most recent reset time, which may be the opening of the shutter covering the sensor. Photon intensity is then converted to voltage. Each pixel of the row of the pixel array can be read simultaneously. After reading all the rows of the pixel array, the device 9 may calculate the average intensity of all columns 134 of the pixel array by adding each pixel of each column 134. Since the image sensor captures filtered reflections of a particular frequency within the field of view 36 or detection area 23, the histogram (eg, plot of intensity and column number) shows that a significant increase in the average intensity of the pixel is a finger or finger tip. Represents the columns corresponding to the reflection of.

The moving average 111 of the column histogram is then determined in step 254. The moving average 111 shown in FIG. 2B is calculated for a predetermined window width n. For example, the moving average 111 of column k of window (2m + 1) is calculated according to the following formula:

COLUMN (i) is the sum of the columns of column i. In a preferred embodiment, the window width is odd. The sum data includes m columns to the left and right of the column k to be calculated. For example, the moving average 111 may be calculated by summing 21 total columns and dividing by 21. That is, ten columns to the left in column k, plus column k, and ten columns to the right in column k.

In step 256, the tip detection routine 228 determines which of the column direction sums or averages is greater than the moving average 111. Segments 136a and 138a corresponding to columns larger than moving average 111 are good candidates for detecting finger 24 in detection area 23.

At 258, the columns associated with segments 136a and 138a are separated from or otherwise truncated from image 248 to limit the data processed to determine row direction histograms 610 and 612. For example, FIGS. 2C and 2E are examples of sets of columns 134 separated from image 100 for row directional histograms associated with segments 136a and 138a, respectively.

In one embodiment, based on where thermal histogram 110 intersects moving average 111, that is, points 186, 188 at first spike 136 and points at second spike 138 ( Based on 190 and 192, tip detection routine 228 determines the width of first slice 184 and second slice 194 of image 100. From the points above, the tip detection routine 228 may include a first slice 184 having three rows containing the first finger 106 and a second slice having four rows including the second finger 108 ( 194). In some embodiments, the slice determined from the points 186, 188, 190, 192 only includes the majority of the finger, and the edge of the finger in the image is not included in the slice, see FIG. 4A.

At 258, the rows of the first and second slices 184, 194 are combined to form row histograms 610, 612, respectively. In step 260, moving averages 611 and 613 are generated for each of the row direction histograms 610 and 612. As in the column histogram, segments 614 and 615 of the histogram larger than the moving average are identified in step 262. Rows associated with the segments 614, 615 are identified. For example, in FIG. 2D, the first finger 106 is in only one row of the first row 180. In FIG. 2E, the second finger 108 is in only one row of the second row 182. In practice, one finger is likely to occupy several rows of pixels 116, see FIG. 4A.

At 264, finger tip candidates corresponding to column coordinates and row coordinates are formed. For example, column coordinates (m, n) and row coordinates (r, s) can define a block of an image containing a finger tip candidate. This is repeated for each slice 184, 194 in the image 100.

Points 186 and 188 and points 190 and 192 generate first column coordinates for first finger 106 and second finger 108, respectively. For example, if the columns are numbered from the left edge, the points 186, 188 may be associated with the first column edge 202 and the second column edge 205 to form column coordinates 202, 205. Can be. The points 190, 192 of the second finger 108 then have column coordinates 206, 210 corresponding to the third row corner 206, the fourth row corner 210. The column coordinates are stored and will be used by the calibration tables to determine which character the finger selected.

The row coordinates are determined by the points 618, 620 of the histogram 610 and correspond to the height of the first finger 106, as detailed below. Row coordinates may be determined by points 618 and 620 corresponding to the first row edge 601 and the second row edge 602. Thus, the row coordinates for the first finger 106 are (601, 602). The points 622 and 624 where the row histogram 612 and the moving average 613 intersect correspond to the second finger 108. The points also correspond to the third row edge 607 and the fourth row edge 608. This provides row coordinates 607, 608 for the second finger 108.

The first finger has column coordinates 202, 205 and row coordinates 601, 602 and the second finger has column coordinates 206, 210 and row coordinates 607, 608. These coordinates define the tip box, ie the boundaries of the tip in some images, which can be used by other routines as detailed below.

Once finger tip candidates are obtained and column and row coordinates are determined, tip detection routine 228 performs a finger tip filter at 266. The filter filters finger tips based on the block size of the finger tip candidates, ie based on column coordinates (m, n) and row coordinates (r, s). The tip detection routine 228 outputs finger tip image data, that is, outputs information of the finger tip or finger tips including the associated row and column coordinates at 268.

3A is another example of detecting coordinates of rows and columns from the histograms. The second image 104 relates to another key press of the first and second fingers 106, 108. The first and second fingers 106, 108 appear close to each other in the third image 104. The first finger 106 is located at six pixels 160, 162, 164, 166, 168, 170. Pixels 160 and 170 are in one column, pixels 162 and 168 are in a second column, and pixels 164 and 166 are in a third column. When the respective rows are averaged or added, the intensity of the first finger 106 is shown by the first spike 172 of the third histogram 114 of FIG. 3B.

The second finger 108 is located at four pixels 174, 176, 178, 166. Pixel 166 also includes a portion of first finger 106. The third histogram 114 has a valley 180 corresponding to the column with the pixel 166, which includes some of both the first and second fingers 106, 108. The third histogram 114 also has a second spike 182 corresponding to the thickest portion of the second finger 108, which is the maximum intensity of light captured by the image sensor.

The crescent shape generated by photographing the first and second fingers 106 and 108 varies in area depending on the size of the user's finger and the distance from the image sensor. That particular area is irrelevant in this disclosure because each column of each image is added to determine the intensity spike. Also, if other objects are used, the images are not necessarily crescent shaped. For example, if a rectangular stylus is used, the image sensor will capture a rectangular representation of that stylus.

Moving average 626 is determined to find a portion of histogram 114 above moving average 626. As discussed above, portion 628 of histogram 626 can help determine the strongest pixel of the image and can help determine the center point of the finger. In the image, the first and second fingers can be seen as one finger that is evaluated and separated by the key blocking routine described below.

9 is a flowchart of a method of a virtual keyboard system for executing a touch detection routine 230 in accordance with an embodiment of the present invention. At 268, finger tip image data from the tip detection routine 228 is received by the touch detection routine 230. The touch of the finger on the working surface 22 is determined when three conditions are met. Conditions are provided to determine if the user intends to touch the work surface or select a key. If the finger moves fast enough to penetrate the beam very sufficiently, the routine determines that the user intended to touch to select a key.

These three conditions include calculating the count or tip age while the tip appears in the images, the tip dimension compared to the beam thickness, and the tip dimension compared to the tip dimension threshold. At 272, the tip age is determined by the number of consecutive frames in which the tip appears. The count starts as soon as a tip is detected in the frame. The count or age is zero for the first frame where the tip is detected. For each successive frame after count zero, one count is added for each frame where the tip is still detected. Thus, if the tip appears in the next frame, the tip age is one. To meet the first criterion, the count or tip age must be greater than or equal to two consecutive image frames and less than or equal to six consecutive image frames.

If the tip age does not meet the first criterion, the touch detection routine 230 returns "Touch not detected" at 274. If the tip age meets the first criterion, the touch detection routine 230 executes the second criterion at 276.

The second criterion determines the dimension of the tip image and compares that dimension with the thickness of the light beam 20. The dimension of the tip is determined by the number of rows in which the tip is fitted. For example, the first finger 106 with row coordinates 601, 602 has a dimension of one row.

The thickness of the light beam 20 is not constant from one end to the other end of the image. For example, the thickness of the light beam 20 is smaller as closer to the light source 12, and the thickness of the light beam 20 is larger as farther from the light source 12. Thus, the first finger 106 interacts with a light beam 20 of a different thickness than the second finger 108. As mentioned above, the fingers appear larger the closer to the device 9, because the fingers reflect more light.

During beam calibration, such as when the device 9 is being tested before being shipped to a retailer or customer, a beam thickness table is created or otherwise obtained. The beam thickness table can be accessed with row coordinates to determine which thickness of the beam to compare with the dimension of the tip height. In order to meet the second criterion, the finger must penetrate at least one third of the beam. Thus, the dimension must be greater than or equal to 1/3 of the thickness of the light beam. Since the thickness of the beam can vary based on light source variations and other situations, the dimension is compared with one third of the actual thickness. Different sizes of beam thicknesses can be compared based on the properties and intensity of the light source of the device 9.

If the dimension does not meet the second criterion at 276, the touch detection routine outputs "Touch not detected" at 274. However, if the second criterion is met at 276, the touch detection routine determines whether the third criterion is met at 278.

The third criterion compares the change in tip dimension from one frame to the next at 278 with the tip dimension threshold. In order to detect the touch of a finger on the working surface, the tip dimensions from the first frame and the continuous frame must vary by more than the tip dimension threshold. The dimension may be referred to as the tip height or the number of rows over which the tip spans.

The tip dimension threshold is not constant but rather depends on the distance of the finger from the device 9. The tip dimension threshold is larger when the finger is closer to the sensor 16 and smaller when the finger is further away from the sensor 16. A table with tip dimension thresholds determined by row and column coordinates may be accessed by tip detection routine 230 to execute step 278. Comparing the images with the size or heights of the fingers in the table provides information for detecting a key touch on the work surface 22. Defects such as the flicker of the light source 12 can affect the usefulness of this table. For example, when the light gets darker, the thickness of the beam becomes thinner, which affects the data collected around the finger.

In an alternative embodiment, the key touch is determined by comparing the images and determining when there is a large increase in finger size in the image sequence. Once the finger breaks the fan beam 20, the device starts the comparison routine. The size of the finger in the images is determined. The processing and control system 218 compares the sequential images to determine if the size is increasing. After the increase in size is determined, the processing and control system 218 waits for another 2-3 frames and determines whether the finger size has increased, decreased, or stayed the same. When the finger size was reduced, the finger did not touch the work surface 22. If the finger size remains the same, the processing and control system 218 waits for another 2-3 frames and determines if a change has occurred. If the finger size increases during the threshold number of frames, a key touch is detected.

At 278, if the change in dimension does not meet the third criterion, the touch detection routine outputs "Touch not detected" at 274. However, if the third criterion is satisfied at 278, the touch detection routine returns touch detection at 280.

In an alternative embodiment, instead of the tip height, the third criterion may be associated with the time period during which the finger appears in the series of images. This time period is used to determine the speed of the finger moving towards the work surface 22. When the speed was zero, a touch was detected, and all the movements of the finger were temporarily stopped.

In one embodiment, the image sensor 16 operates at 30 to 60 frames per second, and the thin light beam is in the range of 3-5 mm in beam thickness. Thus, since the finger 24 blocks the light beam 20 and touches the work surface 22, there may be only 1-3 frames of images of the finger 24. However, so few images may interfere with accurate calculation of the speed of the finger 24.

10 is a flowchart of a method of a virtual keyboard system for executing a touch position calculation routine 232. If the touch detection routine returns touch detection at 280, the touch position calculation routine 232 is executed. The touch position calculation routine 232 receives finger tip image data from step 268, such as row and column coordinates (r, s) and (m, n). Row and column coordinates define a block of the image where the finger tip is likely to be found. In one embodiment, the block is rectangular as in FIG. 4A, but other shaped blocks may be used.

At 282, the touch position calculation routine 232 processes the row and column coordinates to determine the block associated with the finger. In an alternative embodiment, the finger tip may be further smaller than the block defined by the row and column coordinates, such as a tip box defined by the left right, top, and bottom boundaries of the tip. To define precisely, the block is processed further. Tip boundaries are determined by finding a pixel that has the greatest intensity relative to other pixels in the block. This can be accomplished by comparing the pixels individually, or by comparing multiple pixels near the center to determine the center of the block and find the pixel with the greatest intensity. For example, in FIG. 2C, the pixel with maximum intensity would be a row between column and row edges 601 and 602 between column edges 203 and 204. The coordinates of the pixel can be represented by (row edge, column edge), so the coordinates will be (601, 203).

As the pixel has the maximum intensity, the tip box in the block can be determined. The touch position calculation routine 232 determines a threshold intensity value for comparison with pixels near the maximum pixel. The threshold intensity value is a percentage of the intensity of the pixel with the maximum intensity. For example, an intensity of 50% of the maximum intensity may be a threshold intensity value. The touch position calculation routine 232 then determines the right and left boundaries of the tip box, that is, the touch position calculation routine 232 determines the row boundaries of the tip box. The touch position calculation routine 232 compares a single pixel on the right side of the pixel with the maximum intensity, to see if the pixel on the right is greater than the threshold intensity value, and if larger, the touch position calculation routine 232 The two right side pixels of the pixel having the maximum intensity are compared. The touch position calculation routine 232 continues comparing each pixel on the right side of the pixel with the maximum intensity until the pixel being compared is less than the threshold intensity value. If the pixel is smaller than the threshold intensity value, the coordinates of the pixel one pixel earlier than the threshold intensity value are used as the right boundary of the tip box.

This same process continues for the pixels in the same row on the left side of the pixel with the maximum intensity to determine the left boundary of the tip box. In addition, the touch position calculation routine 232 determines the upper and lower boundaries of the tip box as the pixel having the maximum intensity through a similar process of comparing the pixels in the same column.

Once the boundaries of the tip box are defined, the touch position calculation routine 232 determines, at 284, the subpixel resolution of the center of the tip. 4A is a portion of an image that includes a tip box 322 of a finger 324 as captured and processed by the device 9. This tip box includes 200 pixels extending from column edge 300 to column edge 320 and from row edge 400 to row edge 410. The pixel with the maximum intensity of the finger 324 may have a pixel at 405, 310, such that a tip box is formed by comparing the pixels on the left, right, top and bottom of the pixel with a threshold at 405, 310. do.

At 284, subpixel resolution is used to determine the center of tip 324. The touch position calculation routine 232 determines a column-wise histogram 326 and a row-wise histogram 328 for the tip box 322. Each pixel is added together in each row and column to form histograms. The coordinates (row, column) of the center of the finger 324 are determined from the row and column histograms 328, 326. For example, with respect to histograms, the center of finger 324 is at (405.5, 309). The coordinates need not be integer values, and in fact, having non-integer values leads to a more precise center of the fingers.

Subpixel center detection is beneficial because fingers closer to the sensor 16 appear larger and have better resolution than fingers further away from the sensor. Determining the center point with subpixel resolution is more accurate, which provides a more accurate decryption of the user's selection of keys.

At 286, the touch position calculation routine 232 accesses the calibration table 234 and utilizes bilinear interpolation of the center of the tip (rows, columns) to thereby associate (x, y) Determine the physical widgets in the coordinates. In particular, the centers (rows, columns) of the detected objects are converted to coordinates in the real world by using calibration tables such as the calibration tables of FIGS. 14 and 15.

More specifically, bilinear interpolation is used to provide (x, y) coordinates for the physical location of the tip based on the subpixel value of the center of the finger tip. The physical location will be used to determine which key was selected by the user. The central subpixel value is determined at an earlier stage of processing. From the subpixel values, the associated pixel is determined.

17 is a graphical representation of an associated pixel based on subpixel center 630 with non-integer row mill column values, (r, c). 17 is associated with the x value of the coordinates of the physical location. The y values of the coordinates of the physical location are determined in a similar manner.

The x and y values are collected from the x-correction table and the y-correction table, respectively, and refer to FIGS. 14 and 15. The x and y values are determined from (r, c) coordinates and the row and column values that are the nearest integers, for example r1, r2, c1, and c2 from FIG.

To perform bilinear interpolation, the system assumes r1 <r <r2 and c1 <c <c2, and r1, r2, c1 and c2 are all integers. And r2 = r1 + 1 and c2 = c1 + 1. To facilitate discussion, in the embodiment of FIG. 17, the subpixel value at the center of the finger is (2.2,633.2), r is equal to 2.2 and c is equal to 633.2. Thus, r1 is equal to 2 and r2 is equal to 3. In addition, c1 is equal to 633 and c2 is equal to 634. Once the system has r, r1, r2, c, c1 and c2, the system can determine the group of x and y values from the x-correction table and the y-correction table to determine the (x, y) coordinates. .

As shown in FIG. 17, four x values may be determined from the subpixel values, X ₁₁ , X ₁₂ , X ₂₁ and X _{22 at} the center of the finger tip. The rows and columns for X ₁₁ are (2,633) and the value for x from the x-calibration table of FIG. 14 is 302. More specifically, the x value for X ₁₁ is (r1, c1), for X ₁₂ , the value is (r1, c2), for X ₂₁ , the value is (r2, c1) and for X ₂₂ , The value is (r2, c1).

By these values, the value X (rc) of pixel (r, c) is X (r, c) = X ₁₁ * (r2-r) * (c2-c) + X ₂₁ * (r-r1) * ( c2-c) + X ₁₂ * (r2-r) * (c-c1) + X ₂₂ * (r-r1) * (c-c1).

Similarly, the value Y (rc) of pixel (r, c) is equal to Y (r, c) = Y ₁₁ * (r2-r) * (c2-c) + Y ₂₁ * (r-r1) * (c2 -c) + Y ₁₂ * (r2-r) * (c-c1) + Y ₂₂ * (r-r1) * (c-c1) can be calculated.

The X (r, c) and Y (r, c) values allow the system to access the key map and determine which keys have been selected by the user.

Returning to the calibration tables, these consist of mapping from image coordinates (row, column) to real world coordinates (x, y). 14 and 15 are exemplary X-axis calibration table 500 and Y-axis calibration table 600 used to translate the position of an object from the images to the position of the object relative to the keyboard. FIG. 16 is a top view of the apparatus 9 illustrating the X-axis and the Y-axis as referenced in the calibration tables 500, 600. Points 0,0 on the X and Y axes of FIG. 16 pass through the center of the device 9. The light beam 20 is centered on the Y-axis and extends in both the positive and negative directions on the X-axis. The light beam 20 extends to what is commonly referred to as the negative Y-axis, but since only half of the Y-axis is relevant to this discussion, all values discussed for the Y-axis would be considered positive. will be.

14 is an X-axis calibration table 500 having a plurality of rows 504 and columns 502. Columns 502 represent columns 1-8 and 633-640. Rows 500 include rows 1-8, 236-243, and 473-480. X-axis calibration table 500 includes 640 columns and 480 rows, which is consistent with a VGA sensor. In row 1, column 1, the value is negative 313 (-313), which may correspond to 31.3 mm. This value may correspond to point A on the X-axis of FIG. 16. The sensor system cannot accurately capture tips too close to the device 9, so the values illustrated in the X-axis calibration table 500 start at a distance away from the device 9. In row 1, column 640, the value is positive 380, which may correspond to 30.8 mm. This value is represented by point B on the X-axis of FIG. 16.

In row 480, column 1, the value is negative 4414 (-4414), which may correspond to 441.4 mm. This value is represented by point C on the X-axis of FIG. 16. In row 480, column 640, the value is positive 4346, which may correspond to 434.6 mm. This value can be represented by point D on the X-axis of FIG. 16. The area 188 defined by points A, B, C and D, which are transformed to the boundaries of the light beam 20, corresponds to the detectable area 23, which is available to be imaged by the sensor system.

In FIG. 15, Y-axis calibration table 600 includes rows 1-8, 236-243, and 473-480. The values in each row slowly increase from 376 to 5995. Each column of the Y-axis calibration table 600 contains the same numbers for each row. The number of columns corresponds to the number of rows in the X-axis calibration table 500. Repeating the row values for each column of the Y-axis calibration table 600 can assist the software in determining the location information for each image.

For example, in the coordinates (row, column) of the center of the finger 324 determined from the row and column histograms 328, 326, the (x, y) coordinates representing the position associated with the virtual keyboard are the calibration table. It can be located using the. If the row center coordinates are 242 and the column center coordinates are 636, the X coordinate value will be 1312 and the y coordinate value will be 1796. As another example, if the row center coordinate value is 2.5 and the column center v coordinate value is 6.5, four values from the x-axis table and four values from the y-axis table will be calculated, i.e., x- For the axis table, column 2, row 6 (-317), column 3, row 6 (-316), column 2, row 7 (-318), and column 3, row 7 (-317) will be calculated. The average of all the values is -317, which will be the x coordinate. Similarly, for the y-axis table, column 2, row 6 382, column 3, row 6 382, column 2, row 7 384, and column 3, row 7 384 are calculated. will be. The average of all these values is 383, which will return the coordinates of (-317,383) corresponding to the user's finger in the detectable area 23. This position coordinate from the tables will be output from the position detection routine 232, at 288. The location coordinates are used by the key identification routine 236 to determine the selected key from the key map 238.

It should be noted that in this embodiment, 480 pixels are available to represent the distance from front to back on the working surface. If a front-to-rear distance proportional to the side to side distance is to be implemented, the distances shown between the rows may be greater than the distances shown between the columns. The second reason that the differences between the columns may differ from the distances between the rows is related to the shape of the work area on the work surface. The light beam 20 extending from the light source unit 12 extends at an angle from the housing. Thus, pixels capturing data remote from sensor 16 may exhibit larger changes in position than pixels capturing data from a portion of the work area closer to sensor 16.

The calibration table 500 can be created during testing of the device. For example, a calibration table can be created by placing a unique image in front of the device, capturing the image, and then performing data processing on that image. Alternatively, the calibration table 500 places objects in the work area, captures an image, changes the position of the object in the work area, recaptures the image, and the objects in the work area are pixels in the image sensor. It can be generated manually by processing the data to determine how it correlates with.

The calibration table may be stored in memory and used by a processor located on the virtual memory device. Alternatively, the calibration table may be installed on the host device during installation of drivers on the host device that are operable to enable the host device to communicate with the device.

Returning to FIG. 7, the location coordinates of the tip are fed to the key identification routine 236, which accesses a key map to determine the currently selected key based on the location coordinates. For example, looking at image 100 of FIG. 2A, the (x, y) coordinates of the tip correspond to the keys on the keyboard of FIG. 5. For example, the position coordinates for the first finger 106 correspond to '5' on the portion 35 of the keyboard pattern 38 and the second finger 108 corresponds to 'x'. Also, the image 104 of FIG. 3A includes a first finger 106 that selects the letter 'e', and the second finger 108 corresponds to the letter 's'. Position coordinates are determined by using the center point of the tip to retrieve data from the calibration tables.

11 is a flowchart of a method of a virtual keyboard system for executing a key block detection routine 242. The key lock detection routine 242 is for detecting from the perspective of the image sensor 16 that any keys are blocked by any other keys. Some key strokes will shoot the fingers very closely together or overlapping, for example, to start some letters with a capital letter. For example, to capitalize the letter 'a', the keys 'SHIFT + A' must be selected. The 'SHIFT' key is pressed first, followed by the 'A' key. Before the key 'A' is pressed, the finger pressing the 'SHIFT' key can be seen in the image. Once the key 'A' is pressed, the finger still holding the 'SHIFT' key can be blocked by the finger holding the 'A' key. As a result, the blocked finger may not be photographed or may be photographed only partially.

At 294, key detection routine 242 receives current key information from key identification routine 236 and determines if any previously photographed keys no longer appear in the current image. The current key information is the selected key or keys when returned from the location coordinates and the key map. The key detection routine compares the previous key list with the current key list. If some of the previous keys no longer exist in the current image, they may be blocked or may no longer be selected by the user.

At 296, routine 242 receives finger tip image data output at 268 to tip detection routine 228, see FIG. 8. Next, the key block detection routine 242 determines whether any missing keys overlap with current finger tips located within the current image based on the column comparison. For example, if one of the missing keys overlaps with the columns of current finger tip image data by 60 percent, the missing key may be a blocked key. The superposition percentage can be adjusted as needed based on the configuration of the image sensor and the characteristics of the device 9.

In FIG. 3A, only a portion of the second finger 108 is visible by the image sensor 16. The second finger 108 is put down first, so in the image before the image 104, if the only finger photographed is the finger 108, the second finger 108 is the key row coordinates 604 of gsuwo; 606 and current key column coordinates 207, 209, which corresponds to 's' in FIG. 5.

In the image of FIG. 3A, the system can detect one large finger having current key row coordinates of 603, 606 and current key column coordinates of 205, 209. At 295, key block detection routine 242 will determine that the missing key 's' overlaps at least 60% with the current key 'e'.

At 298, the key block detection routine 242 determines whether the missing key is separated from the image sensor 16 by the current key. If so, the routine 242 outputs, at 330, that the missing key is a blocked key. Determination of the distance from the image sensor can be accomplished by comparing the row coordinate numbers. For example, if lower row numbers are closer to the image sensor, the key is blocked if the row numbers are greater than the row numbers of the current key.

The current key information output from the key identification routine 236 and the blocked key data output from the key breaking detection routine 242 are received by the key event generation routine 240. 12 is a flowchart of a method of a virtual keyboard system for executing a key event generation routine 240. At 332, key event generation routine 240 adds the blocked keys to the current key list. At 334, key event generation routine 240 compares the current key list with the previous key list and determines if an event should be output. Below is a list of possible key events.

KEY-DOWN EVENT: If the key is in the current key list rather than the previous key list, the key down event is output at 336 in association with the key.

KEY-UP EVENT: If the key is in the previous key list instead of the current key list, the key up event is output at 336 in association with the key.

KEY-HELD EVENT: If the key is in both the current and previous key list, the key hold event is output at 336 in association with the held key.

Key events are received by the host to display the appropriate text on the screen or display.

13 is a flow chart of a method of an alternative embodiment of a virtual keyboard system that uses a composite key table 338, a calibration table, and a key map table. The composite key table is created by combining the calibration table (x, y tables) and the key map table. This eliminates the step of determining the position of the tip, and instead uses the row and column coordinates to map to a specific key. Thus, computational performance can be significantly improved.

For example, as discussed above with respect to FIG. 2A, the row and column coordinates associated with the tip block can be used to access the composite key table to determine the keys selected by the fingers. In FIG. 2A, the first finger 106 selects '5' and the second finger 108 selects 'x'. The first finger 106 has row coordinates 601, 602 and column coordinates 202, 205, and the second finger 108 has row coordinates 607, 608 and column coordinates 206, 210. )

The various embodiments described above can be combined to provide further embodiments. These and other changes may be made to the embodiments in light of the above detailed description. In general, in the following claims, the terms used are not to be construed to limit the claims to the specific embodiments disclosed in the specification and claims, but rather all possible implementations with the full scope of equivalents to which such claims are entitled. It should be interpreted to include examples. Accordingly, the claims are not limited by the disclosure.

Claims (18)

Capturing images of a work area including an indication of a virtual keyboard having a plurality of keys;
Detecting a selected one of the plurality of keys; And
Outputting an identifier of the selected key to a host device;
Detecting the selected key,
Detecting an object in the work area by analyzing the captured images for variations in light intensity in the captured images; And
Determining an identifier of the selected key based on a first row histogram and a first column histogram,
Wherein detecting the object comprises:
Generating a first row histogram based on light intensity in each of the rows in each of the captured images; And
Generating a first column histogram based on light intensity in each of the columns in each of the captured images. The method according to claim 1,
Detecting whether the object touches a work surface in the work area;
Wherein the detecting comprises:
Extracting a background image from each captured image;
Determining a column moving average for the first column histogram and a row moving average for the first row histogram;
Determining segments of the first column histogram that are greater than the column moving average;
Determining a subset of columns associated with segments of the first column histogram that is greater than the column moving average;
Determining segments of the first row histogram that are greater than the row moving average; And
Determining a subset of rows associated with segments of the first row histogram that is greater than the row moving average. The method according to claim 2,
Providing a light beam extending on the working surface,
The light beam is configured to generate light intensity in the captured images when the object interacts with the light beam. The method of claim 3,
Detecting whether the object touches the work surface comprises:
Determining the dimension of the object in each captured image based on the subset of rows; And
Comparing the dimension of the object to a thickness of the light beam. The method according to claim 2,
Detecting whether the object touches the work surface comprises:
Counting a number of successive captured images containing the object; And
Determining if the number of consecutive captured images is higher than a lower touch threshold and less than an upper touch threshold. The method according to claim 2,
Detecting whether the object touches the work surface comprises:
Determining whether the dimension of the object is increasing by comparing successive captured images. The method according to claim 1,
Determining an identifier of the selected key based on the first row histogram and the first column histogram,
Determining a center of the object by analyzing the subset of rows and the subset of columns;
Accessing a calibration table to determine coordinates of the center of the object; And
Determining an identifier of the selected key from the coordinates by accessing a key map. The method of claim 7,
Determining the center of the object,
Generating a second row histogram based on the subset of rows;
Generating a second column histogram based on the subset of columns;
Determining a first peak of the second row histogram; And
Determining a second peak of the second column histogram. A work area having a display of a keyboard having a plurality of keys;
An image sensor module configured to capture images of the working area, wherein each captured image comprises rows and columns of pixels; And a key selection module configured to analyze the captured images for variations in light intensity in the captured images, wherein the variations in light intensity are associated with an object in the work area. Including,
The key selection module,
Generate a first row histogram based on the light intensity of each of the rows in each of the captured images,
Generate a first column histogram based on the light intensity of each of the columns in each of the captured images,
Determine an identifier of a selected key of the plurality of keys based on the first row histogram and the first column histogram,
And output the identifier of the selected key to a host device. The method of claim 9,
The key selection module,
An object detection module configured to detect the object in the captured images;
A touch detection module configured to detect whether the object touches a work surface of the work area;
A location module configured to determine coordinates from the captured images corresponding to a location on the working surface touched by the object;
A key identification module configured to determine an identifier of the selected key based on the coordinates from the captured images; And
And a key event generation module configured to output a key event identifier to a host device associated with text input by a user. The method of claim 10,
The key event generation module,
A key event storage module configured to store a current key list and a previous key list; And
And a key block detection module configured to compare the current key list and the previous key list to identify a blocked key. The method of claim 10,
The object detection module,
Extract the background image from each captured image,
Determine a column moving average for the first column histogram and a row moving average for the first row histogram,
Determine segments of the first column histogram that are greater than the column moving average,
Determine a subset of columns associated with the segments of the first column histogram that is greater than the column moving average,
Determine segments of the first row histogram that are greater than the row moving average,
And determine a subset of rows associated with the segments of the first row histogram that is greater than the row moving average. The method of claim 10,
The apparatus is configured to provide a light beam extending on the working surface, the light beam configured to generate light intensity in the captured images when the object interacts with the light beam. The method according to claim 13,
The touch detection module,
Determine the dimension of the object in each captured image based on the subset of rows,
Further compare the dimension of the object to a thickness of the light beam. The method of claim 10,
And the touch detection module is further configured to count the number of consecutive captured images comprising the object and to determine whether the number of consecutive captured images is greater than a lower touch threshold and less than an upper touch threshold. The method of claim 10,
And the touch detection module is configured to compare successive captured images and determine if the dimension of the object is increasing. The method of claim 9,
The key selection module analyzes the subset of rows and the subset of columns, accesses a calibration table to determine coordinates of the center of the object, and accesses a key map to identify an identifier of the selected key from the coordinates. Configured to determine a system. 18. The method of claim 17,
The key selection module generates a second row histogram based on the subset of rows, generates a second column histogram based on the subset of columns, determines a first peak of the second row histogram, And determine a second peak of the second column histogram.

Images