DE20122526U1 - Virtual input device operation method for computer system, cellular telephone, involves sensing penetration of stylus into optical beam plane to detect relative position of stylus in the plane - Google Patents

Abstract

An optical system (20) emits an optical fan-shaped beam such that the plane (30) of the beam is parallel to and spaced from the planar work surface (40) of a virtual input device (50). Penetration of stylus into the beam plane (30) is sensed to determine the relative position of stylus in the plane. An independent claim is also included for virtual input system.

Description

REFERENCE ON PREVIOUS SUBMITTED APPLICATIONS

These Application claims priority to pending provisional U.S. patents. Patent application the applicant with the number 60 / 287,115, filed on April 27, 2001 was titled "Input Methods Using Planar Range Sensors ", pending U.S. provisional patent application 60 / 272,120, filed February 27, 2001, entitled "Vertical Triangulation System for a Virtual Touch-Sensitive Surface ", and pending provisional patent application 60 / 231,184, which issued on September 7, 2000 was filed with the title "Application of Image Processing Techniques for a Virtual Keyboard System. "Furthermore, this application is a continuation-in-part from the pending U.S. Patent Application No. 09 / 502,499, filed on Feb. 11 2000, entitled "Method And Apparatus for Entering Data Using A Virtual Input Device. "Each of these applications is hereby incorporated incorporated by reference.

AREA OF INVENTION

These The invention relates generally to a device for detecting or Capture an approach a pen or a user's finger relative to a device, to enter commands and / or data into a system or to this transferred to, and in particular, it relates to such detection relative to a virtual one Apparatus for entering or transmitting commands and / or Data and / or other information about a system is used.

BACKGROUND THE INVENTION

It is very common desirable, virtual input devices to use commands and / or data enter and / or other information about electronic systems transferred to, for example, to a computer system, a musical instrument, or even Phones. Although computer now realized in almost pocket size can be may be typing data or commands on a mini keyboard time consuming and error prone be. While Many mobile phones today can handle e-mail communication the actual Entering messages using the small phone touchpad to be difficult.

One For example, PDA has a lot of the functionality of a computer, but suffers on a small or nonexistent keyboard or keyboard. If a system could be used to determine if a user's finger or a pen a virtual keyboard touched, and which fingers touched which virtual keys on it could be the Output of the system maybe a PDA instead of the keyboard information be entered. (The terms "finger" or "finger" and "pen" are used interchangeably herein.) In this example could a virtual keyboard a piece Be paper, for example, that unfolds to the size of a keyboard, with buttons imprinted to guide the user's hands. It is understood that the virtual keyboard or other input devices easy a work surface represent and no sensors or mechanical or electronic Own components. The paper and the keys would not work indeed Enter information but the interaction or interface between the user's fingers and areas of the paper support. If no paper could be used Areas of a work surface, where keys would exist, to enter information into the PDA. A similar Virtual device and a system could be useful to email in mobile phones enter. A virtual piano-like keyboard could be used become a real musical instrument. The challenge However, it consists in detecting or detecting where the user's fingers or a pen relative to the virtual device.

US Patent 5,767,848 to Korth (1998) entitled "Method and Device For Optical Input of Commands or Data "tries to implement virtual devices by using a two-dimensional TV video camera is used. Such optical systems are based on lighting data and require a stable ambient light source, however, you can Unfortunately Illumination data disturb a picture system. For example, a User fingers in the picture foreground are indistinguishable of areas in the background. Furthermore, shadows or others could image-blocking phenomena, resulting from the virtual device obstructing user's hand result, the implementation of a Korth system in operation something imprecisely let appear. Korth would also the exam the contour of a user's finger, the finger position relative to the virtual device and a determination of finger movement require.

U.S. Patent No. ... by Bamji et al. (2001) entitled "CMOS Compatible Three-Dimensional Image Sensor IC ", Serial No. 09 / 406,059, filed September 22, 1999, discloses a sophisticated three-dimensional imaging system that uses virtual devices to input commands and data into electronic systems The image was captured three-dimensionally by using propagation time measurements A light source emits optical energy toward a target object, such as a virtual device, and can be used to capture the interface between a user fingertip and a virtual input device, such as a keyboard The energy reflected from areas of the object within the imaging path was detected by a group of photodiodes Using the various sophisticated techniques, the actual transit time between the emissions of the optical energy and their detection by the photod group of products. This measurement allowed the vector distance to the point on the target object to be calculated in three dimensions, for example (x, y, z). The described system tested the reflected emitted energy and could operate without ambient light. For example, if the target object were a computer keyboard layout, perhaps a piece of paper with imprinted keys, the system could detect which user finger touched which area on the target, for example which virtual key, in what order. Of course, the piece of paper would be optional and would be used to guide the user's fingers.

Three-dimensional Data obtained with the Bamji invention could by Software to locate the user's fingers, if they came into contact with a touch surface, for example a virtual one Input device. The software could touch the finger with a place on the surface as identify a request, a keyboard event of an application to enter on the linked electronic Device or system, for example, a computer, a PDA, a mobile device, a kiosk device, a Cash register device, etc. During the Bamji system worked and could be used to Enter commands and / or data into a computer system in which A three-dimensional image is used to interface of user fingers and a virtual input device to analyze is a less complex and perhaps less sophisticated one System desirable. Like the Bamji system, such a new system should be relatively inexpensive for mass production be and should consume relatively little energy, leaving a battery operation easily possible is.

The The present invention provides such a system.

SUMMARY THE PRESENT INVENTION

The The present invention locates an interaction between a User finger or a stylus and a passive touchpad (eg. a virtual input device) that defines above a work surface is by using a planar quasi three-dimensional detection becomes. The quasi-three-dimensional detection implies that the Determination of an interaction point essentially in three dimensions accomplished can be made by using a two-dimensional surface as reference which is arbitrarily oriented in three-dimensional space. As soon as a touch the invention locates the touch area to determine where the touch took place on a virtual input device, and what data or command key sequences, according to the localized Area that touches was, in response to the touch accomplished shall be. Alternatively could the virtual input device is a virtual mouse or a trackball include. In such an embodiment would the vorlie ing invention, the coordinates of the touch point with the virtual Capture input device and report which coordinates with linked to an application are, perhaps a cursor on a display (in a virtual mouse or a trackball implementation) and / or so-called digital Ink for to be able to apply a drawing or a writing application (a virtual filler or a pen application). In various embodiments, preferably Triangulation analysis method used to determine where a "touch" of user and Object with the virtual input device occurs.

In a so-called structured light embodiment, the invention includes a first optical system (OS1) that generates a plane of optical energy that produces a fan beam of beam angle φ parallel to and a small distance ΔY above the work surface on which the virtual Input device could be defined. In this embodiment, the plane of interest is the plane of the light generated by OS1, typically a laser or an LED light generator. The two parallel planes may typically be horizontal, but may also be vertical or have any other arbitrary and suitable angle. The invention further includes a second optical system (OS2) responsive to optical energy of the same wavelength as emitted by OS1. Preferably, the OS2 is located above OS1 and angled at an offset angle of θ relative to the fan beam plane, toward the area where the virtual input device is defined. OS2 responds to energy coming from OS1 is emitted, but the wavelength of the optical energy does not have to be visible to people.

The Invention can also be implemented by configurations be used with unstructured light that is active or can be passive. In a passive triangulation embodiment, OS1 is a camera and not an active source of optical energy, and OS2 is one Camera that responds to the same optical energy as OS1 and preferably arranged as described above. In such an embodiment the plane of interest is the projection plane of a scan line the OS1 camera. In one embodiment with unstructured light, such as an active triangulation embodiment OS1 and OS2 cameras and the invention further comprises an active Light source that emits optical energy at wavelengths to the OS1 and OS2. Optional in such an embodiment can OS1 and OS2 each have a shutter mechanism or shutter or Cover mechanism that is synchronized to the active Output light source such that the shutters in OS1 and OS2 are open, when optical energy is emitted and the otherwise closed are. An advantage of a configuration with unstructured light, The two cameras used is that bumps or irregularities on the work surface be better tolerated. The level defined by OS1 can be selected by using a matching set of OS1 capture pixel elements chosen is going to be with the highest Point in the y-dimension (eg a hill) of the work surface to match.

at the embodiment with structured light OS2 will not detect the optical energy until an object, for example a user's finger or a pen, begins, the worktop area to touch, on which the virtual input device is defined. As soon as, however the object penetrates the plane of the optical energy coming from OS1 is emitted, the area of the finger or the pen, which cuts the plane, be illuminated (visible or invisible for the User). OS2 detects the cutting or crossing of the level of interest by detecting the optical energy, which reflects in the direction OS2 by illuminating the object area becomes. Essentially, only one level is for the present invention of interest, which is determined by the configuration of OS1, and all other planes parallel in three-dimensional space could be defined to the virtual input device, as ignored irrelevantly. Thus captures a planar three-dimensional Sensor system user interactions with a virtual input device, which occur in the plane of the emitted fan beam, and ignores any interaction with other layers.

On this way detects or recognizes the present invention, that an object has touched the virtual input device. By detecting that a relevant touch intersection occurred Then, the invention locates the location in two dimensions the touch at the virtual device level. In the preferred implementation can localized operations identifying which virtual keys are on a virtual computer keyboard or a music keyboard from the user be touched. The user can touch more than one virtual key at a time, e.g. the "Shift" key and another Button. It should also be noted that the chronological order the touches determined by the present invention. If so, the user touched virtual keys for "shift" and "t" and then for the letters "h" and then "e", becomes the present Invention recognize that "T" then "h" and then "e", or "The" was entered. It is understood that the present invention is not limited to Ambient light supports and that they are also full in the absence of ambient light functioning is assuming that the user is the location of the virtual Input device knows.

Structured light and / or unstructured light passive triangulation techniques may be used to determine a touch point (x, z) between a user's hand and the detection plane. Since the baseline distance B between OS1 and OS2 is known, a triangle is formed between OS1, OS2 and the point (x, z) whose sides are B, and the projection beams R1 and R2 from OS1, OS2 to (x, z). OS1 and OS2 allow the determination of a triangle angle distance from a reference plane and the angles α ₁ and α ₂ formed by the projection beams, and the trigonometry results in the distance z to the surface point (x, z) as well as the projection beam lengths.

A processor unit associated with the present invention executes software to identify each intersection of a user-controlled object with the virtual input device and determine therefrom the correct user-desired input data and / or commands, preferably by using a triangulation analysis. The data and / or commands may then be input by the present invention as inputs to a device or system for which the virtual input device is being used. If desired, the present invention may be implemented within the companion device or systems, in particular in PDAs, cell phones and other small scale devices or systems which often lack a large user input device such as a keyboard.

Other Features and advantages of the invention will become apparent from the following Description in which the preferred embodiments are set forth in detail, in conjunction with the accompanying drawings.

SHORT DESCRIPTION THE DRAWINGS

1A Fig. 12 shows a planar, quasi-three-dimensional, structured light detection system used to capture user input on a virtual input device in accordance with the present invention;

1B shows a planar quasi-three-dimensional detection system with unstructured active light used to detect user input on a virtual input device according to the present invention;

1C shows a planar quasi-three-dimensional detection system with unstructured passive light used to capture user input on a virtual input device according to the present invention;

2A shows the geometry associated with location determination using triangulation according to the present invention;

2 B shows the use of an optical emitter and a reflector spaced therefrom, as a first optical system, according to the present invention;

3A - 3E show design tradeoffs associated with varying orientations of the OS2 sensor, the OS2 lens, and the detection plane with effective field of view and image quality, according to the present invention;

4 FIG. 4 is a block diagram illustrating functions performed by a processor unit in the example system of FIG 1B be executed, according to an embodiment of the present invention;

5A shows an embodiment in which the virtual device has four user-selectable areas and the companion device is a monitor according to the present invention;

5B shows an embodiment in which the virtual device is a computer keyboard and the accompanying device is a mobile radio transmitter / receiver according to the present invention;

5C shows an embodiment in which the virtual device is mounted or projected on a wall and the companion device is a monitor according to the present invention;

6 shows a planar distance measurement according to the present invention; and

7 FIG. 12 shows coordinate range measurements used in an example calculation of a touch location for use in outputting corresponding information or data or instructions according to the present invention. FIG.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

1A shows a preferred embodiment of a quasi-planar three-dimensional detection system 10 with - in one embodiment with a structured light system - a first optical system (OS1) 20 which has a fan beam plane 30 of optical energy parallel to a planar work surface 40 emits on which work surface a virtual input device 50 and or 50 ' and or 50 '' is defined. Preferably, the fan beam defines a fan angle φ and is spaced from the work surface by a small distance distance ΔY. An object (such as a user's finger or pen) attempting to touch the work surface must first touch the fan beam and thereby illuminate with the emitted optical energy (visible or invisible). While the fan beam level 30 and the Ar beitsflächenebene 40 arranged horizontally in 1A 2, these two planes can also be arranged vertically or actually at any other angle desired for a system. It should be noted that without limitation a work surface 40 a part of a desk, a table surface, a part of a vehicle, for example a tray in an airplane, a windscreen or a dashboard, a wall, a display with a projected image or a display, such as a CRT, an LCD could. Herein, the term "level" is understood to include a subset of an entire level, for example, the fan beam level 30 as a plane, although it has a finite width and does not extend in all directions at last.

By "virtual input device" is meant an image of an input device on the work surface 40 may be present, perhaps by placing a paper bearing a printed image, or perhaps by a system 10 that projected a visible image of the input device onto the work surface, or effectively, there is no any visible image on the work surface 40 gives. The virtual input device 50 . 50 ' . 50 '' does not require any mechanical parts, such as control buttons, and does not need to respond to the touch of a finger or stylus; In short, the virtual input device is preferably passive.

In the example of 1A is the virtual input device 50 a computer keyboard that can be full size or enlarged or reduced in size compared to a commonly dimensioned keyboard. If desired, the virtual input device may be a virtual trackball 50 ' and / or a virtual touchpad 50 '' contain or include. If the system 10 with a virtual keyboard input device 50 , or a virtual trackball 50 ' or a virtual touchpad 50 '' is used, a fan angle φ of about 50 ° to 90 ° and preferably about 90 ° ensure, so that the fan beam 30 the entire virtual input device is surrounded in commonly used distances. In such a virtual input device, a distance distance .DELTA.Y of up to a few mm, preferably about 1 mm has proven to be very good.

The system 10 further comprises a second optical system (OS2) 60 typically a camera with a planar sensor, preferably spaced from and above OS1 20 is arranged and towards the work surface 40 and the plane 30 is inclined at an angle θ, from about 10 ° to about 90 °, and preferably about 25 °. The system 10 further comprises an electronic processing system 70 which monitors among other tasks OS1 and OS2. The system 70 preferably comprises at least one central processing unit (CPU) and associated memory, which may comprise read-only memory (ROM) and random access random access memory (RAM).

In 1A are system 10 elements OS1 20 , OS2 60 and the processor unit 70 being shown on or in a device 80 are arranged. The device 80 can be considered a stand-alone implementation of the system 10 or may actually be a system or device for which the virtual input device 50 is used to enter data or commands. In the latter case, the device 80 without limitation, a computer, a PDA (as in 1A shown), a mobile phone, a musical instrument, etc. If the system or device 80 is not controlled by the virtual input device, the device 90 thus controlled electrically with the system / device 80 be coupled to data and / or command inputs from the virtual device 50 to recieve. If the virtual device is a trackball (or mouse) 50 ' or a touchpad 50 '' For example, with such a virtual device, user interaction may directly output raw information or data containing touch coordinates (x, z) for use by the device 80 include. For example, the user interaction with the virtual input device 50 ' or 50 '' a repositioning of a cursor 160 on a display 140 or could change an application by the device 80 or could be a location for a so-called digital ink 180 Following this, what a user "writes" by using a virtual mouse or trackball 50 ' is used, or a pen 120 ' is used or a virtual touchpad 50 '' , The system / device 90 can be electric with the system 80 about a medium 100 be coupled, which may include without limitation lines or may be wireless or may be a network including the Internet.

In one structured light embodiment, OS1 emits 20 optical energy in a fan beam 30 parallel to the xz plane 30 , OS1 may have a laser line generator or LED row generator, although other optical power sources could be used to make the plane 30 radiate. A row generator OS1 is named so because it emits a plane of light which, when intersected by a second plane, illuminates what OS2 would see as a line on the second plane. For example, if a cylindrical object is level 30 OS2 would light this process as one See section of an elliptical arc whose aspect ratio would depend on the distance of OS2 above the plane 30 and the surface 40 , Thus, the detection by OS2 detects an elliptical arc on the plane 30 a touch operation, for example, that an object, such as. 120R the level 30 touched or penetrated, with ambient light is excluded. Although a variety of optical emitters are used, a laser diode that outputs about 3 mW average power with a wavelength of about 300 nm to perhaps 1000 nm could be used. While ambient light wavelengths (eg, about 350 nm to 700 nm) could be used, the effects of ambient light could be minimized without filtering or without shutters if such wavelengths are avoided. Thus, wavelengths from about 600 nm (visible red) to about 1000 nm (deep infrared) could be used. A laser diode that outputs optical energy at a wavelength of 850 nm would be an economical emitter, although OS2 would preferably include a filter to reduce the effects of ambient light.

While OS1 is preferably stationary in a structured light embodiment, it will be understood that a fan beam 30 could be generated by mechanically panning a single emitted line of optical energy to the fan beam plane 30 define. As in 2 B OS1 can actually be an optical energy emitter 20-A comprising a fan beam and a reflecting mirror 20-B , the fan beam 30 essentially parallel to the surface 40 directed. For the present invention, in the structured light embodiment, the optical energy provided by OS1 20 is emitted, visible or invisible to humans. OS2 60 preferably includes a camera system responsive to optical energy of the wavelength selected by OS1 20 is emitted. By "responsive" it is meant that OS2 recognizes energy of the same wavelength emitted by OS1, and ideally does not recognize or respond to energy that differs substantially from the wavelength, for example OS2 may include a filtering system such that optical Energy of the wavelength, which is not the wavelength emitted by OS1, is not detected, for example, a color filter.

If desired, OS2 could essentially only respond to optical energy emitted by OS1 by synchronously turning OS1 and OS2 on and off simultaneously, for example, under the control of the unit 70 , OS1 and OS2 would preferably include apertures, or shutter mechanisms, as elements 22 are shown that functionally open and close in a synchronous manner. The electronic processing system 70 could, for example, synchronously OS1, OS2 or the shutter mechanisms 22 for a time period t ₁ with a desired duty cycle, where t _{1 is in the} approximate range of about 0.1 ms to about 35 ms, and would then turn off OS1 and OS2. If desired, OS1 could always be operated when the plane 30 only can radiate when the shutter 22 before OS1 20 is open. In the various shutter configurations, the repetition rate of the synchronous switching is preferably about 20 Hz to about 300 Hz to support an adequate image data acquisition speed. To save on operating power and reduce computational overhead, a repetition rate of about 30 Hz to 100 Hz is an acceptable speed. Of course, other devices and methods may also be used to ensure that OS2 is essentially responsive only to optical energy coming from OS1 is emitted. To simplify the illustration, the closures 22 represented as mechanical elements, but in practice becomes the concept of closures 22 understood to include on and off switching light sources and cameras in any variation.

If desired could a source or sources of optical energy used in the present Be used in the invention, a so-called. Signature to carry to enable that such energy better distinguished from the ambient light energy can be. For example, could Such sources are modulated with a fixed frequency, so that Cameras or other sensor units used in the present invention can be used to more easily detect such energy while the Ambient light energy due to the lack of such a signature essentially rejected would be this example is not limiting. In short, signature techniques, such as the wavelength selection for optical Energy that differs from ambient light, techniques that include synchronous operation of the light sources and the camera sensors, and modulated or otherwise labeled light source energy, respectively the signal / noise ratio to improve the information obtained from the present invention becomes.

It should be noted that there is no requirement or need for the work surface 40 reflective or non-reflective with respect to the wavelength, which wavelength is emitted by OS1, since the fan beam or any other emission of optical energy does not reach the surface as such. It should also be noted that the virtual input device is preferably completely passive. Because the device 50 is passive, it can be sized in size so that it is smaller than a full dime sited device, if necessary. Further, the cost of a passive virtual input device is zero, especially if the "device" is simply a piece of paper carrying a graphical image of a current input device.

In 1A Let's first assume that the user of the system 10 not near the virtual input device 50 is. In a structured light embodiment, OS2 detects nothing, although OS1 is a fan beam plane of optical energy 30 can emit, since no object is the plane 30 cuts. It is now assumed that a section 110 a finger of the left or right hand 120L . 120R of a user moves down to a portion of the area of the work surface 40 to touch on the virtual input device 50 is defined. Alternatively, an area could be 110 ' a user-controlled pen 120 ' to be moved down to a relevant area of the work surface 40 to touch. Within the context of the present invention, a touch is interpreted by software associated with the invention as a request to send a keyboard event to an application on an accompanying device or system 80 or 90 , For example, a notebook, a PDA, a mobile device, a kiosk or booth device, a cash register device, etc. expires.

In 1A as the user's finger moves down and begins, the optical energy plane 30 to cut, an area of the fingertip that faces OS1, now optical energy 130 reflect. At least some of the reflected optical energy 130 is detected by OS2 since the wavelength of the reflected energy is the same as the energy emitted by OS1 and OS2 responds to energy of such wavelength. Thus, the planar quasi-three-dimensional detection system captures 10 optical energy reflected by the interaction of a user-controlled object (eg, a finger, a stylus, etc.), the interaction occurring at a plane of interest, that of the fan beam plane 30 is defined. Any interaction that occurs at any other level is considered irrelevant and may be ignored by the present invention.

Thus, an object, such as an area of a user's hand or perhaps a pen, is the optical energy plane 30 thus, there is no reflected optical energy, which is emitted by OS1 130 that by OS2 60 can be detected. Under such conditions the system knows 10 that no user input is made. However, once the optical energy plane is penetrated, the incision of the penetrating object (eg, fingertip, pen tip, etc.) from OS2 60 and the location (x, z) of the intrusion can be detected by the processor unit 70 be determined with the system 10 is linked. In 1A can, if the left index finger of the user the area of the virtual input device 50 which is defined with the coordinates (x7, z3) touches, then the software associated with the invention determines that the letter "t" has been "pressed". Since no "shift key" was also pressed, the pressed letter would be understood as "t".

In the embodiment shown, the system 10 Generate key sequences and the system 80 or 90 which represent data and / or commands that a user would have typed on an actual keyboard. Such input into a system 80 or 90 Can be used for information 140 on a display 150 represent when the information from the user on a virtual input device 50 is entered. If desired, an enlarged cursor area could 160 can be implemented to provide additional visual input to help the user entering information. If desired, the processor unit could 70 the system 80 and or 90 cause audible feedback to help the user, such as an electronic key click sound 170 The "pressing" of a virtual key of the virtual input device 50 equivalent. It is understood that - if the system 80 or 90 a musical instrument and not a computer or a PDA or a mobile device would be - musical tones 170 would be emitted, and the virtual input device 50 instead, it could have the structure similar to a piano keyboard or keyboard linked to synthetic music generators.

1B shows an unstructured active light system 10 in which a camera 20 ' in a first optical system OS1, a plane of interest 30 ' essentially defines the level 30 replaced by the optical emitter OS1 in the embodiment of 1A is defined. The camera 20 ' OS1 is preferably similar to the camera 60 OS2, which is similar to the camera 60 OS2 in the embodiment of 1A can be. For example, OS1 20 ' a sensor array comprising at least one row and preferably a plurality of rows of pixel detector elements. The embodiment of 1B is active insofar as one or more light sources 190 that between OS1 20 ' and OS2 60 are arranged to generate optical energy of a wavelength from the camera OS1 20 ' and from the camera OS2 60 is detectable. To reduce the effects of ambient light on the detection by the cameras OS1 and OS2, Preferably every camera and every optical energy emitter works 190 together with a shutter mechanism, preferably synchronized, for example by the unit 70 , During the times when the closures 22 radiating optical energy from the emitter 190 towards the virtual input device 50 . 50 ' . 50 '' allow, become similar closures 22 allow the cameras OS1 and OS2 to capture optical energy. The interaction of the user object, eg. 120L with the plane 30 ' is detected by OS1 and by OS2. The location of the point of intersection is then calculated, for example by using triangulation methods which will be described later.

In 1B is a hill or an irregularity in the plane of the work surface 40 near the point of contact 110 with the user object 120L shown. An advantage of the existing two th camera OS1 20 ' lies in the fact that the level of interest 30 ' can be selected, perhaps through unity 70 to be directly above the highest irregular area of the work surface 40 to lie. If irregularities in the work surface 40 in the embodiment of 1A would be necessary, it would be necessary to reposition the laser plane 30 relative to the work surface. But in 1B For example, the effect of such repositioning is accomplished electronically simply by selecting a suitable row of pixels from the detector array at OS1 20 ' ,

It should be noted that the configuration of 1B suitable for various methods to improve the signal-to-noise ratio. For example, it may be the closures 22 allow the cameras OS1 and OS2 to collect imaging data for a time to which the emitter 190 are switched off, for example via the control unit 70 , Any imaging data captured by OS1 and / or OS2 will represent the background noise resulting from the ambient light. (Again, it should be understood that to minimize the effect of ambient light, the emitters 190 and the cameras OS1, OS2 preferably operate with a wavelength specification that differs from the ambient light.) If the one detected is what is called a background noise signal, the cameras OS1 and OS2 can now be normal and synchronous with the emitters (n) 190 work. Mapping data from the OS1 and OS2 cameras in sync with the emitters 190 are captured, current data will include, for example, the user interface with the level 30 ' plus any (unwanted) effects due to ambient light. The processor unit 70 (or another unit) can then dynamically subtract the background noise signal from the current data plus the noise signal to arrive at a current data signal, thus improving the signal-to-noise ratio.

1C shows a non-structured passive embodiment of the present invention. The system 10 in 1C is passive, no matter which ambient light source 195 is present, and provides optical energy used during the imaging. Similar to the system 10 in 1B OS1 is a camera 20 ' representing a level of interest 30 ' defined, and OS2 is a camera 60 , Typically, the level becomes 30 ' a distance ΔY 'above the work surface 40 define, typically at a distance of a few mm. A user-object interaction with the layer 30 ' is detected by OS1 and OS2 by taking optical energy from the ambient light source 195 is used. Triangulation methods can then be used to determine the point of interaction or the intersection with the plane 30 ' to locate, as described elsewhere here.

2A shows the geometry with which the location (x, z) of the intersection between a user's finger or object 120R and the plane 30 can be determined by using the triangulation method. 2A and 2 B can be used to describe the analysis of the various embodiments that appear in 1A - 1C are shown.

As noted below, triangulation helps to determine the shape of surfaces in a field of view of interest by geometrically analyzing the triangles that are projected from the projection beams, e.g., R1, R2, onto two optical systems, for example, OS1 20 , OS2 60 be projected. A baseline B represents the known length of the line connecting the centers of the projection of the two optical systems OS1, OS2. For a point (x, z) on a visible surface in the field of view of interest, a triangle may be defined by the location of the point and by the locations of OS1 and OS2. The three sides of the triangle are B, R1 and R2. OS1 and OS2 can determine the angular distance of the triangle from a reference plane, and the angles α ₁ and α ₂ formed by the projection beams connecting the surface point to the centers of projection of the two optical systems. Angle α ₁ and α ₂ and the baseline B completely determine the shape of the triangle. A simple trigonometry can be used to calculate the distance to the surface point (x, z) as well as the length of the projection beam R1 and / or R2.

It is required that OS1 20 is implemented as a single unit. For example, shows 2 B a structured light embodiment in which the first optical system bifurcates: a region OS1-A 20-A is a light emitter located at the distance B from OS2 and from the second area OS1-B 20-B is arranged, a light-reflecting device, such as, a mirror. The incoming fan beam generated by OS1-A is reflected by the mirrors 20-B distracted to the level 30 to build. When aligning 2 B is the mirror 20-B obliquely at about 45 ° relative to the horizontal plane, and the deflection is substantially at a vertical plane to a substantially horizontal plane. In 2 B actually becomes a passive light OS2 60 a camera directed at an angle φ generally in the direction of the field of view of interest, namely where a user's finger or pen will use a virtual input device that is below the fan level 30 is arranged.

The triangulation according to the present invention preferably uses a standard camera with a planar sensor as OS2 60 , The nature of OS1 20 distinguishes between two fairly broad classes of triangulation. In a structured light triangulation, OS1 is 20 typically a laser or the like, whose beam can be shaped as a single line that is moved to project a point of movement onto a surface. Alternatively, the laser beam may be planar and moved to project a planar curve. As noted, another class of triangulation system may be referred to as passive triangulation, in which a camera is called OS1 20 is used. Structured light systems tend to be more complex to have more power and consume due to the need to project a plane of light. Passive systems are less expensive and consume less energy. However, passive systems must solve the so-called correspondence problem, for example, to determine which pairs of points in the two images are projections of the same point in the real world. As will be described below, passive non-structured light triangulation embodiments according to the present invention may be used.

Whether the system 10 As a structured light system in which OS1 actively emits light and OS2 is a camera, or as a passive system in which OS1 and OS2 are both cameras, information from OS2 and OS1 is coupled to a processing unit, e.g. 70 , which can then determine which events occur. If an object, eg. 120R , the projection plane 30 cuts that with OS1 20 is linked, the section with each embodiment is detectable. In a structured light embodiment, where OS1 emits optical energy, the cut is detected by optical energy from that of the intersecting object 120R reflected and detected by OS2, typically a camera. In a passive light embodiment, the section is seen by OS1, a camera, and also by OS2, a camera. In each embodiment, the intersection with the plane 30 captured as if the area of the area 40 below the (x, z) plane intersection of the object 120R would be touched. The system 10 preferably comprises a computer system 70 which receives data from OS1, OS2 and uses the geometry to determine the position of the plane intersection (x, z) from the reflected imaging coordinates in a structured light embodiment, or from the camera imaging coordinates in a passive system. As such, the dual task of initially detecting and continuously detecting touches and penetrating the plane 30 (eg, touch events) and determining the intersection coordinate positions on the plane thereby.

To summarize so far, touch events are captured and declared when OS1 intersects the plane 30 by a penetrating object, such as. 120R recognizes. In a two-camera system, a relationship is built up between the dots in the obtained image of OS1 and that of OS2. Thereafter, the coordinates of the OS2 camera are converted into touch-area coordinates (x-axis, z-axis) around the (x, z) coordinate position of the event within the area of the plane 30 of interest. Preferably, such conversions are performed by the processor unit 70 executed, which executes algorithms to the cutting positions in the plane 30 to compute from map coordinates of the visible points of OS2. Furthermore, a passive light system must distinguish intruding objects from the background in images of OS1 and OS2. If the system 10 is a passive light system, appropriate requirements must be made between the images of the camera OS1 and the camera OS2. If the system 10 is a structured light system, it is desirable to minimize the interference from ambient light.

It is now the calculation of the (X, Z) intercept or fingertip position on the plane 30 considered. In a perspective projection, a plane in space and its image are related by a transformation called homography. Let a point (X, Z) on such a plane be represented by the column vector P = (X, Z, 1) ^T , the superscript T designating transposition. Similarly, let the corresponding mapping point be represented by p = (x, z, 1) ^T.

A Homography is then a linear transformation P = Hp, where H a 3 × 3 Matrix is.

These Homography matrix can be found in which a calibration procedure is used. As the sensor rests on the surface, the sensor position is relative to the area constant, and the calibration procedure needs to be done only once. For calibration, a grating with a known grid on a flat area placed on which the sensor rests. The coordinates Pi of the map points, which correspond to the grid vertices, Pi, are shown in the figure measured. A direct linear transformation (DLT) algorithm can be used to determine the homography matrix X. Such DLT conversion is known in the art; see. For example, Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision, Cambridge University Press, Cambridge, UK, 2000.

Once H is known, the area point P corresponding to a point p in the map is immediately calculated by the matrix vector multiplication as previously stated. Preferably, such calculations are made by the system 70 executed.

An imaging match for passive light embodiments will now be described. The cameras OS1 20 and OS2 60 see the same plane in the room. As a result, the mapping of the image of the line scan camera from OS1 and the camera image from OS2 itself will be a homography. This is similar to the mapping between the OS2 camera image and the plane-30 touchpad as previously described with reference to the fingertip intersection position calculation. Thus, a similar procedure can be used to compute this mapping.

It should be noted that since the line scan camera OS1 20 In essence, the contact area collapsed to a single line sees or touches, the homography between the two images is degraded. For each OS2 camera point there is an OS1 line scan imaging point, but for each OS1 line scan imaging point there is a complete line of OS2 camera points. Due to this degeneracy, the previously described DLT algorithm is (slightly) modified to obtain a point-to-line match. By definition, a passive light embodiment of the present invention has no control over the ambient lighting and this may make it difficult to distinguish between penetrating cut objects or peaks from the general background. In short, how to determine if a particular map point in an OS1 map or OS2 map represents a map of a point on an object, such as a map. 120R or a point is generally background. An algorithm used by the system 70 is executed, will now be described.

Initially, one or more background images I ₁ ,..., I _{n are} assumed, wherein only the contact surface area of the plane 30 in sight. It is assumed that the cameras OS1 and OS2 may be responsive to color and it should R _bi (x, z), G _bi (x, z), B _bi (x, z), the red, green and blue component of the Background image intensity I _i at the pixel position (x, z). Let s _b (x, z) be a summary of R _bi (x, z), G _bi (x, z), B _bi (x, z) over all mappings. For example, s _b (x, z) may be a tri-vector with the mean, central, or other statistic values of R _bi (x, z), G _bi (x, z), B _bi (x, z) at the pixel position ( x, z) over all background images I ₁ ,..., I _n , normalization being carried out as far as possible in order to compensate for variations in the image brightness.

Next, a similar summary s _t for the tip pixels is collected over a new series of maps J ₁ , ..., J _m . This second sum is a single vector and not a mapping of vectors, as for s _b (x, z). In other words, s _t does not depend on the pixel position (x, z). For example, this new sum may be calculated by asking a user to place fingertips or a stylus in the sensitive area of the area and by recording values only at pixel locations (x, z) whose color is different from the sum of the background s _b (see FIG. x, z) at the position (x, z) and in which statistics on all values of j, x, z are calculated.

Given a given new mapping with color components c (x, z) = (R (x, z), G (x, z), B (x, z)) then a particular pixel contributes to (x, z), either be assigned to the top or the background by an appropriate rule of discrimination. For example, a distance d (c ₁ , c ₂ ) between three vectors (Euclidean distance is an example) can be defined, and pixels are assigned based on the following exemplary rule:
If d (c (x, z), s _b (x, z)) d (c (x, z), s _t ).
If d (c (x, z), s _b (x, z)) d (c (x, z), s _t ).
Unknown otherwise.

techniques for reducing ambient light interference, in particular for the embodiment with Structured light triangulation will now be described. At a such embodiment OS2 must differentiate between ambient light and light emitted by the row generator was generated and by a penetrating object reflected back has been.

at Using a first method, OS1 emits energy in one area the light spectrum in which ambient light has little energy, eg. in the near infrared range. An infrared filter on the camera OS2 can guarantee, that the light detected by the OS2 sensor is mainly the reflected from the object (e.g., 120R) into the lens of the camera OS2 is.

at In a second method OS1 operates in the visible region of the spectrum, but is much brighter than ambient light. Although this is in the Principle with every color of the light source can be achieved, it is for applications more useful in the house, a blue-green light source for OS1 (500 nm to 550 nm) to use, as the standard fluorescent lamps have relatively little charisma in this band. Preferably OS2 have a matched filter to ensure that a response to other wavelengths essentially subdued becomes.

A third method of reducing the effects of ambient light uses a standard visible laser source for OS1 and a color camera sensor for OS2. This method uses the same background subtraction algorithm as previously described. Define the following combination using the same terminology as before:
C (x, z) = min ⎨ d (c (x, z), s _b (x, z)), d (c (x, z), s _t ) ⎬.

This combination will be exactly zero if c (x, z) equals the peak sum s _t (since in d (s _t , s _t ) = 0) of the representative object and for the background map s _b (x, z) (since d (s _b (x, z), s _b (x, z)) = 0), and close to zero for other object pit mapping areas and for visible parts of the background. In other words, the object peaks and the background in C (x, z) will hardly be visible. By comparison to positions where the projection plane 30 from the laser emitter OS1 the object peaks 120 clearly, the term d (c (x, z), s _t ) will not be zero, again resulting in a substantially non-zero value for C (x, z). This approach achieves the desired goal of identifying substantially only the object peak pixels that are being irradiated by the laser (or other emitter) OS1. This method can be varied to use light emitters of different colors, to use other distance definitions for the distance d, and to use different sums s _b (x, z) and s _t .

Referring to 1A if a device 80 For example, a compact system, such as a PDA or a mobile device, would be desirable to reduce the size required to implement the present invention. A smaller overall form factor may result if OS2 is at a certain angle θ, as in 1A - 1C . 2A . 2 B shown with respect to the plane 30 or the area 40 is inclined. But when the angle θ decreases, the camera OS2 sees the plane 30 from a shallower angle. For a fixed size of the touch-sensitive area of the plane 30 That is, the rectangle area to be "touched" by a user object to manipulate an underlying virtual input device will decrease the effective area surrounded by the field of view as distance B and angle θ decrease in that the effective OS2 resolution decreases and thus the accuracy of z-depth measurement, as in 3A where L denotes a camera lens associated with OS2 whose plane is shown by pixel detectors as a straight line labeled OS2.

As in 3A indicated, OS2 moves closer to the plane 30 to a shallower viewpoint and to a smaller, less accurately captured camera image. These adverse side effects can be reduced as in 3B is shown by tilting the plane of the pixel detectors in the camera OS2, being nearly parallel to the plane 30 is tilted. In the tilted configuration of 3B It should be noted that a much larger number of imaging scan lines will obscure the cone of radiation from the sensitive area on the plane 30 cut, which increases the depth resolution accordingly. For example, let's consider the relatively small distance Dx in 3A with the big distance Dx 'in 3B which represents the large number of image scan lines now in use. If the OS2 camera sensor plane is more parallel to the plane of the touch surface or the plane 30 this leads to less disturbance of the touch area image. This implies that parallel lines on the touch surface (or plane 30 ) will remain in parallel in the OS2 camera image. One advantage is the simplicity of homography H for affine transformation (a move and a scale). Further, the imaging resolution becomes more uniform over the entire sensitive area within the field of interest made.

It is now the configuration of 3C considered. It can be seen that different points on the touch-sensitive area at the level 30 are at different distances from the lens L of the camera OS2. This means that one can not precisely focus the entire sensitive area of interest if the lens L is as in 3A or in 3B is shown positioned. While closing the camera aperture could increase the depth of field of view, ie, depth of field, the resulting images would be less bright and the image signal to noise ratio would be degraded.

Accordingly, the configuration of 3C be used with the lens L relative to 3B is repositioned. In this configuration, the touch area 30 , the camera OS2 sensor and the camera lens L being regarded as fulfilling the so-called Scheimpflug condition, in which their respective planes intersect along a common line, a line which in 3C lies at infinity. Further details on the Scheimpflug condition can be found in The Optical Society of America, Handbook of Optics, Michael Bass, Editor in Chief, McGraw-Rill, Inc., 1995. Referring to the 3C When the relevant optical system satisfies this condition, all points on the touch surface become 30 in focus. By using a properly tilted sensor OS2 and a properly positioned lens S satisfying the Scheimpflug condition, the image seen by OS2 becomes the points of interest at the surface plane 30 be in focus and will show a high resolution with low distortion. However, the fulfillment of the Scheimpflug condition may result in a loss of imaging brightness because of the configuration of 3B the angle the lens occupies when viewed from the center of the sensitive area on the plane 30 considered reduced. As a result, in some applications, it is preferred to achieve a trade-off between focal sharpness and imaging brightness by bringing the OS2 camera lens into alignment between that of 3B and those of 3C lies. 3D shows such an intermediate configuration in which the lens L is conveniently slightly away from the Scheimpflug-fulfilling orientation with respect to the planes of OS2 and 30 is tilted.

Such intermediate alignments do not satisfy the Scheimpflug condition, but only to a small degree, and therefore exhibit good focusing properties over a configuration whose lens axis is directly towards the center of the sensitive area of the plane 30 is directed. 3E Figure 11 shows another alternative intermediate configuration, one in which the Scheimpflug condition is exactly met, but the camera sensor OS2 is tilted away from the horizontal. The configuration of 3E can achieve an exact focus, but with slightly lower image resolution and more interference than in the configuration of 3C ,

4 Figure 12 is a block diagram illustrating effective areas of the processor unit 70 within the system 10 Fig. 12 shows which processor unit preferably executes the various triangulation and other calculations described herein about (x, z) intersections with the plane of interest 30 to capture and identify. As left area of 4 gets information from OS1 20 and OS2 30 each entered in pixel tables 200-1, 200-2. In 4 The OS1 and OS2 inputs refer to a stream of images of the digitized images obtained from the optical system 1 (FIG. 20 ) and the optical system 2 ( 60 ) in a planar range sensor system 10 produced according to the present invention. In a preferred embodiment, the optical system generates at least about 30 frames per second (fps). Higher frame rates than 30 fps are desirable because the user's or pen's fingertip can move several pixels between two images while "typing" on a virtual input device. Pixel image Module 200-1 . 200-2 Build digital images from OS1 and OS2 in memory using the calculation unit 70 is linked. A synchronization module 210 ensures that the two optical systems produce images of the digitized images in approximately the same time. If desired, a double-buffering system may be implemented to allow the construction of an image while the previous image is processed (in time) by other modules. A touch detection module 220 detects a touch (for example, that the finger of a user or pen penetrates the optical plane detected by OS1) when the outline of a fingertip or a stylus appears in a selected line of the image. When a touch is detected, the tip detection module draws 230 the outline of the corresponding fingertip in the appropriate pixel table 200-1 or 200-2 on. In 4 No table is generated in a structured light embodiment where OS1 is a light beam generator, and touch detection will use the input signal from OS2, not OS1.

The touch position module 240 uses the peak pixel coordinates from the peak Erfas sungsmodul 230 at the time when a touch from the touch sensing module 220 is reported to find the (xz) coordinates of the touch on the touchpad. As indicated, a touch is equivalent to a level penetration 30 that with an optical emitter OS1 in one embodiment associated with structured light, or in a passive light embodiment associated with a viewing plane of a camera OS1 is linked. Mathematical methods for converting the pixel coordinates to the XZ touch position are described elsewhere herein.

A key identification module 260 uses the XZ position of a touch and maps the position to a key identification in which a keyboard layout panel 250 is used, preferably stored in memory, with the calculation unit 70 is linked. The keyboard layout panel 250 typically defines the upper / lower / left and right coordinates of each key relative to a zero origin. As such, a function of the key identification module 260 , carrying out a search in the blackboard 250 and determining which key contains the (x, z) coordinates of the touch point. When the touched (virtual) key is identified, the translation module forms 270 press the key at a predetermined KEYBOARD CODE value. The KEY DATA value or KEYCODE VALUE is output or passed to an application included in the accompanying device or system 80 (which is executed on a companion device) waiting to receive a message of a keyboard press event. The application being executed interprets the keypress event and assigns a meaning to it. For example, a text input application uses the value to determine which symbol or character was typed. An electronic piano application determines which musical note was pressed and plays that note, etc.

Alternatively, as in 4 As shown, the XZ touch coordinates can be applied directly to the application 280 be supplied. The application 280 could use the coordinate data to control the position of a cursor on a display in a virtual mouse or virtual trackball embodiment, or to quickly control a digital ink, its location on a display for a drawing or handwriting application with a virtual fountain pen or a virtual pen embodiment is shown.

5A is a simplified view of the system 10 in which a virtual device 50 now a controller with five areas, and in the accompanying device 80 . 90 includes a monitor. In this embodiment, the accompanying device 80 or 90 with a display or a display 150 shown the icons 140 may include one of which is a cursor 310 is surrounded, and a user can move this, in which he is a virtual device 50 ' used here, for example, a virtual trackball or a mouse. Inside a virtual device 50 ' becomes, for example, if an area of the user's hand 120R (or pen) a virtual area 300-1 presses, the cursor shown 310 on the accompanying device 80 . 90 instructed to move to the left. If the virtual area 300-2 is pressed, the cursor should move up. If the virtual area 300-3 is pressed, the cursor should move to the right, for example, to "select" the icon of a loaf, and if the virtual area 300-4 is pressed, the cursor should move towards the bottom of the display on the device 80 . 90 move. If the user is the fifth area 300-5 presses, a "thumbs-up" area knows the accompanying device 80 . 90 that the user selection is completed. In 5A if the user now the area 300-5 press, the "Hotdog" icon will be selected 80 . 90 would be a kiosk in a supermarket, selecting the "hotdog" icon would produce an indicator that shows where in the market hot dogs are found or the price of various brands of hot dogs being sold or the device 80 . 90 could even spend hot dogs. If the device 80 . 90 would be used in a transport setting, the icons (or words) could indicate different destinations, and the device 80 or 90 could indicate the routes, timetables and prices to the destinations, and could even spend tickets for use in a bus, subway, airline, boat, etc. For example, a user could have two areas of the input device 50 ' press, which represent the travel starting point and the travel destination, using the system 10 provide an illustration of suitable transport vehicles, schedules, fees, etc. to be displayed and, if desired, print out. It is understood that the information provided by the system 10 may simply be the raw (x, z) coordinates that a software application executed by a companion device may be used to reposition a cursor or other information on a monitor.

It goes without saying 5A that the virtual device 50 ' is passive; their outline can be printed out or painted on an underlying work surface, or perhaps the outline of the system 10 be projected. The different areas of interest in the virtual device 50 can be identified in terms of coordinates, relative to the xz plane. Consider the information in Table 1 below, that of the information in the keyboard layout 250 in 4 corresponds to:

TABLE 1

If the user's finger (or a stylus) is an area of the virtual input device 50 touches, determines the touch position module 240 (please refer 4 ) the (x, z) coordinates of the touch point 110 , In 5 lies the touch point 110 within the "B" area 300-4 , The key identification module 260 uses the information of the keyboard layout 250 in this example, as shown in Table 1, to determine where in the relevant (x, z) plane the touch point coordinates have occurred. For example, assume that the touch coordinates (x, z) are (1.5, 0.5). A search routine, preferably with the unit 70 (please refer 1A ) associated memory are stored and that of the unit 70 is executed, it is determined that 1 <x <2 and -1 <z <1. By searching for information in Table 1, the key identification module will determine that the touch point 110 within entry B falls. In this example, the companion device receives 80 and 90 Data from the system 10 that instruct area B to be pressed. The processor unit 70 in the system 10 may cause the companion device to receive such information as is required to perform the associated call with the event, for example, to move the cursor down the display.

5B shows an embodiment of the system 10 similar to the one in 1A shown. In 5B is the virtual input device 50 a computer keyboard and the accompanying device 80 . 90 is a mobile receiver / transmitter, for example a mobile phone. It is understood that the system 10 actually within the device 80 . 90 could be implemented. As such, OS1 could be a fan beam 30 from a lower portion of the device 80 . 90 and OS2 could be provided in an upper area of the same device. The virtual input device 50 could - if desired - optically from a device 80 . 90 be projected. Alternatively, the virtual input device could 50 be printed on a foldable substrate, for example. Plastic, paper, etc., within the device 80 . 90 can be taken, then removed and unfolded or unrolled and on a flat work surface in front of the device 80 . 90 could be placed. The location of the virtual input device 50 in front of a device 80 . 90 would be like that OS1 a fan beam 30 which surrounds the virtual input device, and OS2 could be cutting 110 an object, such as a user's finger or pen, etc., where a location of the fan beam is any area of interest in the virtual input device 50 covered or overlying.

In 5B OS2 will capture unreflected optical energy until the object 120R the fan beam 130 whereby some optical energy emitted by OS1 will be reflected ( 130 ) and will be detected by OS2. Relative to the (x, z) coordinate system used in 1A is shown is the intersection 110 about the place (13.5). Referring to 4 It is understood that the keyboard layout panel 250 has at least one entry for each virtual key, for example "1", "2", ... "Q", "W", ... "SHIFT" stored on the virtual input device 50 are defined. An entry search process similar to that described with reference to 5A is described is carried out, preferably by the unit 70 , and the relevant virtual button, which under the touch point 110 can be identified. In 5B is the relevant key "I", which letter "I" on the display 150 as part of an e-mail message text 140 shown in the mobile phone 80 . 90 from an area of the user's hand 120R (or by a pen) is entered. The ability to quickly send messages by touching the mobile device 80 . 90 to enter by the virtual keyboard 50 is used in contrast to the time-consuming input of messages by the mobile keyboard is used.

In 5C is an embodiment of a system 10 shown in which the work space 40 a vertical wall is, perhaps in a shop or business center, and the virtual input device 50 is also arranged vertically. In this embodiment, the virtual input device is 50 with several icons and / or words 320 shown when viewed from a user's hand 120 touched, for example, at a touch point 110 , a suitable text and / or a graphic illustration 140 on the display 150 in the accompanying device 80 . 90 let appear. In this example shown, the icons 320 Places or areas in a store, and the display 150 becomes more interactive Provide information in response to the user touch of an Icon area. In a business center, the various icons may represent entire businesses, or departments or areas within a business, etc. The detection and location of touch points, such as the 110 is preferably carried out as described with respect to the embodiments of 3A and 3B has been described. Preferably, a processor unit leads 70 within the system 10 Software that is also within or loadable from a processor unit 70 is stored to determine which icon or text portion of a virtual input device 50 was detected, and what commands and / or data to a host system 80 . 90 to be transferred.

Referring to the embodiment of 5C if the virtual input device 50 is suitable to be changed frequently, for example if there is a menu in a restaurant where the ad is 150 Detailed information, such as calories, content of sauces, etc. can show the device 50 inside a wall 40 be backprojected. It is understandable that, if the layout and location of the various icons 320 themselves, the picture information that changes within the unit 70 in the system 10 is stored, will also change. The ability to quickly change the nature and content of the virtual input device without necessarily being limited to fixed size icons in a fixed location can be very useful. If desired, some icons may actually be sized and located on the device 50 be fixed, and their touch by a user may be used to select a re-map of what is on the input device 50 shown and what is done by software within the unit 70 is shown. It should be understood that in addition to simplifying the display information that may include advertising, the accompanying devices 80 . 90 Can be used to buy advertising coupons 330 for users.

It is now up 6 Referenced. The nature of the registration of a touch event and the location of its location is determined by the system 10 determined in a way that depends on the system 10 is a structured light system or a passive light system. As noted earlier, in a structured lighting system OS1 a line generation laser system, and in a passive light system, OS1 may be a digital camera. Each system defines a level 30 that when viewed from an object such as. 120R , is cut, triggers a touch event whose (x, z) coordinates are then to be determined. Once the (x, z) coordinates of the virtual touch are determined, the present invention may decide which input or command is meant by the person using the system. Such input or command may be transmitted to an accompanying device, which of course may also incorporate the present invention.

If the system 10 is a passive light system, a touch event is registered when the outline of a fingertip appears in a selected image line of OS1, a digital camera. The touch location on the (x, z) plane 30 is determined by the pixel position of the corresponding object tip (eg. 120R ) in OS2 when a touch is detected in OS1. As in 6 As shown, the distance or distance from the camera OS1 to the touch point is an affine function of the number of pixels from the "near" end of the pixel image.

As noted, in a structured light embodiment, OS1 will typically be a laser line generator and OS2 will be a camera that is primarily sensitive to the wavelength of light energy emitted by OS1. As noted, this can be accomplished by installing a narrow band light filter on OS2 such that only wavelength corresponding to that emitted by OS1 can pass. Alternatively, OS2 may include a shutter that opens and closes in synchronism with a pulse output signal from OS1, for example, OS2 may see optical power only at times when OS1 emits optical energy. In both embodiments of a structured lighting system, OS2 will preferably only detect objects that are the plane 30 cut and reflect energy emitted by OS1.

In the aforementioned case, the touch detection detection and the distance calculation are performed by the system 10 executed. Thus, a touch event is registered when the outline of an object, such as the fingertip 120R , appears within the field of view of OS2. As in the aforementioned example, the range distance may be calculated as an affine function of the number of pixels from the "near" end of the pixel image.

Another example of analytic steps that is in 4 are carried out by the present invention are now given. Assume that the virtual input device is a keyboard 50 is how she is in 1A shown and the system 10 Output information expects that at least the scan code that corresponds to the virtual key that the user has on the virtual keyboard 50 "Touched." In 1A and in 2A Let it be assumed that the upper area (ie, the row with the virtual keys "ESC", "F1", "F2", etc.) is at a distance of about 20 cm from the optical system OS1 20 lies. It is further assumed that the camera is OS2 60 on a PDA or other device 80 is attached, about 10 cm high and at a known angle α ₁ = 120 ° relative to the plane 30 is placed. It is also assumed that the camera is OS2 60 a lens having a focal length of about 4 mm, and a camera sensor having a group of 480 rows and 640 columns.

The Z coordinate of the upper left corner of the virtual keyboard 50 should be set by convention at x = 0 and z = 0, for example (0,0). The homography H, which maps the dots in the image to dots on the virtual device, depends on the tilt of the camera OS2 60 from. An exemplary homography matrix for the aforementioned configuration is as follows:

The previously mentioned Matrix preferably only needs one time during a calibration phase be determined, as will be described elsewhere here.

It is now up 1A and 7 Referenced. It is assumed that the user 120L the area of the virtual keyboard 50 which corresponds to the letter "T", which letter "T" may be printed on a substrate to give guidance to the user's fingers, or may be part of an image of the virtual input device, for example through the system 10 can be projected. If the coordinate system previously defined is used, the "T" key may be defined as between the horizontal coordinates x _min = 10.5 and x _max = 12.4 cm and between vertical coordinates z _min = 1.9 and z _max = 3.8 cm lying as seen in 7 shown.

It will be up now 6 Referenced. Before the user finger 120L (or pin) the plane of the sensor OS1 20 does not detect light and sees an image consisting of black pixels as in the vignette 340 is shown at the bottom of the screen. However, once the user object becomes the optical level 30 cuts, becomes the cut event or the interface for OS1 20 visible, noticeable. OS1 20 now creates an image similar to that in the vignette 350 at the bottom of 6 is shown. When the tip is moving down 110 of the user object (eg the finger 120L ) the level 40 reached, more of the finger become visible. The finger contour can now be determined, for example by the unit 70 in which edge detection is performed. Such a provision is at the bottom of 6 as a "TOUCH" event vignette 360 shown. A touch detection module 220 in 4 then determines that the user object is the surface 40 touched and informed the peak detection module 230 from this event.

As in 1A to see is the virtual "T" key in the second row or row of the virtual keyboard 50 , and is therefore relatively close to the sensor OS1 20 , In 6 This situation corresponds to the fingertip in position 110 ' , As in further 6 shown is the projection of the bottom of the fingertip position 110 ' on the sensor of optical system OS2 60 relatively close to the top of the picture. The edge of the fingertip figure thus created is similar to that in the vignette 370 from above 6 is shown. In the vignette 370 Two gray squares represent the bottom edge pixels of the fingertip.

Instead, the user has the space key or another key closer to the bottom of the virtual keyboard 50 pressed, ie farther away from the sensor OS1 20 , that would be through the fingertip position 110 in 6 situation does not appear. Such a relatively distant location on the virtual keyboard is mapped to a pixel that is closer to the bottom of the image, and a border image similar to that in the vignette 380 at the top of 6 sketched would have surfaced instead. Intermediate situations of virtual key touches would produce edge mappings more similar to those used as vignettes 390 at the top of 6 are shown.

In the example explained above, in which the virtual key "T" is pressed, the tip detection module leaves 230 in

4 an edge detection algorithm, and thereby finds the bottom center of the "blob", which represents the general area of the touch, in the image line 65 and column 492 lies. The ho mogeneous mapping coordinate vector p, given below, is therefore formed as:

This homogeneous map coordinate vector p is then multiplied with the homography matrix H, around the coordinates P of the user's fingertip to get in the reference image of the virtual keyboard:

The user object or the finger 120L is thus called the virtual keyboard 50 touching at a location point, which has the coordinates x = 11.53 and z = 2.49 cm. The key identification module 260 in 4 looks for the keyboard layout 250 for a key such that x _min ≤ 11.53 <x _max and y _min ≤ 2.49 <y _max These conditions are met for the virtual key "T" since 10.5 <11.53 <12.4 and 1.9 <2.49 <3.8. Reference will now be made to 4 , The key identification module 260 therefore, determines that a user object is the virtual key "T" on the virtual keyboard 50 touches and informs the translation module 270 from this event.

This event does not necessarily have to be a keystroke. For example, the user object or finger may have previously touched the "T" key and may thereafter remain in that touch contact with the key, in which case there should be no keystroke event to the application 280 be transmitted on the accompanying device 80 or 90 expires.

The key translation module 270 Preferably stores internally the top status or the bottom status of each key internally. This module determines for each image whether a key has changed state. In the aforementioned example, if the key "T" was found to be in the down status in the current picture but was in the up status in the previous frame, the translation module would 270 a KEYCODE message to the application 280 send. The KEYCODE code will contain a "BUTTON DOWN" event identifier along with a "KEY ID" tag that identifies the "T" key and thereby the application 280 informs that the "T" key has just been "pressed" by the user object. If the "T" key had been found to be in the down state of the previous frames, the KEYCODE would contain a "KEY HERO" event identifier along with the "KEY ID" associated with the "T" key is. Sending the "KEY HELD" event on every frame (except the first frame) with the key in a down state frees the application 280 of maintaining each state via the keys. Once the "T" key is found to be in the up state in the current image, and in the previous images was in the down state, the translation module sends 270 a KEYCODE with a "KEY UP" event identifier again with a "KEY ID" flag identifying the "T" key and informing the application 280 in that the "T" key has just been "released" by the user object.

Out The above results that it is sufficient that the Image pictures only the tips of the user object, for example the fingertips contain. The various embodiments of the present Invention use less than full three-dimensional map information, which is captured within a relatively shallow space only just above a virtual input or a virtual transfer device is defined. A system implementing these embodiments, can be relatively inexpensive be manufactured and works with its own battery source. Indeed could the system be constructed to be within general devices, such as PDAs, mobile devices, etc. is included to the input or transmission of information accelerate by a user. As described, can cause unwanted effects be reduced by ambient light by the wavelengths in the embodiments selected with active light be synchronized by camera (s) and light sources, by using signal processing techniques, the pictures capture and subtract pictures, the background noise represent.

modifications and variations are possible in the disclosed embodiments without the scope of the present invention as shown in the following claims is defined, leave.

Claims (14)

System that does it with a virtual transfer device used user-manipulated user object allows Transfer information to an accompanying device, with: one central processing unit comprising a memory, at least saves a software routine; a first optical system, defining a plane that is substantially parallel to and spaced above a assumed area of the virtual transmission device; one second optical system having a relevant field of view, which encloses at least parts of the plane and to an intrusion of the User object in the plane responds to the virtual transmission device to interact; a means for determining the relative position an area of the user object at the level; the System information to the accompanying device transmits, so that the user object with the virtual transfer device the Operation of the accompanying device can cause. The system of claim 1, wherein the means for determining is designed such that it is the determination of the relative position performs by triangulation analysis. The system of claim 1, wherein the means for determining a processor unit that executes the routine to to determine the relative position. The system of claim 1, wherein: the first optical System means for generating a plane of optical energy; and the second optical system has a camera sensor, which detects a reflected part of the optical energy when the user object enters the plane. The system of claim 1, wherein: the first optical System has at least one laser to create the plane, or an LED to generate the plane; and the second optical system has a camera sensor, which reflects a Part of the optical energy detected when the user object in the Level penetrates. The system of claim 1, further comprising means for Improving the responsiveness of the second optical system on the penetration of a user object, while the sensitivity to ambient light is reduced. The system of claim 6, wherein the means for improving at least one of the following features comprises: (a) providing a signature associated with the creation of the layer, (b) Selection of a common wavelength for the energy within the Plane defined by the first optical system and for the response of the second optical system, and (c) synchronization of the first optical system with the second optical system. The system of claim 1, wherein the first optical System has a first camera sensor, which forms the plane. The system of claim 1, wherein: the first optical System has a first camera sensor, which forms the plane; the second optical system has a second camera to the penetration capture; and further with: a source of optical energy, in the direction of the virtual transmission device is directed; and a means for synchronizing the operation of at least two of the first optical system, the second optical Systems and the source of optical energy; being the effect the ambient light on the accuracy of the information with the system is reduced. The system of claim 1, wherein: the first optical System a generator for optical energy of a desired wavelength having; and the second optical system essentially only on optical energy of the desired wavelength responds. The system of claim 1, wherein the accompanying device is a PDA, a portable communication device, an electronic device, an electronic game device, and / or a musical instrument, and the virtual transmission device may be a virtual keyboard, a virtual mouse, a virtual trackball, a virtual stylus, a virtual trackpad and / or a user interface selection means , The system of claim 1, wherein the virtual transmission device on a work surface which is a table surface, a desk surface, a Wall, a cash register application, a sales application, a kiosk, one surface in a vehicle, a projected display, a physical Display, a CRT and / or an LCD. The system of claim 1, wherein at least the first one Operating system or the second operating system is a camera sensor with a lens and an imaging plane; being the lens and / or the imaging plane is inclined to the resolution and / or the depth of field to improve. The system of claim 1, further comprising means to improve the distinctness of the user object from a Background object.