WO2009089045A1

WO2009089045A1 - Image based interaction and communication method

Info

Publication number: WO2009089045A1
Application number: PCT/US2009/000127
Authority: WO
Inventors: Jonathan A. Kessler; Boris Ralchenko
Original assignee: Kessler Jonathan A; Boris Ralchenko
Priority date: 2008-01-08
Filing date: 2009-01-08
Publication date: 2009-07-16

Abstract

Systems and methods for image based communication and interaction are described. In one aspect, a virtual white board is displayed onto a display surface. The virtual whiteboard facilitates drawing and writing using a device equipped with a video sensor. A set of image frames are received from a handheld device. The image frames represent respective snapshots of the current status of the virtual whiteboard. The image frames are evaluated to modify the position of the marker and for further analysis.

Description

TITLE

Image based interaction and communication method

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001 ]The present application is related to and claims the benefit of priority of the following provisional applications: application Ser. No. 61/010,348, filed January 8, 2008, entitled "Image Based Communication System and Method," and application Ser. No. 61/065,592, filed February 13, 2008, entitled "Image Based Communication System and Method." The disclosures of the foregoing applications are hereby incorporated by reference in their entirety, including any appendices or attachments thereof, for all purposes.

BACKGROUND OF THE INVENTION

[0002] 1. Field of the Invention

[0003 ]The present invention relates to man-machine communication and image processing systems and methods.

[0004]2. Description of the Related Art

[0005]People are increasingly relying on computers, projectors and surrounding technology to convey ideas. For example, SMART Board interactive technology by SMART Technologies records users annotations on an interactive whiteboard. Users can write on the interactive whiteboard in digital ink or use a finger to control computer applications via point, click and drag motions. Unfortunately, SMART whiteboards utilize specialized, proprietary equipment having certain disadvantages. First, it is an expensive solution. Second, the underlying technology records user action as snapshots therefore it is very difficult, if not impossible, to uniquely identify who made a particular mark or annotation on the whiteboard. [0006]The proposed solution overcomes the above challenges by utilizing standard equipment that is widely available, such as a computer, projector or television (TV) and a handheld device such as a cellular telephone or PDA. Moreover, the present solution is completely portable, therefore, it can be easily moved from one location to another. Specialized maintenance is not required. Instead, all that is needed is an unobtrusive software update. Additionally, the proposed solution can exist as a mostly software driven solution and also comprises features that can drastically increase productivity.

SUMMARY OF THE INVENTION

[0007]In accordance with one aspect of the present invention, described herein is an image based communication system and method (HIT™ technology). The system and method provide a natural way of communicating with a computer powered device using a HIT™ enabled device. The system and method further provide a user friendly way of visually communicating with humans via said computer powered device and HIT™ enabled device. The computer powered device can be a personal computer, either stand alone or connected to the Internet, a TV device controlled by a set-top box or a similar configuration. The HIT™ enabled device can be a dedicated device designed in accordance with the teachings herein or a generic off-the-shelf device such as a cellular phone adapted in accordance with the teachings herein. In the latter case, the cellular phone is configured as a HIT enabled device by loading and enabling certain functionality via a computer program on the cellular phone.

[0008]The HIT™ enabled device comprises the following components: an image sensor, a processing unit, a communication channel and at least one interface/operational control such as a button. One embodiment of the device comprises: a pointing mechanism, such as a laser pointer having a distinctive pattern, e.g., a cross pattern, an- image sensor comprising a webcam, a communication apparatus comprising a USB connection to a host computer and at least one virtual or real button control. Additional features may include a virtual/soft "menu button" area. When a user selects the button, a drop-down menu appears with a number of options. Alternatively, the device can comprise an actual/hard "menu button" or a control button. When the user clicks/activiates the button, a drop-down menu appears at a predefined location on the display screen. Preferably the HIT™ enabled device is a wireless device having processing done with a built-in embedded environment.

[0009]In one embodiment, the image displayed on the screen is captured by the HIT enabled device, then analyzed by the device and/or a host computer. The computer then redraws the image on the screen and the cycle repeats. The system may include more than one HIT enabled device working simultaneously. A user can "write" on top of the screen or can use the device as a "point and click" device. An exemplary application of the present system is as a digital "whiteboard" application. Thus, it is an object of the present invention to provide an improved image acquisition and processing system.

[001O]It is another object of the present invention wherein the system includes an operating system, such as a WINDOWS™ based operating system.

[0011]It is another object of the present invention wherein the system includes a HIT™ enabled device.

[0012]It is a further object of the present invention wherein the device functions at a distance of 10cm to 3 meters. The distance does not have to be contiguous and can be discrete, working at 10 cm, 1 m and 3 m. In another embodiment, the operational distance should be between 5 cm to 5 meters.

[0013]It is another object of the present invention wherein the device operates at a 90 degree view angle.

[0014]It is a further object of the present invention wherein the device communicates with the system wirelessly or other communication channel, for example, cable.

[0015]It is another object of the present invention wherein the device is small enough to hold in one hand and is ergonomic enough to allow a user to write with it like a pen.

[0016]It is another object of the present invention wherein the device can pinpoint a screen position.

[0017]It is a further object of the present invention wherein the device can pinpoint at a resolution of 5 X 5 pixels or 2 % of the screen area. [0018]It is another object of the present invention wherein the device can generate onscreen graphics at the pinpointed screen position.

[0019]It is a further object of the present invention wherein the device enables graphics to be persistent in a manner that allows a display of a tracing of the screen positions pinpointed by the device as it moves in a fluid motion. An example of this is drawing or writing on a display medium, such as a display screen.

[002O]It is another object of the present invention wherein an I/O button is provided on the device to start and stop drawing while said button is depressed.

[002I]It is another object of the present invention wherein the device allows a user to erase graphics from areas of the screen or to erase the entire screen. The erase mode can affect a larger area than the drawing. The area of a single erase pinpoint can be a radius of 30 pixels around the screen position pinpointed. The erase motion can be traced like the drawing feature.

[0022]It is a further object of the present invention wherein the device operates like a gun. For example, when a button on the device is pushed the device will display graphics to be defined at the targeted screen location.

[0023]It is another object of the present invention wherein the device can overlay graphics on broadcast or DVD images.

[0024]It is a further object of the present invention wherein the device is used for interactive programming. The device receives coordinate instructions from a broadcast signal. Specifically, the signal comprises instructions detailing target screen coordinates and text messages.

[0025]It is another object of the present invention wherein the device contains a toolbox of graphics, line colors and sound effects to present to the user upon device action.

[0026]It is another object of the present invention wherein the system includes a display medium.

[0027]It is a further object of the present invention wherein the display medium comprises display technology such as liquid crystal display (LCD) technology, plasma display technology, projection display technology (such as, DLP) and cathode ray technology (CRT). [0028]It is another object of the present invention wherein the LCD display device or the plasma display device measures up to 42".

BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Additional features and advantages of the present invention will become apparent from the following description, taken in combination with the appended drawings, in which:

FIG 1 depicts aspects of an exemplary embodiment of an image based communication system and method in accordance with the teachings presented herein.

FIG. 2 depicts an exemplary process flow of a HIT enabled device in accordance with the present invention.

FIG 3 depicts aspects of an exemplary process flow of a host computer in communication with the HIT™ enabled device in accordance with the teachings presented herein.

FIG. 4 depicts aspects of an exemplary embodiment of present invention in single-user mode as taught herein.

FIG 5 depicts aspects of an exemplary move/draw loop method in accordance with the present invention.

FIG 6 depicts aspects of an exemplary embodiment of present invention in multi-user mode as taught herein.

FIG 7 depicts additional aspects of the present invention in multi-user mode in accordance with the teachings presented herein.

FIG. 8 depicts aspects of an exemplary embodiment of present invention in mesh mode as taught herein.

FIG. 9 depicts additional aspects of the mesh mode in accordance with the present invention.

FIG 10 depicts aspects of an exemplary calibration method in accordance with the present invention. FIG 11 depicts additional aspects of the calibration method of FIG 10 in accordance with the teachings presented herein.

FIG. 12 depicts aspects of an exemplary marker placement method in accordance with the teachings presented herein.

FIG 13 depicts aspects of an alternative marker placement method in accordance with the teachings presented herein.

FIG. 14 depicts aspects of an additional marker placement method in accordance with the teachings presented herein.

FIG 15 depicts aspects of a second exemplary calibration method in accordance with the present invention.

FIG. 16 depicts aspects of an exemplary ghost recognition algorithm in accordance with the present invention.

FIG 17 depicts aspects of an exemplary system process flow of the image based communication system taught herein.

FIG 18 depicts aspects of an exemplary scaling method in accordance with the present invention.

FIG 19 depicts aspects of an exemplary marker size determination method in accordance with the teachings presented herein

DETAILED DESCRIPTION OF THE EMBODIMENTS OF THE INVENTION

[0030]The aspects, features and advantages of the present invention will become better understood with regard to the following description with reference to the accompanying drawing(s). What follows are preferred embodiments of the present invention. It should be apparent to those skilled in the art that these embodiments are illustrative only and not limiting, having been presented by way of example only. All the features disclosed in this description may be replaced by alternative features serving the same purpose, and equivalents or similar purpose, unless expressly stated otherwise. Therefore, numerous other embodiments of the modifications thereof are contemplated as falling within the scope of the present invention as defined herein and equivalents thereto.

IMAGE BASED COMMUNICATION SYSTEM AND METHOD

[003I]FIG. 1 depicts an exemplary embodiment of an image based communication system in accordance with the present invention. The system comprises a HIT™ enabled device 100, a processing unit 105 and a display medium 110. The processing unit can be a host computer such as a desktop or laptop computer, a SetTop box, a remote computer or any combination thereof. The display medium can be a display screen such as a computer monitor's screen, a television screen connected to the host computer or a virtual display based on an image generated by a projector connected to the host computer.

[0032]The host computer generates a "marker" on the display medium that is recognizable by the HIT™ enabled device. The marker serves as a remote point of reference for the HIT™ enabled device and also allows a user to track the movement of the device. When the user moves the device, the remote reference point (marker) moves accordingly and the user is able to draw or "write" on the display medium/screen. Additional software components recognize and smooth drawing/writing. The system can store the movements in a storage medium and can "replay" the stored movements.

[0033]When the user "presses" a button on (or provides an input) the HIT™ enabled device the host computer generates an image on the display screen. The HIT™ enabled device acquires and recognizes the image and further determines the "hit" or coordinates of the image. This scenario enables "point and click" operation which is different from existing "point and click" tools where user interface elements always appear (like buttons and similar widgets on a standard WINDOWS interface).

HIT™ ENABLED DEVICE

[0034]The HIT™ enabled device comprises an image detection module or component, an image recognition module or component, an I/O module or component, such as one or more input buttons, and a communications module or component. In another embodiment, the device further comprises an emitter for providing a visual indication of a point on the screen. The HIT™ enabled device can be a cellular phone, a web camera or any other device configured with the requisite technology.

[0035]The image detection module acquires an image being displayed on the display medium for further processing. In one embodiment, the detection module is stored locally on the HIT™ enabled device.

[0036]The image recognition module extracts recognizable elements from the acquired image. In one embodiment, the recognition module is stored locally on the HIT™ enabled device. In another embodiment, the recognition module may be part of the host computer package.

[0037]The I/O buttons are used to activate certain functionality, including switching among the various operational modes (for example, drawing mode versus moving the marker) and triggering certain operations such as calibration and viewing the display menu. Preferably, the buttons are part of the HIT™ enabled device, however a keyboard connected to the host computer can be used to facilitate the operations.

[0038]The communication component can comprise a wired communication channel, a wireless

TKA communication channel or a combination thereof. For a cellular phone adapted as the HIT enabled device, a preferred communication solution is BLUETOOTH, however other communication protocols are suitable such as IrDA, WiFi or ZigBee.

[0039]FIG. 2 depicts an exemplary process flow of the HIT™ enabled device in accordance with the present invention. During the management process of the device, the device detects local actions (e.g., button 1 has been activated) and listens to any communication from the host computer. Appropriate action is then taken, for example, start/stop, calibration and change parameters of the image acquisition.

HOST COMPUTER

[0040]Depending on the application, the host computer can be an off-the-shelf product or a custom designed machine. In either case, only the communication module (e.g., a BLUETOOTH^® dongle) may be required. In one embodiment, the host computer's functionality is implemented as a software package.

[0041]The host computer is responsible for establishing and maintaining communication with the HIT™ enabled device, exchanging pertinent parameters and displaying proper images on the screen. The host computer and the HIT™ enabled device may exchange commands and a given image or set of images will be displayed at a certain time. The host computer logs data including the actions of the user and has an ability to "replay" the actions on a per user basis.

[0042]In one embodiment, the host computer performs additional operations such as the encryption/decryption of data and establishing and maintaining communication with remote users. FIG 3 depicts an exemplary process flow of the host computer in communication with the HIT enabled device in accordance with the present invention.

DISPLAY MEDIUM

[0043 ]The display medium can comprise any display screen. The system utilizes the resolution of the screen during image processing.

COMMUNICATION COMPONENT

[0044]The system utilizes any compatible communication protocol to transport communication signals among system components. The following description describes one application of the system in accordance with the present invention.

WHITEBOARD APPLICATION

[0045] A preferred application of the image processing system is as a virtual whiteboard. The virtual whiteboard application allows one or more users to emulate actual whiteboard action using, for example, a television connected to a computer. In one embodiment, a HIT™ enabled device comprising a cellular phone with a camera and BLUETOOTH^® functionality can be used as a virtual pen for the purpose of annotating, drawing and writing. Other HIT™ enabled device configurations containing similar elements, i.e., a camera, communications path and controls, can also be used. In one embodiment, the host computer executes a software program defining a virtual whiteboard.

[0046]The host computer running the whiteboard application draws a marker at a certain point on the display screen. The HIT™ enabled cellular phone acquires the image, identifies the location of the marker and determines the difference between previous and current images. The HIT enabled cellular phone communicates this difference to the host computer. The host computer then redraws the marker, and depending on the "move/draw" mode, draws the line between two consecutive points representing the center of the marker. Further scaling, smoothing and handwritten recognition methods are used to enhance the writing on the fly. The host computer then stores the actions.

[0047]During whiteboard implementation, the system can operate in various modes including single user, multi-user or mesh modes. Single user mode occurs when only one user is using the application. Multi-user mode occurs when at least two users are registered to the system and are using the application. Mesh mode occurs when more than one whiteboard application is executing at the same time over the network. In mesh mode, users of each instance of the application can see the annotations of all other users as the annotations appear. FIG. 4 depicts aspects of the single user mode in accordance with the present invention. As shown, the aspects of the single user mode comprise initialization, calibration and a move/draw loop.

INITIALIZATION

[0048]The initialization procedure registers the HIT™ enabled device/cellular phone with the host computer's whiteboard application. The application is a server and the cellular phone is the client in terms of BLUETOOTH communication. Specifically, the application listens to the connections and the cellular phone tries to establish connection once the whiteboard software on the cellular phone is activated. [0049]After establishing communication, the cellular phone and application exchange pertinent parameters including the sensor's maximum resolution, working resolutions, update frequency, the ability to use windowing and similar. From the application's perspective, screen resolution, the marker's dimensions and the marker's initial color are conveyed to the cellular phone.

CALIBRATION

[0050]Calibration identifies the geometry (e.g., the location of the device in terms of screen pixels) and the color of the marker. In one embodiment, the application draws the marker at a predefined location on the screen. The user points the cellular phone to the screen and captures the image of the marker. The system calculates the approximate distance between the cellular phone and the screen, angle to the screen and the color of the marker as it is perceived by the cellular phone. Geometrical properties serve as input parameters for scaling and smoothing. The color of the marker is used to facilitate multi-user mode when each user uses a marker having a distinguishable color. Ambient lighting and the cellular phone's camera may change the way the colors are interpreted by the camera. Depending on the application, the system may utilize alternate approaches to calibration.

FIRST CALIBRATION APPROACH

[0051] With this first approach, the calibration process begins when an associated button is activated or pressed. FIG. 10 depicts one calibration process. First, the application paints brackets 1001, 1002, 1003 and 1004 on four corners of the screen. The placement, color and size of the brackets are predefined by the system. The brackets may or may not be visible within Field Of View (FOV) 1010 of the HIT™ enabled device. The system then identifies the location of the target on the screen surface using the brackets. FIG 11 depicts aspects of this recursive process. Recursion ends when more than one bracket is visible inside FOV 1010. The next placement of each bracket is separately determined. If one bracket is visible by the camera, i.e. located inside FOV 1010, it remains there at the next step while the placement of the remaining brackets are recalculated. Each step brackets are placed closer to each other until all criteria are fulfilled.

[0052]Once calibration is done, the system acquires the image (the system determines the image in terms of screen pixels) and draws a visual indicator such as a cross pattern at the target's location.

SECOND CALIBRATION APPROACH

[0053] Various factors influence how a marker color is perceived by the camera or sensor. These factors include the type of display technology being used in the system, ambient lightning and a cellular phone's camera functionality. In another embodiment of the present system, the calibration process identifies the exact color of the marker instead of a predetermined reference color of the marker.

[0054]Referring to FIG. 15, the calibration process begins by displaying a predetermined pattern 1501, such as a square having a certain size and color on the screen 1500. The sensor detects this pattern 1511 together with other elements to be filtered out. In one embodiment, the calibration process is further defined as follows:

acquirelmage() ; // The pixel matrix is accessible to the system buildMaskUsingReferenceColorO ; fϊndTheShape(); determineExactColorQ; buildMaskUsingExactColor(); determineSizeOfThePattern();

[0055]buildMaskUsingReferenceColorO - This module builds an image mask consisting of "0" and "1", where "1" replaces the pixels with a predetermined reference color. As the result of this step, pattern 1521 and other elements are represented as a bitmask. Additional process steps include:

[0056]int buildMaskUsingReferenceColor(Pixel p)

{ if (Red(p) > 127) return 0; // The pixel has more than half of Red component if (Green(p) > 127) return 0; // The pixel has more than half of Green component if (Blue(p) < 127) return 0; // The pixel has less than half of Blue component return 1 ; // All obvious checks were rejected

}

[0057]This module determines the shape of the pattern (e.g., a square) using the mask data as input. For example, the system determines that a given pattern is a square because the associated columns and rows have approximately the same number of Is. The module comprises a standard, albeit computationally intensive method that the system performs once. The module outputs the coordinates of the pattern in terms of pixels. The mask and the original image are the same size and there is a one to one correspondence between the pixels of the image and elements of the mask. As a result, all elements other than the pattern 1521 are filtered out. The system has identified the location of the calibration pattern.

[0058]determineExactColor() - Using the shape of the pattern as input, the system uses this module to determine the exact color of the pattern 1511 by averaging the color of a few pixels comprising the pattern.

[0059]To illustrate, the system first determines the shape of the pattern, e.g., a square. Next, the system selects a few pixels inside the square pattern and identifies the color of these pixels. The color of one selected pixel may be (red=220, green=12, blue=33) and the color of a second selected pixel may be (red=l 70, green=15, blue=28). Note the difference of 50 units (20%) in the red component. To overcome the pixel variations, the system averages the colors of the selected pixels to approximate the exact color of the pattern.

[0060]buildMaskUsingExactColor() - This module generates a second mask using the exact color of the pattern instead of the reference color. Additional process steps include:

[0061]int buildMaskUsingExactColor(Pixel p)

{ if( abs( Red(p) - Red(exactColor) > 25) return 0;

// The Red component must be within 10% band around the exact color if( abs( Green(p) - Green(exactColor) > 25) return 0; if( abs( Blue(p) - Blue(exactColor) > 25) return 0;

return 1 ;

[0062]During the above process the system generates sufficiently, distinct masks which are useful in a multi user environment.

[0063]determineSizeOfThePattern0 - This module determines the size of the pattern 1521 using the second mask as an input. This is a very important step because based on the size of the pattern the system identifies the size of the marker to be displayed and the shift between consecutive frames during operation. The method to identify the size of the pattern is essentially the same method of identifying the pattern on the screen. However, here the input data comprises the original image using exact color masking. FIG. 15 depicts additional aspects of the calibration process in accordance with the present invention MARKER IDENTIFICATION AND PLACEMENT

[0064]To identify the movement of the image sensor (webcam or cell phone), the system generates a marker and places the marker at a predefined location on the screen. The system then determines the next position of the marker based on the current position of the marker and the movement of the sensor. The system utilizes three approaches to marker placement on the screen, a "standard marker placement" method, a "permanently placed marker" method and a "jumping marker" method.

STANDARD MARKER PLACEMENT APPROACH

[0065]The standard marker placement approach is a universal method of determining marker screen placement; however, it is particularly suitable for use when the sensor is situated near to the display screen. Referring to FIG. 12, the system determines the center point of the sensor and associates a marker with the center point. First, the system generates marker 1210 and displays it on the screen. The same marker appears on the sensor as object 1200. The system identifies the marker 1200 and records its position. Next, when user moves the device, the position of the marker 1200 is shifted relative to the old recorded position 1201. The shift is calculated based on geometrical properties of the marker. On the next step, the marker 1210 on the screen is painted on the new position and the segment 1212 is painted on the screen. Object 1211 depicts an old position of the marker. During this step, the system generates an unobtrusive cursor that serves as a writing point or "nib" and associates the nib with the marker. The marker covers the nib on the screen. The system recalculates and updates the location of the marker on the screen with every frame.

PERMANENTLY PLACED MARKER APPROACH

[0066]The system utilizes a "permanently placed marker" approach if the marker is always visible by the sensor, that is, the sensor is relatively far away from the screen and the screen is always within the sensor's field of view. With this approach, the system generates and associates a nib with the center point of the sensor instead of the marker. The system then displays the nib at the marker's location on the screen. FIG. 13 depicts additional aspects of the "permanently placed marker" approach in accordance with the present invention. Permanent marker 1310 is placed in one of the corners of the screen. Object 1301 represents the screen on the sensor and object 1300 represents the marker on the sensor. The nib 1311 may or may not be visible by the sensor, generally speaking the method does not require the nib to be recognizable. When user moves the HITTM enabled device, the sensor view still contains all the screen 1301 within its field of view, the marker 1300 is the recognizable element. The then paints the nib 1311 at its new position. The segment 1312 is painted between old position 1313 of the nib and its new position 131 1.

JUMPING MARKER APPROACH

[0067]The "jumping marker" approach comprises the previous two methods. Referring to FIG. 14, the system initially places a marker 1410 at a predefined location on the screen. Next, the system associates a nib with the marker 1410. The marker 1401 is visible within screen image on the sensor. Nib is not recognizable element in this approach, even though it can be visible by the user and on the sensor. During sensor movement, the system determines the new location of the marker 1401. If there is a good probability that the marker will remain within the field of view of the sensor in the next frame, the system moves the nib 1411 but does not update the marker's 1410 location. To determine whether there is a good probability that the marker will remain within the sensor's field of view, the system defines a "sure view" area 1400 of the sensor's field of view. If the marker remains within the sure view area, there is a good probability that the marker 1401 will remain within the field of view in the next frame. The system can dynamically determine the dimensions of the sure view area 1400 or the area can be preset according to selected criteria. For example, if the system comprises a display screen having 800 x 600 pixels, the dimensions of the sure view area can be predefined to (200, 150) at the top left and (600, 450) at the bottom right. Here, the width and height of the sure view area is 50% of the width and height of the screen.

[0068]If the system determines that in the next frame marker 1401 may appear outside of sure view area 1400, then the new location of the marker 1410 on the screen is determined such that marker 1401 will appear at the centre of sure view area 1400 - the marker has jumped to a new position. The nib 1411 however is repainted independently of the marker. Object 1412 represents the old position of the nib. The nib is painted at the new position and the marker 1410 remains at the old location.

MARKER SIZE

[0069]Depending on the geometrical properties of the system and the distance between the sensor and the display screen, the size of the marker displayed on the screen can create undesirable effects. If the marker is too big, the marker becomes obtrusive and user experience suffers significantly. If the marker is too small, marker recognition may not be robust enough to ensure reliable operation. To overcome these shortcomings, the system determines the size of the marker as it is perceived by the sensor and updates the marker's size accordingly. As a system rule/criterion, the marker should be large enough for the system to recognize it reliably without diminishing the user's experience by being obtrusively large. The size of the marker as perceived by a sensor depends on the distance between the sensor and the screen; the closer the sensor is to the screen, the smaller the size of the marker can.be. In one embodiment, the system determines the initial size of the marker after the calibration process terminates. Referring to FIG. 19, after the system detects the marker 1900, it calculates the size of the marker and compares the markers size with a predefined "minimal recognizable size." If the detected marker 1900 is larger than the minimal size the system adjusts the size of the marker 1910 on the screen accordingly. Specifically, the system shares information with a recognition module and a displaying module about marker size in terms of the percentage of the optimal (that is, a minimal recognizable) size. For example, if the optimal size is 100 and the detected size is 120, the recognition module (cellular phone) sends the message to the displaying module encoding value 1.2 as the size of the marker. The displaying module in turn adjusts the size of the marker on the screen. This process can be done iteratively.

[0070]Marker size processing is especially important when the user operates in the short and middle distances ranges and variation in distances is relatively high. SCALING

[0071] When the user operates the sensor at a distance that is relatively far away from the screen any small movement of the sensor may result in a big shift of the writing point on the screen. To address this, the system scales sensor movement in relation to the distance between sensor and screen.

[0072]Referring to FIG 18, if the distance between the sensor and the screen is relatively large, any small movement of the sensor can result in a large movement on the screen. This is undesirable. The system uses scaling to reduce large movements by a configurable factor. In operation, the system detects the sensor movement and determines the marker's shift. The new position of the marker is defined as follows:

[0073]The distance is determined based on the distance between old 1801 and current/new 1800 positions. In the example, the new marker's position 1812 is in a corner of the screen. Correcting the computed distance by a constant factor generates the new marker's position 1810 on the screen. The constant factor can be used to scale up or scale down the distance.

[0074]The direction of the marker's new location is the same as the calculated direction. In other words, the movement vector keeps the direction but uses only a fraction of the distance detected.

GHOST IMAGES

[0075]The present invention contemplates using various display technologies as the display medium. However, an issue of ghost images arises with certain display technology. Ghost images appear when excited pixels getting darker within considerable longer interval than time required to excite the same pixels. The human eye may not notice or ignore the ghost images, however, an electronic sensor will detect and record them. The present system uses a ghost recognition method to solve the problem of "ghost markers." FIG. 16 depicts additional aspects of the ghost recognition method in accordance with the present invention. The marker on the screen 1610 is detected by the sensor as 1600. When marker is repainted on the screen on its new position 1610, the ghost image 1611 is still detected by the sensor as 1601. On the next frame, the marker is painted on the new position 1610 while two ghost images 1611 and 1612 still remain on the screen. Those ghost images are detected by the sensor as 1601 and 1602. The "ghost elimination" method is used to distinguish between real and ghost images. The method tracks the first marker appearance and eliminates ghost images based on differences between calculated and detected places. The method tracks multiple markers using timing sequences. In addition, the method adds intervals between a user's movement and any actual writing.

[0076] Another approach is to update hardware parameters of the screen and/or the sensor so that the ghost images disappear or become non-recognizable by the sensor.

FILTERING AND SMOOTHING

[0077]The system utilizes filtering and smoothing processes to reduce and/or eliminate undesirable movements which may distort the annotations being displayed. The system filters for small and large movements that affect the displayed image.

SMALL MOVEMENT FILTERING

[0078]Small movement filtering filters out any undesirable movement caused by a user's shaking hands as the user attempts to write to the screen from a distance. The amplitude of the hand shaking is different for each user and may be calibrated. In one embodiment of the present system, small movement filtering is defined as follows:

if (distance(current_position, next_position) < small_movement_interval ) ignore movement ; else current_position := next_position.

[0079]The small movement interval of filtering depends on the distance between the sensor and the screen. The interval can also be unique to each user. To identify the interval, the system prompts a user to draw a horizontal line, vertical line, diagonal line and circle. For each line and circle, the system determines the interval as an average between an ideal line and the actual line being drawn.

LARGE MOVEMENT FILTERING

[0080]Large movement filtering filters out any irregular movement caused by external factors, for example, the sun's reflection of a car's windshield that is caught by the sensor. The system utilizes an additional filter to eliminate the undesirable effects of sudden movement of the sensor or when the system miscalculates the position of the marker.

[008I]To prevent the sudden jump of the marker, the system will ignore such movements by defining "any movement of more than n pixels (e.g. 100 pixels) is an error, please disregard." Large movement filtering is defined as follows:

if (distance(current_position, next_position) > big_movement_interval ) ignore_movement ; else current_position := next_position.

[0082]A user may set the big_movement_interval. Alternatively, the system can determine the interval setting wherein the interval can be a static default interval or be dynamically determined depending on usage pattern.

[0083]Employing filters alone can significantly improve any drawing and writing to the screen. However, to further refine annotations generated on the screen, the system can also utilize a smoothing process.

SMOOTHING

[0084]The system utilizes a shape recognition process to smooth handwritten characters being displayed on the screen. To illustrate, the English and French languages have similar alphabets. Although, the French word 'garcon' may be smoothed to the English word "garcon", it is not ideal to do so. Instead, the system recognizes the shape of the pattern "C" and smoothes the annotation accordingly. The smoothing process employs a set of rules that are expressed via a simple syntax of "is" and "is not" connectives. For example:

"C" is "C"

"C" is not "C"

"E" is "E"

"E" is not "E"

where the handwritten characters on the left side of the "is/is not" connectives are associated with predefined character/symbol patterns on the right. The system comprises a default set of rules which are updateable for system robustness. This feature becomes especially important where adding or removing an extra sign around a letter may create a different meaning among languages.

[0085]During operation, the system detects the shape of a handwritten character and utilizes the smoothing rules to correspond the shape of the handwritten character with the shape of a matching character from the active set of predefined character patterns. Hence, the system is language independent working only with the shape of the characters comprising a given alphabet.

MOVE/DRAW LOOP

[0086]FIG. 5 depicts aspects of the move/draw loop in accordance with the present invention. The drawing depicts one common scenario of the move/draw loop. In a more sophisticated environment, the application may send a request to the cellular phone to acquire and send the image. By doing so, the application could be more flexible in terms of refresh rate. Using this approach requires a higher level of integration with the particular model of the cellular phone. MANAGEMENT FUNCTIONALITY

[0087]In one embodiment the system also provides management functionality. During the management operation, movements are logged for storage. The log file is updated at a configurable time interval, e.g., once every few seconds. For the single user mode, the format for the string utilized may be as follows:

where <timestamp> is a system time at the moment of the event, <ID> is the cellular phone ID. Each cellular phone is assigned a unique ID including remote cellular phones in the mesh mode. <type> is the enumerated type of data being logged and <data> is the actual data that is interpreted by the application in "replay" mode. Data encoding is per type base, so the data format for calibration will differ from the data format for movement.

SYSTEM PROCESS FLOW

[0088]Figure 17 depicts another exemplary embodiment of the inventive system of the present invention. The following description contains the following terms which are defined as follows:

[0089]Segment: a single straight line. A segment represents a shift between two consecutive frames.

[0090]Path: a set of segments representing a single element of the writing or drawing on the screen. For example, letter "O" consists of single path, letter "T" consists of two paths - horizontal and vertical lines. A path is initially built from segments and can be further smoothed.

[0091]Symbol: a logical element that consists of one or more paths. A symbol may represent any alphanumeric or other recognizable unit.

[0092] Referring specifically to FIG. 17, the system process flow includes:

[0093]Start - the system checks screen parameters (type of screen, resolution, etc), the connection to any sensor and the sensor's parameters. [0094]Calibration - system calibration is done as provided herein.

[0095]Detect the motion - the system software detects any motion, calculates the shift between consecutive frames and redraws the marker if needed depending on a selected marker redrawing method.

[0096]Paint the segment - the system paints a segment. In doing so, the system can utilize small motion and big motion filters. The system paints the segment if the segment passes through the filters.

[0097]Path complete - a path is complete when a user releases an I/O button associated with the sensor (i.e., the user begins the path process by pressing the input button and terminates the path by releasing the input button. If a path is incomplete, the system adds segments to it.

[0098]Analyze the path - when the path is complete, the system utilizes the smoothing and detection components. During operation, the system may or may not immediately recognize the desired symbol for display. In the latter case, the system tries to recognize as a single symbol last paths combined. Once the symbol is recognized, all previous paths that are not recognized are considered symbols.

[0099]Paint the path - the system paints the smoothed path. Any previously painted path can be repainted if it becomes part of the newly recognized symbol.

[0100]Start new segment - user presses the I/O button to begin the system cycle anew.

[0101]System cannot detect motion - The system may be unable to detect motion in a variety of circumstances. Examples include: case 1- when a user lays the cellphone aside and the screen is no longer in the sensor's field of view; case 2 - someone or something may block the projector unit that is projecting the image; or case 3- the system may lose track of the marker. If the system is unable to detect motion, the system goes into standby mode. When standby ends and the process resumes, the system can re-detect the marker and continue processing (case 1). The system can also be set to recalibrate either automatically or via user input (case 2). Finally, the system can close the session after a predefined time period has elapsed (via a configurable timer). DATA TRANSFER BETWEEN WIRELESS DEVICE AND HOST COMPUTER

[0102] When working with a wireless HIT™ enabled device such as a HIT™ enabled cellular phone, it becomes apparent that transferring an image from the device to the host computer can create a significant problem due to the size of the image, the communication bandwidth and the power requirement. For example, an image of 640x480 consists of 307200 pixels. Each pixels is 24 bits (8 bits per color) totaling 7372800 bits or more than 7 MBits. The standard Bluetooth protocol (version 2.0) allows up to 2.1 Mbits/sec. To compress the image is undesirable because information is lost. The present system utilizes a wireless data transfer component to address this shortcoming.

[0103]During operation, the system sends a reference color to the cellular telephone. The cellular telephone's computational power can adequately build the mask because it is a simple operation of linear complexity. For example, the memory requirement is also very modest. The mask is image encoded without losing any information. The memory required to keep the mask is 1 bit per pixel which is 24 times less than original image. Thus, all that is need to transfer is 300KBits (307200 bits is the exact number) of information at the maximum in order to perform all required operations on the host computer.

[0104]In practice, it is unlikely that the marker will occupy the entire frame, therefore the marker can be defined to occupy a small area on the screen/frame. The minimal size of the marker and the "solidness" of the marker enables reliable recognition by the system. Further, the system stores the shape and dimensions of the marker for further processing. Because the marker has a solid shape, the system will recognize that a sequence of "0011001100" is not part of the marker's shape.

[0105]Consider the estimated dimension of the marker being 100 x 100 pixels and located between lines 220 and 350. The system sends at least 130 lines to reliably transfer the marker. The cellphone sends marker information line by line, one per frame, as follows:

Line 0: 1010 0000 - This line contains a data sequence having a management code. The system will not interpret this sequence as marker data because the data does not define a solid shape. 1010 is the management code. 0000 is an "empty line" code. Lines 1 - 219: 1010 0000 - These lines contain the same data sequence as Line 0. This process sends 8 bits per line instead of 640 bits per line.

Line 220 - Marker data begins with this line because it does not begin with a management code.

Lines 221 - 350: These lines also contain marker data sequences.

Line 351 : 1010 1111 - This line contains a data sequence having a management code by end_of_frame command code.

[0106]Based on the foregoing, the amount of information to encode and to transfer lossless data from the cellphone to the host totals 84968 bits, as follows:

130*640 bits = 83200 bits of real data

220* 8 bits = 1760 bits of management data.

constant header - the header can be as simple as an 8 bit sequence of 1010 1010.

[0107J84968 bits is less than 85 KBits, therefore the system can update images faster than 20 Hz which is more than adequate for application processing.

[0108]It is possible to decrease the amount of data to be transferred by encoding (basically removing part of the lines and encoding index where pertinent data starts) lines containing marker data approximately in half, to about 50KBits, but actual implementation must weigh in processing power of the cellphone and the frame update frequency.

MULTI USER MODE

[0109]The general sequence for an exemplary multi user mode is similar_as for single user mode. The difference is that the application keeps listening to the connections after the first cellular phone has already been connected. The application assigns each connection (e.g., cellular phone connection) with a distinct marker color. The uniqueness of the color is ensured by the application. The calibration process is performed per cellular phone as well. The application keeps track of each cellular phone separately. For example, the BLUETOOTH^® standard allows up to 7 active simultaneous connections and up to 255 total connections.

[0110]Logging is defined per cellular phone, and each entry is time stamped as it is in single user mode. Replay mode is configurable and it is possible to "replay" a single cellular phone, all phones together or just selected ones. FIG. 6 is an exemplary embodiment of the system of the present invention in multi-user mode. As shown, there are three cellular phones configured as HIT™ enabled devices that are registered to the system. Each cellular phone is assigned a distinct color marker to visually distinguish which cellular phone controls which drawing. FIG. 7 depicts aspects of the multi-user mode in accordance with the present invention.

MESH MODE

[011 l]Several applications can be connected to the mesh. The mesh can be established over an IP network, thus enabling multi-site communication. Each application in the mesh can be either in single user mode or multi user mode. Registration and calibration can be performed per single application instance only, thus having no limitation on the number of cellular phone connections per mesh. The logging is done per application as well. The replay function can consider cross- logged connection events in order to synchronize output.

[0112JFIG. 8 is an exemplary embodiment of the system of the present invention in mesh mode. As shown, there are at least two applications connected to each other in the mesh mode over an IP network, i.e., the Internet. Application 1 is in the multi user mode and Application 2 is in single user mode. In operation, a drawing made on a screen associated with a first computer is transferred via the Internet to a second computer remote from the first screen for display on the screen associated with the second computer. In this embodiment, the markers of the remote end are not displayed. Communication can be done in almost real time and delay is mostly related to the quality of the IP network. FIG. 9 depicts further aspects of the mesh mode in accordance with the present invention.

[0113]The embodiments of the invention described above are intended to be exemplary. Those skilled in the art will therefore appreciate that the above description is illustrative only, and that various alternatives and modifications can be devised without departing from the spirit of the present invention. Accordingly, the present is intended to embrace all such alternatives, modifications and variances which fall within the scope of invention.

[0114]Moreover, the techniques may be implemented in hardware or software, or a combination of the two. Preferably, the techniques are implemented in control programs executing on programmable devices that each include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device and one or more output devices. Program code is applied to data entered using the input device to perform the functions described and to generate output information. The output information is applied to one or more output devices. Each program is preferably implemented in a high level procedural or object oriented programming language to communicate with a computer system, however, the programs can be implemented in assembly or machine language, if desired. Each such computer program is preferably stored on a storage medium or device (e.g., CD-ROM, hard disk or magnetic diskette) that is readable by a general or special purpose programmable computer for configuring and operating the computer when the storage medium or device is read by the computer to perform the procedures described in this document. Features of the system may also be considered to be implemented as a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner.

Claims

CLAIMSWhat is claimed is:

1. A method comprising: displaying a virtual whiteboard onto at least one display surface; communicating by at least one remote handheld camera equipped device at least one marker position to a computing device for processing; and evaluating a set of image frames representing respective snapshots of the virtual whiteboard to modify an image over the virtual whiteboard.

2. A method as recited in claim 1, wherein the whiteboard surface is a computer screen.

3. A method as recited in claim 1, wherein the display surface is a projection surface or a television set.

4. A method as recited in claim 1, wherein the marker is movable or constant.

5. A method as recited in claim 1, wherein the at least one remote handheld device is a web camera, a custom made device, a cellular phone, a smartphone or a handheld organizer.

6. A method as recited in claim 1, wherein evaluating the image frames further comprises: identifying an initial location of the at least one marker; calculating a projective homography indicating a transformation between a sensor coupled to the at least one remote handheld device and a coordinate system associated with the virtual whiteboard; determining a new position for use by a virtual whiteboard application to move the at least one marker from the initial position to the new position.

7. A method as recited in claim 6 wherein the transformation is a function of a user panning the at least one remote handheld device with respect to a presentation on the display surface.

8. A method as recited in claim 6, further comprising: distributing computation between the at least one handheld device and a host computing device communicating with each other.

9. A method as recited in claim 8, wherein communicating between the at least one remote handheld device and the host computing device is wired.

10. A method as recited in claim 8, wherein communication method between remote handheld device and the host computing device is wireless.

11. A method as recited in claim 10, wherein communicating between the at least one remote handheld device and the host computing device is one directional.

12. A method as recited in claim 10, wherein communicating between the at least one remote handheld device and the host computing device is two directional.

13. A method as recited in claim 12, wherein communicating between the at least one remote handheld device and the host computing device is half duplex or full duplex.

14. A method as recited in claim 1, wherein the at least one remote handheld devices is registered to a system prior to operation.

15. A method as recited in claim 1, wherein the at least one remote handheld devices is automatically registered to a system.

16. A method as recited in claim 1, wherein the at least one remote handheld devices is registered to a system one time.

17. A method as recited in claim 1, wherein the at least one display surface are greater than one and are distributed and are located in a different physical location while maintaining a single instance of an application.

18. A method as recited in claim 1, wherein the at least one display surfaces are distributed over a LAN or an internet.

19. A method as recited in claim 1, wherein the at least one display surfaces are implemented as part of a web browser.

20. A method as recited in claim 1, wherein a single instance of an application is hosted on a single computer, a dedicated computer cluster or on several computers that are connected via network with ad hoc application support.