Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
  • G06K5/00—Methods or arrangements for verifying the correctness of markings on a record carrier; Column detection devices
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
  • G06F3/013—Eye tracking input arrangements
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
  • G06F1/16—Constructional details or arrangements
  • G06F1/1613—Constructional details or arrangements for portable computers
  • G06F1/1626—Constructional details or arrangements for portable computers with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
  • G06F1/16—Constructional details or arrangements
  • G06F1/1613—Constructional details or arrangements for portable computers
  • G06F1/1632—External expansion units, e.g. docking stations
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
  • G06F1/16—Constructional details or arrangements
  • G06F1/1613—Constructional details or arrangements for portable computers
  • G06F1/1633—Constructional details or arrangements of portable computers not specific to the type of enclosures covered by groups G06F1/1615 - G06F1/1626
  • G06F1/1684—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
  • G06F1/1686—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675 the I/O peripheral being an integrated camera
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
  • G06F1/16—Constructional details or arrangements
  • G06F1/1613—Constructional details or arrangements for portable computers
  • G06F1/1633—Constructional details or arrangements of portable computers not specific to the type of enclosures covered by groups G06F1/1615 - G06F1/1626
  • G06F1/1684—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
  • G06F1/1694—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
  • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
  • G09G5/39—Control of the bit-mapped memory
  • G09G5/393—Arrangements for updating the contents of the bit-mapped memory
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F2200/00—Indexing scheme relating to G06F1/04 - G06F1/32
  • G06F2200/16—Indexing scheme relating to G06F1/16 - G06F1/18
  • G06F2200/163—Indexing scheme relating to constructional details of the computer
  • G06F2200/1637—Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G2360/00—Aspects of the architecture of display systems
  • G09G2360/12—Frame memory handling
  • G09G2360/121—Frame memory handling using a cache memory

Definitions

• PDA portable PCs
• cell phones digital watches, laptops, etc.
• PDA portable PCs
• cell phones digital watches, laptops, etc.
• a PDA e.g., Palm ®
• the PDA often fits into a shirt pocket and has a screen of about 3" x 3" and 160 x 160 pixels.
• the present invention provides a method and apparatus that allows a user to view a document, image or object that appears stationary in space while the portable information appliance is moved. This allows the user to call on his or her natural abilities (eidetic memory, "Mind's Eye”%) to associate object components with locations in space, which may allow the user to remain better oriented and to navigate the object space and attributes more efficiently.
• his or her natural abilities eidetic memory, "Mind's Eye”
• a PDA, a Pocket PC or other portable information appliance includes a position sensor system.
• the device is tuned for low power and portable applications, and is also tuned to the relatively narrow range of human (body-arm-hand-finger) movement (position, velocity, acceleration of the body-arm-hand-finger-portable device mass).
• Sliding window software is provided, which imparts motion to the imagery shown on the display. The sliding window software is compensated with respect to the user's body-arm-hand-finger system.
• the illustrative embodiment also stores the image content, often in a vector or bitmap or similar rendering base format.
• the user is allowed to view an object through the portable display window, where the object appears to be affixed to a reference platform.
• the reference platform may be, for example, a room, a building, a wall, the earth, or any other reference coordinate or location, as desired.
• the illustrative embodiment may change the display window imagery based on the relative position of the portable information appliance to the selected platform reference point(s). This allows the user to envision the document as being stabilized or fixed relative to a room, wall, earth, etc., which may make it easier to navigate large documents more effectively.
• Figures 2a and 2b show arm-hand-finger acceleration data in vertical and horizontal directions, respectively;
• Figure 3 reveals a display of a PDA showing both global and local views simultaneously
• Figure 6 shows the position of an accelerometer relative to the X direction
• Figure 7b lays out a MEM's delta arrangement of accelerometers for sensing in the X, Y and Z directions;
• Figure 8 shows an application of the invention for tracking facial features.
• one illustrative embodiment 100 of the present invention includes a position sensing and processing block 110, an image management and rendering block 120, a user interface block 130 and a portability functions block 140.
• the position sensing and processing block 110 provides the position sensing and/or environmental compensation functions used by the system to acquire the position information for the device.
• the position sensing and processing block 110 includes a spatial-temporal (and/or other) sensors and A/D converter block 111 , a data filter block 112, a feature extractor block 113, a state variables x, y, z, pitch (P), roll (R) and yaw (Y) (and others including time and temperature as required) block 114, and a motion compensation and gain and stabilization function block 115.
• the image management and rendering block 120 may include a database or storage or communication facility to acquire or store a picture or representation of a full object. Included may be bit maps, web pages, texture maps, multi-resolution images, attributes, elevation data, 3D model attributes and vertex lists, etc., as needed to render the document(s) or object(s) of interest on the display of the portable information appliance. Collectively, this functionality may be called the image source module 121. Typically, PDA's today have about 8 or more Mbytes of memory available. In addition to accommodating standard PDA storage functions, this memory must be re-allocated over the image source, roaming memory, rendering buffers, program and data store (not shown) and image memory, if desired.
• PDA's and other portable information appliances often have one or more communication ports, such as RS-232, Infra Red, USB, or other communication ports, as well as in some cases I/O ports connecting to the PDA system bus.
• communication ports may be used to transmit data to the portable information appliance periodically or in real-time in a delta update fashion (incrementally supplying the information as needed or in background mode).
• an 8 bit per pixel black and white 1280x1024 image to be shown consumes in uncompressed form about 1.3 Mbytes.
• a 24 bit per pixel color 1 80x1024 image consumes about 3.9 Mbytes in raw form.
• the image may be any type of image, such as a web page with graphics embedded, a Word ® or other processor document which has been rendered in a bitmap or other format, a map, a 3D rendered object, or any other image as desired.
• a compression algorithm may be used to pre-process the imagery to be stored in the portable information appliance.
• a compatible decompression algorithm implemented in the portable information appliance can then be used to reconstruct the image periodically or in real-time.
• the reconstruction can be applied to the whole image, or to segments of the image, to supply imagery or image attributes to the roaming memory function 122 downstream.
• JPEG is one such compression algorithm, which the portable information appliance can use to compress the images by a factor of 10, 20 or more for example, thereby enabling the device to store more content.
• roaming memory 122 in conjunction with the stabilization aspect of component 115, one can have roaming and stabilization for a sliding window effectuation.
• images can be converted to single-bit, diffusion halftone images to match the black and white display capability of a PDA and to minimize the files size and memory needed.
• a 1280x1024 image stored in this form consumes about 163 Kbytes.
• the images can be stored in multi-resolution forms to support different levels of detail, scaling, and magnification functions on the 160x160 pixel display of a PDA.
• the total image source memory consumed for each page is therefore as follows: 1280x1024, 640x512, 320x256, 160x128 or 160x218 Kbytes.
• the roaming memory component 122 may include RAM, addressing and address computation, and caching functions.
• One function that may be included in the roaming memory component 122 is a look ahead-function, which looks ahead and stores image content as needed and as anticipated by filtered velocity and acceleration data.
• the roaming memory component 122 may also be used to store the entire image or selected segments of the image that are visible or about to be visible to the user.
• the roaming memory component 122 may store a region of interest around the point of observation whose extent may be constrained by velocity and acceleration components along any of, for example, six degrees of freedom.
• the roaming memory component 122 may also be used to contain the image data itself and/or information and instructions needed to affect the rendering of the document or object space. These may include, for example, elevation points, model vertices, color attributes, vector instructions, OpenGL instructions, Postscript instructions, etc. Because the memory of many portable information appliances is often segmented, the image itself, while it may be available in whole form, may also be segmented and reassembled by the roaming memory component 122 and rendering engine functions block 123, working in concert.
• the roaming memory component 122 can be used to affect a coarse grained motion rendering function, moving patch by patch or block by block through the object space.
• Roaming memory 122 can be considered as a buffer to allow real-time information, thereby permitting the user to pivot and translate x, y and z information.
• the rendering engine 123 may include a mix of hardware, firmware and/or software, as desired, to render or resize or re-address in more fine grained detail the position or attributes of documents or objects to be placed into the image memory 124 for the image to be displayed.
• a typical hardware renderer may include a graphics chip having vector generator functions, geometry engine functions, tiling and stitching functions, occlusion functions, prioritization functions, and/or hidden surface removal and shading and highlighting and lighting functions. Any mix of these may be used or present, as desired. More often, the functions will include vector generator, stitching and scaling and image motion functions.
• the sensor update is synchronized with the image update to reduce the perception of transport delay and positional variance.
• the image shown on the screen of a PDA may match the actual location sensed as closely as possible.
• the video update (screen update of the document or object image) signal may, for example, trigger the reading of the sensors in a synchronous manner.
• Sensor positional update may be driven by the vertical refresh signal or a signal, which may be an integer harmonic of the vertical refresh signal. For example, reading the sensors at a 15 Hz rate as derived from the 30 Hz PDA screen update or refresh signals may improve perceived positional accuracy.
• the screen may be refreshed using the image memory (or memories) at vertical sync to preclude mid-screen shears or tears as the image is updated.
• the user interface module 130 may allow the user to select modes, scales, gain and effect compensation.
• Modes may include, for example, discrete or continuous compensation.
• Scales may include, for example, affect fine or coarse grained imagery.
• Gain may affect conformal mapping and/or scaled motion and presentation. Compensation may allow the user to cause the system to null out errors.
• Other modes may allow navigation of objects in the x and y directions alone or in combination with other degrees of freedom. In some embodiments, object opacity or transparency may also be selected.
• the user may select hybrids or one or the other of the objects as stationary or mobile modes.
• hysteresis of motion (lock and jump) may be selected.
• the user asserts his or her choices via physical buttons, knobs, touch pad, etc. or icons on the display via user selection component 131.
• the state of these selections may be made available to the user via an appropriate annunciation means. (See below description on the user interface involving display formats and modes).
• Portability functions block 140 may include time, temperature and compensation functions enabling the portable information appliance to use lower power and smaller size components than otherwise could be used. These functions may work in concert with functions shown in position sensing and processing block 110, and may include a sensor rotation platform, a calibration — li ⁇
• test or similar device leading to error, distortion, noise and drift detection and recovery information.
• an electro-mechanical connection may be used to effect some embodiments of the invention.
• the present invention may be provided in a non-intrusive, compact, add-on device that is adapted to be mounted to a portable information appliance such as the Palm ® .
• a portable information appliance such as the Palm ® .
• Many portable information appliances have an input port such as, for example, an RS 232, USB, IR, parallel or any other type of input port.
• the present invention may be implemented as an add-on device that is connected to the back of the Handspring ® PDA in the accessory slot.
• the present invention may be connected to the PDA via a port that is adapted to accept the PDA cradle. More specifically, a thin (0.2 inch thick) add-on device
• the wraparound connector 502 may further include a pass-through connector function so that the user need not remove the add-on device 500 when plugging the PDA
• the add-on module 500 may contain its own power
• processing resources including a microprocessor, image memory and sensor (camera, MEM's, ...), IO resources firmware and software and user interface controls and processing instructions, as desired.
• processing resources including a microprocessor, image memory and sensor (camera, MEM's, ...), IO resources firmware and software and user interface controls and processing instructions, as desired.
• the present invention may be incorporated into the portable information appliance itself.
• the sensors 111 may be selected to sense one or more degrees of freedom in any combination including temperature, time, pitch, roll, yaw, translational degrees of freedom including X, Y, and/or Z axis, etc. Any one or a mix of sensors may be used.
• the sensors may include surface- bound sensors such as a ball-mouse, an optical-mouse or any other type of surface-bound sensor.
• the sensors may also include locale-bound sensors, such as video-metric, pseudolite (micro-GPS), acoustic, magnetic, RF, etc.
• the sensors may also include Free-Travel sensors, such as inertial (pitch, roll and yaw, translation, etc.) sensors, piezo-ceramic film or piezoelectric material (no DC component which may be ideal for some applications), micro-electro mechanical system (MEM's) sensors, global positioning systems (GPS) and differential GPS augmented for real-time operation using, for example, an inertial sensor suite for intra-update readings, RF, etc.
• Free-Travel sensors such as inertial (pitch, roll and yaw, translation, etc.) sensors, piezo-ceramic film or piezoelectric material (no DC component which may be ideal for some applications), micro-electro mechanical system (MEM's) sensors, global positioning systems (GPS) and differential GPS augmented for real-time operation using, for example, an inertial sensor suite for intra-update readings, RF, etc.
• the device may be tuned to the relatively narrow range of human (body-arm-hand-finger) movement.
• Figures 2a and 2b below show arm-hand-finger acceleration data in the vertical and horizontal directions, respectively.
• the acceleration data were recorded during a test run of an Arm- Hand-Finger-PDA system, and represents moving a PDA up and down mostly and then left and right.
• the maximum acceleration shown is about 17 milliGs.
• one range for the acceleration sensor e.g., MEM's sensors
• the worst case acceleration range may be, for example, about +-70 milliGs. This is a relatively low value.
• MEM's ICs go down to as little as 1000 milliGs or 1 G of full span. Accordingly, commercially available one G sensors may be used. However, it is contemplated that a MEM's sensor may be manufactured that is optimized to more precisely match the acceleration range of the arm-hand-finger motion, if desired.
• the data were recorded using a video camera recording at 60 Hz but read at 100 millisecond increments (corresponding to a 10 Hz positional update rate to the PDA imagery, which is reasonable).
• a 5 to10 Hz update rate is toward the lower end of what would be desirable.
• An update rate of > 15 Hz and about 20 to 30 Hz might be favorable, depending on the update rate of the display and the application.
• imaging sensors may be used in combination with frame grabbers and image processing software to acquire anchor or reference or fiduciary points including those in the environment and/or in the portable information appliance. This kind of approach may be called "video-metric position sensing".
• a communication function may be provided to transmit the positional data of the portable information appliance relative to the selected platform.
• Imaging devices may include, for example, visual, IR, UV, multi-spectral, among others.
• Cameras like stationary camera(s) ⁇ anchored to the user's environment- mounted in the environment, such as a room, can monitor the position of the portable information appliance relative to anchor points, fiduciary points, or other points of reference in the environment.
• the anchor points can be special purpose or native to the environment. In the case of special purpose reference points, markers can be used. White spheres mounted to actors have been used in the field of digital animation.
• anchor points which are native to the environment, the system may identify statistically solid points in the room. These need not be stationary. However, in some embodiments, the system identifies consistent, stable and stationary anchor points, such as a white chair or a lamp.
• the visual sensing system ideally may have cameras, which preclude loss of line of sight of the portable information appliance.
• the camera(s) may be mounted such that the x, y, or the x, y and z components and in some cases the rotational degrees of freedom can be obtained using any of the available techniques, but this is not required.
• the cameras on acquiring the location of the portable information appliance, via template matching and center of gravity position calculation, may transmit the portable information appliance location to the portable information appliance, which then adjusts its displayed output as a function of selected functionality and self-position.
• visual spectrum camera(s) mounted in the portable information appliance may be used.
• the system may then be trained (via, for example, one time training and calibration or via subsequent periodic or continual training, calibration and refinements) to acquire anchor points within the environment.
• a viewing system 901 having a position sensor 902, a display 903 coupled and attached to position sensor 902 via structure 907, and a visual sensor 904 connected to the position sensor.
• the visual sensor could sense features 906 on — lo an operator 905, and the features would be anchors which the visual sensor identifies.
• the position sensor would determine a position of the operator according to a position of the anchors as identified by the visual sensor.
• the position of the anchors would indicate what portion of the subject matter, such as an object, document, map, etc., is to be on the display.
• the operator would move as needed to see the desired contents on the display.
• the visual sensor, position sensor and display would incorporate the appropriate processing of algorithms, software and so forth, to effect this viewing system.
• Commonly used mouse position sensors may be used in some embodiments.
• Some common mouse position sensors include, for example a rolling ball, which drives a slotted wheel that permits light to pass from an LED to a sensor. The sensor typically counts the number of light pulses to identify the motion of the rolling ball.
• a mouse of this sort may be used to transmit positional data to a portable information appliance via, for example, the standard RS-232 input port at the base of a PDA.
• X and Y position data may be transmitted and received in this manner.
• Acquiring software may periodically access a buffer containing the mouse pulse count. This count may be used to learn how far the mouse has traveled in x and in y. The count may be calibrated with respect to distances moved.
• a gain factor can be assigned so that for one inch actually moved, two inches of motion are shown, for example.
• a gain of one provides a one to one conformal mapping to the world, which may be used as a documents and users reference platform(s). Gains can be negative, zero or positive, depending on the application.
• mouse systems that are opto-electronic may be used.
• Opto-electronic mouse systems typically provide a pulse count in a manner similar to the electromechanical mouse systems described above. Having no mechanical parts, they often require little to no maintenance or cleaning to remain functional.
• Such opto-electronic mouse systems can be mounted into the bottom connector of a PDA or similar device, as desired. These systems may be particularly suitable for desktop applications, where the user does not need the mouse to free float in space.
• the sensor system may be a MEM's-only implementation.
• MEM's devices are typically small and low cost. In some cases, however, some low cost MEM's devices can produce relatively large acceleration biases and noise.
• Figure 6, for example, illustrates an accelerometer 600 along X axis 601 for the purpose of showing short term position error due to acceleration error.
• the acceleration bias of some low cost MEM's technology is about 5 mGs.
• Aout Atrue + AB.
• Xout Xtrue + sigmaX.
• the error induced over each second is about 1 inch.
• One inch of uncertainty, accumulating every second, can result in unacceptable error. As such, if the user wishes to dwell on a particular location in an image, he or she may see the image drifting an inch every second, in some cases.
• user participation may be used.
• the user may, for example, press a button (or leave a button unpressed) or similar annunciation and/or selection to notify that the portable information appliance is stationary and that calibration may be performed.
• an effective MEM's-only implementation may involve periodic calibration. This may be added relatively unobtrusively by having the user press (or not press) a button whenever he or she wants to activate the viewed-object-as-stationary mode of scrolling.
• the portable information appliance may perform at least some of the following functions: 1) with appropriate instructions to the user, the portable information appliance may alert the user for the need to hold the PDA or similar device stationary and at a user-favored orientation just prior to pressing the button; 2) it may enable the portable information appliance to calibrate itself with respect to whatever the user considers a stationary and proper orientation with respect to gravity, etc.; 3) it may encourage the user to limit activation for only relatively short bursts of time, which may alleviate long term double integration errors; and other functions, as desired.
• the output of a MEM's 111 device may be measured.
• the result may be filtered using filters 112 to eliminate out of band noise.
• the relevant information 113 may then be extracted, and encoded as X and/or Y and/or Z, for use by the system as 114.
• the location value may be stored in the compensation function 115 for subtracting or inverse filtering later, when the user releases the measurement button and/or annunciator.
• Another method may characterize the acceleration bias as a function of time and temperature and to compensate, sometimes in software, using look up tables or parametric equations as the system senses time and temperature. Limiting the bandwidth of the device may be desirable to limit the range of peak- to-peak noise spikes and associated vibration of the image. A position sensing bandwidth of 5 Hz is recommended during slow movements and 30 Hz for capturing slew.
• the bandwidth of the system may be mode dependent. High accelerations of sufficient duration may switch the portable information appliance to a high bandwidth slew mode.
• the portable information appliance may be implemented to include multiple bands of acceleration detail resulting in fine grained, medium grained and coarse grained acceleration and motion readings available for system selection and use. This can lead to a hierarchical arrangement and availability of motion control, transport delay, modes and effects.
• accelerometers alone are selected, and if the system must have sufficient accuracy and tolerance with respect to varying degrees of freedom such as pitch, roll and yaw, then six accelerometers may be used to determine translational motion. Three accelerometers, each 120 degrees apart in one plane and at a tilt of 30 degrees, for example, in an orthogonal plane, and three gyroscope MEM's may be used to supplement the readings of the other sensors and to help eliminate bias and noise effects.
• the vector components can be used to compensate for configuration gain differences as can one-time or periodic calibration events.
• Figure 7 shows one possible arrangement 700 of a delta arrangement of MEM's accelerometers 701 , 702 and 703. These are arranged to report translational acceleration along three degrees of freedom, x, y and z and angular orientation, pitch, roll and yaw, in this case.
• MEM's devices When longer times are desired between calibration updates, double integration errors using some MEM's devices can be overcome by, for example, combining the MEM's devices with a GPS or differential GPS or some other position sensing system (magnetic, specifically a magnetometer, for example), which has acceptable positional accuracy from time to time.
• the MEM's device can be used to provide readings in- between the slower updates of the GPS, for example. It is envisioned that some applications of GPS may be enabled to higher accuracy as may happen in military operations.
• the inertial sensor 111 , 112, and 113 provide the signal, filter for in-band content, and integration to provide velocity and then position information in real time. These may be implemented by a combination of hardware, firmware and software as desired.
• the motion compensation function 1 15 may provide inverse information to the roaming 122 and the rendering 123 functions. Gain may be applied to effect 1 :1 conformal mapping so that for each inch the portable information appliance is moved, the image is moved in reverse to effect conformal motion in object space. The object, therefore, may appear to remain stationary. Specifically, for each inch the portable information device is moved to the left, the image displayed is moved one inch to the right resulting in a net velocity of 0 inches for the displayed image. Alternatively, and for some applications, alternative gain factors may be used.
• the motion in object space may be small. Movement in object space may be minified, so that major movement of the portable information appliance can produce minor movement in the object space. For other applications, movement in object space may be magnified, so that minor movement of the portable information appliance can produce major movement in the object space.
• User interface may involve display formats and modes. Magnification may be implemented as a discrete or continuous function of (1 ) user selection; (2) distance from the eye reference point; (3) distance with respect to a user chosen or default reference platform or point, which may be a room in a building, a table or the earth's surface, etc. User scaling may be done by, for example, enabling, selecting, or pressing a switch, knob or soft-switch or soft-knob or similar control device.
• Motion and the scaling can be accomplished in any number of ways, including selecting an appropriate discrete image, using 3D graphics techniques such as trilinear interpolation and/or by arithmetically transforming (e.g., via affine transforms) vector representations of the images, etc.
• the images may be maps, alphanumeric, photographic, 3D CGI, volumetric, etc.
• An embodiment for bit mapped image content is to use selected discrete images for low cost, low performance portable information appliances, and to use trilinear interpolation for higher performance portable information appliance. The latter affects smooth transitions and utilizes discrete images of the same scene, each with differing levels of detail from high to low. The blend of any pair is shown on the screen as the depth changes from one to the other in a continuous fashion.
• the magnification can be discrete or smooth. It can be implemented using pre- stored images having power of two resolution differences, for example. Interpolation can be implemented between any pair of proximate resolution representations for continuous changes.
• Scaling and accompanying interpolation can be performed periodically on user request, or automatically or in real-time on a pixel-by-pixel basis, using a combination of hardware, firmware and/or software as required to meet performance objectives.
• Interpolation can be implemented using, for example, nearest neighbor, bi-linear, tri-linear, 2D or 3D cubic B-Spline, etc.
• the combining of global and local views can be used to provide a "bird's eye" view of the region of interest.
• the portable information appliance display format may be augmented to include simultaneously available views of the local and the global scenes.
• Figure 3 details an example showing both global and local views 200, simultaneously.
• Item 201 shows the combined global and the local views 200 presented on the portable information appliance display.
• the region of interest dominates the presentation.
• Item 202 shows the inset, which displays the complete document and a region of interest inset (local view inset 203).
• FIG. 4 shows the resulting co-resident global and local views implemented into a control device with a window such as with a Palm ® PDA 300 using the present invention.
• a hybrid of the mobile and stationary object transport When there are objects that are bigger than the range of motion available to the user, a combination of traversing methods may be used. For example, if the range of motion available to the user has been exhausted, the user may need to be able to return to a point within the range of available motion. This may have the effect of repositioning the user to a new reference point relative to the document or object.
• a hybrid of the anchored display and the mobile display functionality (or conversely, a hybrid of the mobile object and the stationary object) may be used.
• the system may add an additional sensor whose function is to indicate when the mouse (or, in some cases, the portable information appliance itself) has been lifted from a surface. This may be accomplished by, for example, a weight-activated switch. Alternatively, or in addition, optical feedback may be used, wherein the light path is a function of surface proximity.
• the user When the user lifts the mouse or other device, the user halts traversal in object space, returns to a convenient place in the user's available range of motion, rests the mouse or other device on or proximate to the surface again, and resumes traversal in object space from the lift-off point.
• the hybrid functionality may be implemented using: a button or a pressure sensor, conveniently placed for ergonomic principles, for example, in the upper left or right comer of the device.
• Voice actuation, motion or position induced actuation e.g., shaking, a marked positive and negative acceleration, user selected or trained (signature) pattern of motion, motion in the depth axis as if lifting off a virtual surface, or any other activation method may be used to activate the device.
• Motion or position induced actuation may be implemented using, for example, properly compensated inertial sensors (a sensor in the Z axis is good for document viewing), video-metric methods, pseudolite (micro-GPS) methods, acoustic position sensing methods, magnetic methods, RF methods, etc. It is envisioned that some users and applications may benefit from movement in discrete steps, fixed for example in two or three inch increments. Hysteresis may be used to enable transport in the object space to happen in bursts while the actual movement of the mouse (or in some cases, the portable information appliance itself) is smoothly done under operator control. Hysteresis may be accomplished within the sensor component, in firmware or software, as desired.
• the sensor data may be continuously gathered, accumulated and compared against an x or y window comparator for example. Once the accumulated value of distance traveled exceeds a modulus, 3 inches for example, then the x or y position of the view space is incremented to the next corresponding view address (image patch or view volume block).
• One may perform 3D and volumetric viewing with the invention. If the portable information appliance is implemented using free space position sensing, for example, the user may move the portable information appliance in ways that enable walking around the object and to see into the object. Seeing into the object or volumetric viewing may be enabled by, for example, using the free space sensors in combination with a 3D object model and the appropriate volume viewing software.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Computer Hardware Design (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• Human Computer Interaction (AREA)
• User Interface Of Digital Computer (AREA)
• Position Input By Displaying (AREA)
• Processing Or Creating Images (AREA)
• Control Of Indicators Other Than Cathode Ray Tubes (AREA)

Priority Applications (6)

Applications Claiming Priority (4)

Publications (2)

Family

ID=26823912

Family Applications (1)

Country Status (8)

Families Citing this family (84)

Family Cites Families (35)

• 2002
  • 2002-04-18 US US10/125,747 patent/US7714880B2/en not_active Expired - Fee Related
  • 2002-11-12 IL IL16201002A patent/IL162010A0/xx unknown
  • 2002-11-12 KR KR10-2004-7007540A patent/KR20040090953A/ko not_active Withdrawn
  • 2002-11-12 WO PCT/US2002/036258 patent/WO2003044724A2/en not_active Ceased
  • 2002-11-12 JP JP2003546287A patent/JP2005518005A/ja active Pending
  • 2002-11-12 EP EP02803614A patent/EP1573650A2/en not_active Withdrawn
  • 2002-11-12 CA CA002467299A patent/CA2467299A1/en not_active Abandoned
  • 2002-11-12 AU AU2002366070A patent/AU2002366070A1/en not_active Abandoned

Also Published As

Similar Documents

Legal Events