WO2001031570A9 - Suivi du deplacement d'un instrument d'ecriture - Google Patents
Suivi du deplacement d'un instrument d'ecritureInfo
- Publication number
- WO2001031570A9 WO2001031570A9 PCT/US2000/029758 US0029758W WO0131570A9 WO 2001031570 A9 WO2001031570 A9 WO 2001031570A9 US 0029758 W US0029758 W US 0029758W WO 0131570 A9 WO0131570 A9 WO 0131570A9
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- light
- writing
- writing instrument
- sensors
- instrument
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06F3/0386—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry for light pen
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
- G06F1/26—Power supply means, e.g. regulation thereof
- G06F1/32—Means for saving power
- G06F1/3203—Power management, i.e. event-based initiation of a power-saving mode
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
- G06F1/26—Power supply means, e.g. regulation thereof
- G06F1/32—Means for saving power
- G06F1/3203—Power management, i.e. event-based initiation of a power-saving mode
- G06F1/3234—Power saving characterised by the action undertaken
- G06F1/325—Power saving in peripheral device
- G06F1/3259—Power saving in cursor control device, e.g. mouse, joystick, trackball
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/0354—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
- G06F3/03542—Light pens for emitting or receiving light
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
- G06F3/0428—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04886—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Definitions
- This invention relates to tracking motion of a writing instrument.
- the surface on which the pen is 20 moving may have an array of pixels or other sensing locations each of which responds when the pen is at that location.
- the pen tracking is done entirely by electronics mounted in the pen.
- the moving pen communicates with stationary sensors that are separate from the 25 pen, and triangulation algorithms are used to track the motion.
- the invention features a method that includes conveying light from a moving writing instrument as an indication of the location and path of the writing instrument on a 5 two dimensional writing surface; sensing the light at two or more sensors and generating a sequence of signals representative of the sensed light; and applying a technique to reduce the effect of variations of the light intensity in a third dimension with respect to the generated signals.
- Implementations of the invention may include one or more of the following features.
- the technique may be based on optics that are configured to enhance the uniformity of signal response of the sensors.
- the lens may be a spherical lens or an aspheric lens.
- the sensors may be arrays of sensitive pixel elements or analog
- the technique may be based on algorithmic processing of the generated signals.
- the algorithmic processing may include linearizing the signal response of the sensors based on parameters associated with the writing instrument.
- the technique may be implemented in digital hardware or in analog circuitry.
- the 20 algorithmic technique may reduce the effect of variations of the light intensity based on other than dimensional effects.
- the signals may be grouped in frames, and the signal processing technique may include processing of multiple frames to cancel noise.
- the light conveyed from the moving writing instrument may be
- the sensor signals may be chopped at the frequency of modulation. Opposite gains may be applied to each of the chopped signals depending on the on or off state of the light conveyed from the writing instrument that corresponds to the 5 signals.
- the frame rate may be varied.
- the chopped signals may be integrated over time.
- the light conveyed from the writing instrument may include a strong short pulse imposed at on the modulation frequency, a phase lock loop may determine the modulation frequency from the sensor signals, and the sensor
- the 10 signal may be sampled at the times triggered by the phase lock loop during the duration of the strong short pulse.
- the characteristics of the conveyed light may be used for synchronization between the writing instrument and the sensors.
- the conveyed light may include periods of lower frequency
- the sensor signal associated with the higher frequency bursts may be used to lock onto a modulation clock.
- the invention features a method that includes conveying light from a moving writing instrument in a 20 time-changing pattern of directions, sensing the light at two or more sensors located at two or more different locations spaced from the writing instrument, and determining the location of the writing instrument by detecting a phase difference between signals measured at the two or more sensors.
- Implementations of the invention may include one or more of the following features.
- the time-changing pattern of directions may include a rotating pattern with respect to an X-Y plane on which the writing instrument is moving.
- the signal radiated in the 5 positive X direction may be in phase quadrature to the signal radiated in the Y direction.
- the invention features apparatus that includes sensors configured to receive light from a writing instrument moving across an X-Y writing surface, the light having 10 variations in intensity along a Z-axis normal to the writing surface, andoptics configured to enhance optical power of the light received from the writing instrument.
- the optics may be a ball lens or an aspherical
- the optics may include a single spherical lens and the lens and the corresponding sensor may be together configured to enhance the optical power of light received at large angles or longer distances or at disadvantageous positions of the writing instrument.
- the optics may include a special lens configured to
- the optics may include two cylindrical lenses, one nearer the sensor to project light horizontally onto sensor, and the other positioned to collect light in the Z-axis dimension, the other lens having a body that is
- the algorithmic processes may enhance
- the processes may determine the integral power of the overall signal distribution on the sensor and calculate a subpixel 5 position based on half of the integral power position.
- the processes may use a polynomial approximation on the signal distribution and calculate a subpixel position as a position of approximated maximum.
- the calibration procedure may produce parameters to be used in combination with data from the sensors.
- the calibration parameters may correct for manufacturing deficiencies of the optics and the sensors, and the algorithmic processes may use a straight triangulation technique to determine a position of the writing instrument.
- the calibration parameters may correct for manufacturing deficiencies of the optics and sensors
- the algorithmic processes may determine the position of the writing instrument using polynomial series, where coefficients in these polynomials are determined during the calibration procedure.
- the invention features a method that includes receiving light from a moving writing instrument at a an 20 array of sensing elements of a sensor, reading the sensing elements in sequence to generate a sequence of signals indicative of light sensed by the elements of the array, and resetting each of the elements after it is read and before at least some of the other elements in the array are read.
- Implementations of the invention may include one or more of the following features.
- the array may include a CMOS or CCD position sensor. All of the elements may be read before all of the elements are reset .
- the invention features a method that includes determining a sequence of three-dimensional positions of the moving writing instrument based on the signals.
- the invention features the combination of a writing instrument having an elongated housing 10 configured to be hand-held, a light source in the housing, and a lens in the housing configured to receive light from the light source and to convey the light through a free-air path to optical sensors spaced from the writing instrument, the lens being configured to be semi-reflective.
- the light source includes an array of light sources arranged around an axis of the writing instrument and configured to emit light in a direction normal to the axis.
- the lens may be configured to internally reflect and concentrate the light and to emit it by reflection from a reflective external surface of the lens.
- the lens may have a cylindrical body having an upper surface that receives the light and a lower annular surface that reflects the light toward the optical
- the reflective external surface may include a conical surface oriented at a 45 certain degree angle to a longitudinal axis of the writing instrument.
- the light source in the pen may include LEDs arranged in a ring.
- the invention features a device configured to turn the light source on and off in response to a user applying pressure from the writing instrument to a writing surface, the device being configured so that an amount of pressure required to trigger the device is not so large as to disrupt normal writing 10 motion of the writing instrument on the writing surface.
- the writing instrument may include a ballpoint cartridge having a writing point and the device may be positioned at the opposite end of the cartridge from the writing point.
- the 15 device may be a switch or a pressure sensor.
- the invention features a holder having a receptacle for receiving at least a portion of the writing instrument for storage of the writing instrument,
- the writing instrument and the holder containing respective 20 elements that enable wireless transmission of signals associated with motion of the writing instrument and tracking of the writing motion based on the signals.
- the holder may be a pen cap and may include a clip configured to attach the holder to a stack of pages or to a notebook.
- the holder may include at least two light sensors and a processor that processes signals from the light 5 sensors to determine a sequence of positions of the writing instrument.
- the holder may include a receptacle for holding the writing instrument and for enabling recharging of batteries in the writing instrument.
- the invention features an element that 10 enables wireless transmission of a signal associated with motion of the writing instrument and tracking of the writing motion based on the signal, the element being built into a cell phone, a PDA, a webpad, or a clipboard.
- the invention features, a holder that 15 has a mechanism for attaching the holder to a writing substrate in an orientation that enables the elements to be used in conjunction with the wireless transmission.
- the clipping mechanism may include a switch to activate functions of a processor in the holder when the clipping mechanism is manipulated.
- the invention features, a holder that includes a receptacle for the writing instrument and a recharging circuit connected to recharge the battery when the writing instrument is in the receptacle.
- the invention features a CMOS sensor adapted to receive light associated with motion of a writing instrument and to provide signals indicative of an angle of receipt of the light with respect to a known direction, and a lens aligned to 5 direct the received light to the CMOS array.
- the lens may be optimized for collection of light from an area in which the motion of the writing instrument occurs.
- the lens may be a field lens or a Fresnel lens.
- the lens system may be configured to collect light in a dimension 10 normal to a plane of motion of the writing instrument and to project the light onto the sensor in a direction parallel to the plane of motion.
- the invention features calibrating by positioning a writing instrument at a succession of positions on a 15 writing surface, generating signals at sensors from light received from the writing instruments at the succession of positions, and determining calibration parameters for the writing instrument for use in calibrating a process that determines the positions of the writing instrument as it is being moved.
- the calibration parameters may include coefficients used in polynomial series that are part of the position determining process.
- the positions do not lie on a regular rectangular grid.
- the invention features (1) identifying locations on a writing surface that correspond to input elements to be entered into an electronic device, the writing surface being nonelectronic and separate from the electromc device, (2) using a 5 writing instrument to point to selected ones of the identified locations corresponding to input elements to be entered, and (3) sensing the locations at which the writing instrument is pointing and entering the corresponding data into the electronic device.
- the writing surface includes a 10 sheet of paper
- the input elements comprise characters of language or commands that are printed on the writing surface.
- the invention features moving a writing instrument across a non-electronic writing surface to indicate a path, and remotely sensing the path and generating 15 signals for use in entering the path into an electronic device that is separate from the writing surface.
- the invention features modulating _ light that is conveyed from a moving writing instrument to light sensors spaced from the writing instrument at a predetermined 20 frequency, and using a phase locked loop associated with the sensors to lock onto the phase of the modulated light.
- the invention features, circuitry for tracking writing motion of a writing instrument using wireless transmission of signals between the writing instrument and a
- the stationary element including a main processor and a separate preprocessor, the preprocessor being connected to perform at least data capture with respect to motion of the writing instrument, the main processor being connected to 5 perform at least data communication with respect to the tracking.
- the preprocessor may also be connected to perform user interface functions and sub-pixel data storage and the main processor may also connected to perform background cancellation and sub-pixel calculation.
- the invention features a reflective element configured to reflect light received from outside of the writing instrument to the sensor for use in tracking motion of the writing instrument.
- the reflective element may 15 reflect light to the sensor when the writing instrument is being used for writing and disable the reflective element from reflecting light to the sensor when the writing instrument is not being used for writing.
- the invention features receiving light 20 from a moving writing instrument at a light sensor having an array of sensitive pixel elements, and determining the location in the array at which the maximum intensity of light has been received from the writing instrument, the location being determined with sub-pixel accuracy.
- the sub-pixel location is determined by determining the integral pixel location that is closest to the subpixel location, and finding a fractional center of gravity of a subarray that is centered on the integral pixel location.
- the invention features indicating locations on a non-electronic surface that correspond to inputs to an electronic device, and detecting the locations and inputting them into the electronic device.
- the invention features a clip for 10 clipping paper on which the writing instrument is to be moved to the sensor.
- the mechanism may be part of a clipboard or a notebook
- the clip may include a mechanism that enables a user to cause the clip to grip or to release the paper.
- the 15 clip may include an activation button and a spring and a lever operated by the button.
- the lever may be configured to rotate in response to the button.
- the button may be configured to be pushed or puUed.Other advantages and features will become apparent from the following description and from the claims.
- Figure 1 shows pen tracking
- Figure 2 shows a pen
- Figure 3 shows a lens in a pen.
- Figure 4 shows a lens in a pen.
- Figures 5 and 6 show reflection of light in a pen.
- Figure 7 shows a tracking method
- Figure 8 shows a pen
- Figure 9 shows a pen.
- Figure 10 shows a holder
- Figure 11 shows a lens in front of a sensor.
- Figure 12 shows a holder
- Figure 13 shows a holder
- Figures 14 and 15 show half of a holder.
- Figure 16 shows another half of the holder.
- Figure 17 shows a field of view.
- Figure 18 shows a block circuit diagram
- Figure 19 shows a state diagram
- Figure 20 shows a timing diagram
- Figure 21 shows a timing diagram
- Figure 22 shows geometry of tracking
- Figure 23 shows a circuit diagram
- Figure 24 shows a timing diagram
- Figure 25 and 26 shows a rotating beam technique
- Figure 27 shows a channel diagram
- Figure 28 shows a channel circuit
- Figure 29 shows a paper keyboard.
- Figure 30 shows a spherical lens
- Figure 31 shows an aspherical lens.
- Figure 32 shows a two-lens arrangement.
- Figure 33 shows a clip
- Figure 34 shows a clip
- Figure 35 shows a sliding belt clip
- Figure 36 shows two views of a clip.
- Figure 37 shows two views of a clip .
- the sensors to measure pen position with respect to the sensors.
- the sensors are CMOS or CCD linear or 2D arrays, Position Sensitive Detectors (PSD) or other light sensitive detectors.
- PSD Position Sensitive Detectors
- the sensors can be clipped to the edge of writing surface allowing reconstruction of 5 writing on that page.
- the position of the pen is determined by mapping the sensor reading to the actual XY position of the pen on paper.
- This electronic input device looks like a regular pen with a holder. The user writes with it just as with any ordinary pen on paper,
- the pen notebook or any other flat surface. It is used to capture handwriting text or drawings.
- the pen stores all its movements during its use by recording sensor measurements into its memory. The pen then downloads it to a computer, personal digital assistant, handheld computer or cellular phone.
- the handwriting as it appears on a
- a pen or other writing instrument 10 that leaves a visible trace 12 of writing or drawing in the usual way on a sheet of paper or other writing surface 14 may also have a source 16 that emits infrared (IR) light 18 for use in automatically 20 tracking the motion of the pen.
- IR infrared
- the light is detected by IR sensors 20, 22 that are held stationary relative to the pen at a nearby location, for example, near the edge 23 of the paper.
- the sensors deliver sequences of signals that represent the position of the pen on the writing surface (e.g., angle 24) at which the light 25 is received from the pen for each of a succession of measurement
- Circuitry associated with the sensors uses an algorithm to process the directional information (and the known distance 26 between the sensors) to determine a succession of positions of the pen as it is moved across the writing surface.
- the algorithm can 5 use a mathematical model that translates pixel signals of the sensors into positions on the writing surface.
- the algorithm could be a quasi-triangulation algorithm using calibrated parameters (distance from lens to sensor and horizontal offset between their centers of refractive index) or it could be a polynomial 10 approximation.
- the tracked motion of the pen can be used to recognize handwriting or capture drawings created using the pen or used in a wide variety of other applications.
- the tracked motion information can be sent to a local personal computer or to a central computer 15 through a personal digital assistant, a handheld computer, or a cellular phone for central storage and processing.
- the pen carries a finite source of light close to the tip. This source emits light which intensity in the test point depends on the XYZ position of a test point with the source in the origin.
- a multichannel detector is located at another location. It collects some portion of the light emitted by the source on a pen. Intensity delivered to different channels varies depending on the XYZ position of channel inputs with respect to the location of this 5 source. Intensity data are sufficient to calculate the XYZ position of a source relative to the detector. Intensity data are also subject to noise including source instability, detector noise, and other kinds.
- Our goal is to establish such a property of the detected signal that would be invariant to the motion of a pen in Z direction and to some sources of noise.
- This aperture may contain a lens, for example.
- Each detector can be a two channel detector, like a PSD, or a "multichannel" detector like a CMOS and CCD device.
- PSD two channel detector
- CMOS and CCD device CMOS and CCD device
- CMOS and CCD detectors have a number of pixels. Each pixel defines a particular space on the detector and has an analog output. These analog outputs can be digitized for later processing in firmware or software, or can be
- Algorithms used in software can alternatively be implemented in hardware in the same way.
- the pen is positioned in different known XY locations on the paper and readings of both detectors are taken and stored for future processing. After a sufficient number of points has been accumulated, common linear algebra methods are used to calculate the coefficients in (1).
- system (1) is substantially non linear. It is important to locate calibration points in such a way that a necessary resolution is achieved across the entire writing area. We do not locate the calibration points in the nodes of a regular rectangular grid. Instead, we use mathematical models to match a calibration
- the IR source in the pen can be an LED 13 that emits IR light 15 at the tip 17 of the pen when 20 pressure is applied during writing.
- the LED source 13 is formed by a ring of LEDs 19 arranged around the longitudinal axis 21 of the pen (only two LEDs are shown).
- Light from the LEDs is project downward toward the pen point and into a body/lens of acrylic material 18 that operates as a light pipe.
- the acrylic lens diffuses and transmits the received light so that light emitted from the pen is delivered along optical paths in 5 all directions around the pen.
- the pipe 18 is polished and reflective and concentrates light 502 from the LEDs 19 inside by not allowing the light to escape sideways.
- the bottom part of the pipe is also polished at the 45 degree conical surface 504 at the bottom of the 10 pipe.
- a reflective cylindrical shell 506 helps to confine and cause mixing of the light that is emitted from the LEDs.
- a conical body 508 supports the light pipe. Downwardly directed light within the pipe is reflected from the conical surface 504 and delivered to the air at all angles around the pen.
- Figures 5 and 6 illustrate side and top views of internal reflection of light in the light pipe. Most light from the LEDs passes along the length of the body of the pen and is reflected at a 90 degree angle toward the sensors. Some other light finds its way out of the pen at angles different from 90 degrees.
- the light that is emitted from the pen is confined to a vertical space 11 that is near to the writing surface 13 so that as much of the light as possible can reach the sensors (not shown), which are also positioned within a small distance of the writing surface.
- the pen in this example ( Figure 2) includes a ball-point pen cartridge 23 that terminates in a writing point 25.
- a pressure switch 26 delivers a signal that can be used to turn on the LEDs and to 10 trigger functions of circuitry 28 also mounted in the pen.
- Circuitry 28 and LEDs 19 are powered by a battery 30. The components are all held in a housing 15.
- the pen position in an x-y coordinate system 40 parallel to the writing surface is determined from two angles 15 alpha and beta that are sensed by the two sensors 20, 22, and the known distance 26 between sensors.
- the pen is powered by three miniature AAA-like NiCd rechargeable batteries 51 that are held in the back of the pen (For better weight distribution the batteries will 20 be moved closer to the tip, with the circuitry placed at the back).
- the batteries power the electronic circuitry 28 directly without any DC-to-DC converter.
- the power is delivered only when the pressure switch 26 is activated.
- the light activation switch travels only a short distance (e.g., 0.25mm).
- the switch is preloaded by a
- Pressure sensors would be an way to effectively match pressure on the pen refill with activation of LED, as many off-the-shelf 5 switches have an activation pressure above the desired level of 20 to 30g.
- the electronic board 28 positioned behind the battery generates modulation frequency pulses at approximately 100Hz and a 50% duty cycle for the IR LEDs along with the bursts of 1 to 10kHz to 10 generate pen on and pen off signals for sleep mode.
- the light emitted from the pen is visible in all directions to enable the pen to be used in any orientation in the hand.
- the LED light should be in an IR region away from the visible light spectrum so that ambient light from the sun and light fixtures does not interfere excessively with the IR emission and is not visible to the human eye.
- the IR source at the pen and the orientations of the sensors in the 20 holder are arranged to assure that as the pen tilts and rotates during normal writing or drawing, its IR beam reaches the sensors.
- Figure 9 shows a more detailed isometric view of a partially assembled pen.
- the sensors can be housed in a typical pen cap 70 in which the pen can be held when not in use.
- the pen is removed from the cap and 5 the cap is positioned at a stationary location near the writing surface and in the vicinity of the pen.
- the sensors are linear CMOS arrays (available, for example, as 1024 pixel arrays from Photo Vision Systems, LLC (PVS) (P.O Box 509, Cortland, NY 13045) (part number LIS1024)).
- PVS Photo Vision Systems, LLC
- Part number LIS1024 Part number LIS1024
- Other linear 10 CMOS sensors from PNS or other companies with the same or a different number of pixels could also be used.
- the analog output of each sensor is a sequence of 1024 analog signals, one from each sensor pixel.
- the holder may include a clip 62 to attach 15 the holder to the edge of a pad of paper or a notebook.
- a third sensor in the form of a photodiode 56 in the middle of the holder is used (among other things) to wake up the processor from a sleep mode (described below) when writing begins (e.g., the pen begins to emit light).
- the third sensor signal may also be used to synchronize the circuitry in the pen with the circuitry in the sensor system. All three sensors are covered by IR filtering windows that face the writing surface.
- the front surface 100 of each of the main sensors has a vertical height 104 of 125 micrometers and a distance 106 from the front surface 108 of the lens 110 of four millimeters.
- the FON 112 is 10 degrees.
- the pen 5 tip 114 directs IR light into the FOV when the pen tip is on the paper 116.
- the two sensors 88. 90 are positioned 100 mm apart.
- Each of the sensors has a field of view (FON) 94, 96 centered on an FOV axis 195, 197.
- the axes of the FOVs are not 10 parallel but are toed in by an angle 199 to increase amount of overlap of the FOVs.
- the FOV of each sensor has a breadth of 150° in the horizontal (x-y) plane and a height of +/-5° in the vertical plane.
- the FOVs do not cover some locations 101 on the writing surface 15 that are close to the edge of the paper 303, and the FOVs are arranged so that the dead zone 98 does not extend more than 25mm from the holder.
- the sensors 117 and the lenses 119 are mounted on a holding bracket 121 with 20 the IR filter 123 in front.
- the bracket is mounted on a printed circuit board 125 and is held in a housing 127.
- the centers of two main sensors are separated by 100mm.
- the light from the pen is collected from two sensors in order to identify the linear position of a modulated light source within a
- the linear position may be computed by triangulation, a lookup table, polynomial approximation, or a combination of any of these.
- the sensors are flat, linear, multi-pixel sensors. Different pixels of 5 each of the sensors are illuminated when the light source is in different locations within the field. As the light source moves across the field, the corresponding movement of the light across the pixels of the sensors may not be linear, but the lack of linearity can be handled because the linear position may be computed by 10 math and knowledge of optics combined with calibration data from the pair of sensors.
- the system uses, in some implementations (in the polynomial example shown below, the number of parameters can be greater)and supplied in a separate file) only four parameters to be passed from each pen to a host or server to process the data and linearize it.
- calibration parameters for a pen are stored in the memory of the pen during production test and calibration.
- the parameters can also be stored on the server or on a PC used by the user instead of being passed on from the pen during downloads.
- a lens or a set of lenses accompanies each sensor.
- the goals of the 25 optical system are to maximize the efficiency of the light delivery,
- a spherical lens 753 is used to focus the light on the sensor 755.
- the focal plane has a shape of a semicircle.
- the distance 759 from the lens to the sensor is optimized as are the other optical and mechanical properties of the lens including focal length, diameter, thickness, and material.
- an aspheric lens 760 may be designed with a focal point positioned on the sensor as the light source travels around the periphery of our field where the total power of the light source delivered to the sensor will be the weakest.
- this aspherical lens is designed to have a focal plane, which coincides
- two perpendicular cylindrical lenses 770, 771 may be used instead of one lens.
- the length of the sensor limits the focal length of the lens in the horizontal axis. Therefore the lens diameter must be small and the lens must be located close to the sensor. In the vertical axis the lens may be located further from the 25 sensor so the diameter can be larger. The larger diameter will allow
- the first cylinder (closer to the lens) will focus the light in the horizontal axis.
- This lens may be spherical because the spot size is not very important in the horizontal axis and will not vary that much within 5 our given field.
- the second cylinder will have power in the vertical axis. It is important to focus as much light as possible on the sensor in the vertical dimension. For this to be true, the light must travel an equal distance from the cylinder to the sensor for all angles. To accomplish this, the second cylinder should be bent into an 10 aspheric shape. Any of these two cylindrical lenses can be Fresnel lenses in order to save space.
- the sensor system is held in a housing 79 that has a bottom 80 and a top 82.
- the bottom 80 holds a clip 62 (not
- Paper can be inserted between the clip 62 and the bottom of the pen holder when a clip button 86 is depressed. When the button is released, the clip grasps the paper.
- the pen clip holds a 7mm-thick stack of paper sheets or a standard notepad 83. The clip positions the pen holder on the paper so that the side 87 that faces
- the pen is vertical with a tilt of no more than +/- 1 °, thus assuring that the sensors will receive IR light from the pen when it is being used to write on the surface.
- the holder can also just sit on top of paper or notebook without use of clip.
- the two sensors are mounted behind IR filtering windows 89, 91, and the photodiode 93 is mounted in the middle.
- An "ink well” 95 can receive the tip of the pen 97 for temporary storage, and a tube 99 provides a place 5 to store the pen. The pen can be fully inserted into the tube and the batteries in the pen can be recharged during storage.
- the clip mechanism shown in figure 33 is used.
- the figure shows the steps in operating the mechanism, as follows.
- Step 1 Mechanism is not activated.
- Step 2 When the push button 780 presses on a spring 782, the 15 latter releases the bracket 733, and lets the hinge spring 784 unfold the clip 785.
- Step 3 Paper 786 is inserted between the clip and the body 787 of the pen holder.
- Step 4 Lever 785 (the clip) is against the paper, and spring 788, 20 which is significantly weaker than the hinge spring gives in and starts collapsing. The clip rotates around rotating point 789.
- Step 5 The clip presses the paper against the bottom of the holder.
- Spring 788 collapses, and both levers move down by the amount of paper inserted.
- a clip button can be used to transfer horizontal motion to vertical 5 motion by a lever, which lowers the clip. Or, the button can push against a lever that rotates, transferring horizontal motion to vertical movement, which lowers a clip.
- the clip has two vertical sliding bars 790, 791 connected to a horizontal bar 792. There is a 10 " spring 793 between the horizontal bar and the body of the holder. There are two vertical guides 794, 795 for the vertical bars to go up and down.
- the button may move a lever in multiple axes to lower the clip.
- the button can activate a lever that rotates and moves linearly to lower the clip.
- the operator pushes down on a sliding panel (button) 901.
- the sliding button contacts the center of two spring-loaded levers 903, 905 moving these fulcrums down in the same direction of the button/panel.
- the 5 bottom ends of the levers are fixed about rotating pins 907, 909 and thus the bottom? ends move downward about twice the distance of the button.
- the top ends of the levers are fitted with pins 911, 913, which ride in guide slots 915, 917 and in two tabs that are bent up vertically from the bottom clip. The end result is a 10 movement vertically downward of the clip, which is about twice the distance as the button/panel travel.
- the sliding panel 930 may be placed in a horizontal orientation and, by means of a rigid flexible belt, achieve the desired result with a horizontal pushing motion as 15 opposed to the vertical motion of the button.
- FIG. 5 A circuit block diagram of the holder is shown in figure 18.
- An ASIC 205 is powered by a battery 511 , or from an AC adaptor 513, or from a USB connection to a host computer 211.
- the two CMOS sensors 201, 203 have outputs that are connected 165, 167 through operational amplifiers 169, 171 through a multiplexer 180 and a 10 12-bit A-to-D converter to the ASIC.
- the analog output of the CMOS sensor is subjected to signal processing in the form of offset cancellation and automatic gain control.
- the signal-to-noise ratio requirements of the processing implies use of 5V power for the CMOS sensors and all analog 15 signal processing circuitry.
- Some A/D converters are operated at a 2.5v reference, and the signal from the CMOS sensor may scaled down by factor of 2 using a resistive divider.
- the ASIC could be model Clarity 2B from Sound Vision, located in Framingham, MA) and is based on an ARM7 core.
- the ASIC 20 firmware implements data acquisition, data storage, file system management, I O service (LEDs and switches), RS232, and USB communications, power management for idle and sleep modes, optical calibration, and test mode.
- the multiplexer enables the A-to-D converter 182 to alternate between the two CMOS arrays 201 and 203 to minimize the time skew between the two sensors.
- the clock frequency of the A-to-D converter is 1.2 MHz.
- Each CMOS sensor is clocked at 600kHz. 5 Data acquisition uses the ASIC's direct memory access (DMA) facility.
- DMA direct memory access
- a wakeup input of the ASIC is driven by a PLL 513 that receives an input signal from the photodiode 515.
- the photodiode is driven by modulated light from the pen.
- the ASIC is clocked by a 48 MHz crystal and a clock divider 517.
- I/O features are provided through a USB port 211 and an RS232/IrDA port 209.
- Firmware and data are processed in SDRAM 207 and stored in a flash memory 519.
- An optional LCD 172 can be provided for user display.
- USB provides two data transfer modes from the holder to a PC: bulk and real-time. Real time transfer is interrupt driven and is used for keyboard and mouse replacement applications.
- a dual function transceiver is used to implement both RS232 and IrDA communication.
- the RS232 communication is used as a dial 20 up connection to the server over cellular phone.
- the holder can support different types of external connections, including USB, Serial, Parallel, IrDA, Bluetooth, Firewire, or any kind of communication port.
- USB Serial, Parallel
- IrDA IrDA
- Bluetooth Firewire
- Firewire any kind of communication port.
- connection is an external storage type of connection.
- Such a connection is made with a computer, or other device, called host, that is capable of displaying graphics and has an adequate user interface. While connected to a host device, the holder behaves as an external storage device. A user of the host device can browse
- the software residing on the host is capable of converting, displaying, printing, and editing on the host screen files stored on the holder or copied from the holder to the host. While connected to the host, the pen can also behave as
- connection is with a portable internet or modem enabled device, such as a cellular phone.
- a portable internet or modem enabled device such as a cellular phone.
- the optional LCD display notifies a user of the pen's status, for example, with respect to connections and downloads over Internet- ready cellular phones where communications are not reliable.
- This display can be mounted on top of the holder. If an LCD is used, the 25 LED may not be needed.
- a single three-color (green, yellow, red) LED 170 indicates normal acquisition of writing data, downloads to a PC and over cellular phone, and monitoring of battery and memory status.
- Pen, clip, and inkwell switches 141, 143, and 145 are used to control the ASIC and a reset switch 147 is used to reset the ASIC.
- Figure 19 shows a state diagram of the states of operation of the invention. Shaded blocks indicate study states. Clear blocks indicate transition states. Among other things, the figure identifies 10 the manner in which the multicolor LED is used to indicate the state of operation.
- the green light blinks as many times as there are pages in memory.
- the green light is not on at power up when the memory is empty, and stops blinking after 30 seconds or sooner if 15 the user starts to write.
- the LED is pale green.
- the green light goes off for faulty acquisition triggered by, for example, obstructed light, a pen that is off the writing surface, or a discharged battery.
- Low battery status is indicated by a blinking yellow light when no writing is occurring. However, when writing, the yellow light blinks intermittently with pale green if the battery is low.
- Memory nearly full status is indicated by a double-blink of the yellow light when writing is not occurring and a yellow light blinking intermittently with pale green when writing is occurring.
- Download status (which may start independently whether the pen 5 is in or out of the holder or ink well) is indicated by a bright green light after successful download. Blinking green, signifies that download is in progress. When no service is available for downloading or the download signal is week, a red light blinks. The red light double blinks for an Internet problem, for example 10 when a server is down. A triple blinking red light indicates a wrong setup for communication including a wrong user ID or server address. This requires a code sent back to pen from server after unsuccessful match of data from pen with account on database.
- Battery recharge status is indicated by a solid green light after a successful recharge and by a multiple blinking green light when the holder is plugged into the AC adapter and charging.
- a combination of signals from the battery monitoring circuitry and the fast charge signal from a charger (high when not charging) can be
- the pen can be used during recharging. If the pen is removed from the ink well and is used during recharging, the yellow light is replaced with all normal indicator lights described above.
- Writing to flash status is indicated by a continuous yellow light.
- the power up indications take the 10 priority. Then the trouble indications are displayed up after a 30 second timeout. If the ASIC needs to indicate low memory or low battery conditions during download, the download indications take precedence. After a reset of download status, the trouble indications are displayed.
- the clip switch 141 indicates that the clip is being opened and closed as a way to notify the circuitry that the user is beginning a new page.
- the pen switch 143 indicates when pen is in or out of the holder.
- An ink well switch 145 indicates when the pen is in or out of the ink well.
- the pen switch and the inkwell switch indicate when the pen is in the holder or the inkwell and remove power from the data acquisition and storage electronics when the pen is in the holder or
- the pen switch also opens new files (or pages) on activation, while the ink well switch does not.
- the clip switch indicates when the clip is activated, as well as a new page and beginning of a new file (each page is a file).
- the reset switch resets the ASIC if the software freezes.
- the switches are normally ON as follows:
- the holder also includes a miniature connector for USB and RS232 interfaces as well as an antenna for use with Bluetooth or other wireless technology.
- the USB and RS232 connector are also connected to the wake-up power circuitry so that pen holder can 15 power itself up when cable is plugged into the miniature connector.
- Angle signals generated by the sensors are processed by the ASIC and stored in flash for later transmission to other devices such as cellular phones, PDAs, and PCs (not shown) where they can be used for handwriting recognition or to capture drawings.
- the 20 transmission can be done using, for example, USB, RS232, IrDA, or Bluetooth protocols.
- the flash memory is structured as a FAT (file allocation table)- compatible file system, where each file represents one page of handwritten information.
- FAT file allocation table
- Each file has a unique name of 12 5 characters, including 3 characters of extension and a separating "dot".
- uncompressed data is stored in a temporary buffer in SDRAM and compressed by a data store task before being stored into a file in flash memory. Each page is stored in a separate file. A previous page is compressed before new page acquisition is started.
- Such a format comprises encoded data coordinates and timestamps.
- the file Before being compressed, the file has the following format:
- the file is structured in four byte segments. Each segment corresponds to either one pixel or one timestamp. Each pixel has a most significant bit (MSB) of zero, and consists of two 15-bit 5 numbers that are the sub coordinates of corresponding CMOS sensors. Timestamps are distinguished by a MSB of one, and can store either full date and time of the next pixel (called full timestamp), or incremental counter of pixels since the last full timestamp.
- MSB most significant bit
- Each file begins with the full timestamp.
- An incremental timestamp is inserted in the end of every written stroke. Because all pixels are scanned evenly in time, such a combination of timestamps enables efficiently recover the whole history of handwriting in the future processing.
- the PC When the holder is connected to a PC using a USB cable, the PC automatically recognizes the holder as a PC-compatible USB device, and the contents of the holder file system becomes visible for the PC through the PC-file system extension.
- the user can 20 browse through it and view the files using a handwriting viewer.
- the holder When an RS232 cable is connected between the holder and, for example, a cellular phone, the holder automatically powers itself up, and starts transmission of data files from the memory of the
- the data is sent in the compressed form to the server and is kept there until requested for an addressee. Then it is decompressed and 5 translated into one of the following formats: .tif, .pdf, .gif, .ps specific to e-mail or FAX service.
- a preprocessor (not shown) can be used for background cancellation, and storage into flash memory, while the
- the preprocessor performs all communication and I/O functions.
- the preprocessor can be implemented as a programmable device such as PLD, FPGA or digital ASIC or a DSP.
- a frequency multiplication is performed to generate a high-frequency pixel clock and a clock for the preprocessor from the pen LED
- the second processor can be a processor of another portable device such as a cellular phone or PDA.
- Position data is collected in a succession of samples spaced 10 20 milliseconds apart to adequately capture writing motion at a typical speed of 5 cm per seconds for a resolution of 0.5 mm.
- the ASIC operates as a master, generating the clock and all necessary signals for the sensors.
- the sensors in the holder use the pixel clock from the ASIC.
- a frame signal is generated by each sensor and read back into the ASIC.
- the LED pulses from the pen and the signal acquisition performed on the holder are not synchronized in some 5 implementations.
- the data acquisition is synchronized with the pen modulation frequency. Synchronization significantly improves angle resolution.
- the multiplexer data acquisition alternates between the two sensors 15 for each pixel.
- A-to-D converter Operating the A-to-D converter at a sampling rate of 1.2MHz maximum and alternating between the two sensors allows for a pixel sampling frequency up to 600kHz.
- Each CMOS array has 1024+4 pixels, which produces a frame rate of approximately 20 600Hz.
- a slower rate of 300Hz might be used to achieve more pixel exposure to light and accordingly better signal-to-noise ratio.
- Each sensor operates in a mode in which each pixel is reset after being read into A/D converter.
- the IR LED duty cycle is 50% out of three frame intervals. For that duty cycle, the LED frequency cannot exceed 200Hz.
- each CMOS delivers END_FRAME signals. From each CMOS the acquisition cycle for each of the three sequential frames of data is started by the END_FRAME signal, which coincides with the last pixel of the
- the ASIC extracts three arrays, each corresponding to 1024 pixels.
- the arrays must be correctly aligned so that the i-th element in each of them corresponds to the i-th pixel of the CMOS.
- This array A is 1024 elements long and carries the beam 5 information with the background removed.
- subarrays shorter than 1024-elements long can be extracted, for example, three 1020- element long subarrays, that start at pixels 3, 1032 and 2061 (base 10 0).
- the readouts of the two sensors are digitized simultaneously (or quasi-simultaneously when using only one A-to-D converter with a multiplexer).
- Determining the angle of receipt of the light at each of the sensors depends on determining the pixel location of the peak light intensity along the array of the sensor.
- the peak value and its index in the array A are found, call them Amax and M. If either of the two Amax values
- the point is discarded. In that instance the LED is considered to be off with the pen not touching the paper. If M ⁇ W/2 or M > (1024-W/2), the point is discarded as being too close to the edge of the field of
- the result, P is the fractional position of the beam with respect to the axis of the sensor (in pixels).
- the pixel pitch L 7.77 microns
- the refraction coefficient of the lens material is N (1.5 for glass, 1.4 for plastic, 5 1.8 for SF6). Parameters, distance D, index of refraction N, and horizontal offset, Off, will be adjusted using calibration data for correct mapping of writing.
- Base B the distance between sensors (mm); typically 150
- Points are stored as coordinate pairs (X,Y). When saving points into the memory, coordinates are saved continuously, except as follows:
- a marker (a pair of unique values) is written into the memory, for example (NaN,NaN) which will signify later that the pen was lifted (NaN stands for not-a-number as defined in the IEEE arithmetic standard). After that, no new points are added to 10 the file until the signal is detected again. This approach allows the pen to tell the playback program exactly where to interrupt the restored trajectory line.
- the size of the move squared is calculated as (X1-X0) 2 + (Yl-Y0) 2 .
- the typical value for the significance of the move squared is 0.04 mm 2 .
- Coordinates are stored in the temporary buffer and are compressed only before storing in flash memory. Each page is stored in a separate file. Therefore, there is no need for an end of page mark.
- the ASIC When the pen is taken out of the holder or the ink well for writing, the ASIC turns on in the sleep mode and waits until an optical signal is detected from the pen.
- the holder When the holder is awake and it detects that writing is interrupted 10 for a predefined period of time, the holder returns to the power- saving sleep mode.
- the ASIC enters sleep mode by reducing its normal 48MHz clock frequency to 750kHz. SDRAM update refresh rate also changes accordingly to keep data intact.
- the holder power is almost entirely off when the pen is inside the 15 holder or in the inkwell.
- RS232 receiver and USB monitoring circuits consume very little standby current. These circuits wake up and enable power to the rest of the electronics on detection of active levels for RS232 or USB, when connected to a cellular phone over the cable or by USB cable to a PC.
- the pen holder is 20 completely off when the pen is inside the holder.
- the pen transmits modulated IR pulses.
- the pulses are detected at the holder causing the PLL to wake up the 5 processor, which starts normal acquisition mode as soon as the ASIC switches back to the 48 MHz system clock.
- the modulation clock of the pen LED (represented by lkHz bursts in 10 the output light) is extracted using PLL circuitry 132 tuned to the modulation frequency of the IR light.
- All acquired data is initially stored in SDRAM 134 using DMA.
- the update rate of the SDRAM remains unchanged when going from acquisition mode to sleep mode.
- the memory requirement is 15 1 Mbyte for 50 pages of compressed or 10 pages of uncompressed data.
- the 5:1 compression algorithm must have fast and computationally simple coding with no limitation on decoding.
- the acquired data is initially stored in SDRAM during writing.
- the ASIC writes all data from SDRAM into flash memory 138. Only a short time is needed to write a full page of hand- written text data into flash. The transfer is indicated to the user by lighting a yellow LED 140 on the holder.
- 8Mbit flash memory stores compressed files representing a maximum of 50 pages of handwritten text.
- the compression algorithm allows at least 6-to-l compression without observable distortion of text.
- the holder is powered by two AA NiMH batteries connected in series to provide 3.0V.
- the pen's three NiCd batteries are recharged by a trickle cunent.
- the pen batteries have a large capacity and are almost 10 never recharged completely.
- the trickle cunent charging is enough to maintain the battery charge.
- a special mode is provided when the pen and the pen holder are both in the charger to charge all the batteries including the pen batteries with the full charging cunent.
- Battery life is ten handwritten pages or a week of average use 15 without compression of data for storage in memory.
- An average user may write 2 characters per sec, or 120 char/min, or 7200 char/hr.
- the average handwritten page is approximately 700 characters. To produce ten pages, the battery must work for 5 hours.
- the holder When connected to a USB port, the holder can get power from the USB host.
- the charge on batteries is maintained at a high enough level to start the circuitry prior to switching to USB power.
- Power from the USB connector is provided only after the ASIC establishes communication over the USB link and notifies a PC on
- the PC provides up to 0.5A.
- Battery charging current is set at 0.4A and is monitored to switch the charger into trickle charge.
- the holder circuitry is activated when the pen is taken out of the holder or the ink well. Some holder circuitry, like the RS232 driver and wake-up power circuitry take power directly from the battery. Other circuitry takes power from a 3.3v supply generated by an onboard switching regulator from the battery voltage of 2 to 3 volts. 10 When connected to a USB link, the 3.3v is generated from USB power.
- 5v is generated for the analog circuitry from the 3.3v supply.
- Synchronization of the pen and the pen receiver can produce a 15 better signal resolution and conespondingly better angle resolution and resolution of writing.
- the pen produces periodic bursts 401 of higher frequency pulses, such as pulses at 1 - 10kHz (suggest we show some timing diagrams) that can be 20 easily detected by the PLL.
- the PLL will detect not only the actual modulation clock but also its phase, which enables a signal to be generated to start data acquisition and synchronize it with the pen LED.
- control signals, LED_ON and LED_OFF trigger signal acquisition.
- only two frames will be required for background cancellation, one for the IR signal when the pen LED is on, and the other, when the LED is off.
- a shutter mode is provided that resets all pixels at one time.
- Vendors to manufacturers of digital sensors produce small power- saving sensors and sensors along with the image processing circuitry that can be integrated into the pen on paper or 3-D pen applications.
- 3-D positioning of a light spot is possible using two 2-D photo arrays. Projection of a point of light onto two planes defines a single point in 3-D space. When a trajectory of 3D positions is available, motion of an IR pointer-pen can control a 3-D object on a PC screen. When the pen moves in space, it drags or rotates the
- the DMA can handle the data acquisition, but the vertical
- PLL pen light detection circuitry
- Two separate channels can be used for analog signal processing 5 and A-to-D conversion. Such an implementation could use more economical parts that do not require fast settling times, frequency bandwidth and slew rate.
- CMOS sensors have a limited dynamic range. Although an 10 adjustable electronic gain may be used for both CMOSs simultaneously after the output of the CMOSs, this arrangement may be not be ideal, for two reasons.
- the signals for the two CMOSs may be different in magnitude when the pen is being moved in certain areas of the
- the gain of the CMOS can be changed by changing the exposure rate for each CMOS independently. As shown in figure 21, the pen transmission rate remains 100Hz, while the frame rate 601 of the
- CMOS is shifted among 300Hz, 600Hz, and 1200Hz.
- the background cancellation is straightforward.
- the algorithm uses every other frame (frames 1, 3, and 5).
- the algorithm uses every fourth frame (frames 1, 5, and 9).
- the 5 pixel rates are 300kHz, 600kHz, and 1.2MHz. Changing the frame rate can be accomplished by the ASIC without any additional hardware.
- Each CMOS may be connected directly to its own ADC or both could be connected to one ADC that would be able to handle 1.5 10 megasamples/sec and have a 4V reference voltage.
- the ADCs then may feed into a digital multiplexer so that the signals can be fed into the ASIC.
- PSDs may be used to detect the IR 15 light from the pen.
- Each PSD determines the angle between the page and a line of sight between the pen and the PSD. The two angles from the two PSDs and the distance between the PSDs are sufficient to compute the location of the tip of the pen.
- the IR light at the pen is modulated to generate pulses at a modulation frequency and with a 50% duty cycle, as described above.
- Two analog techniques may be used to discriminate the PSD signal that is translated into the angle for use in triangulation.
- the incident synchronous light pulses 5 are chopped at the light modulation frequency, and opposite gains (+1 and-1 respectively) are applied to those signals, depending on whether the LED is on or the LED is off. This allows for subtraction of background noise. Then the signal is integrated using a time constant that it is responsive to the signal variations 10 on one hand and averages out noise on the other.
- the modulation frequency could be 3kHz
- a second approach to discrimination uses a sample and hold technique.
- the shape of the optical signal has a 50% duty cycle at
- the modulation frequency is discriminated using a PLL and is used to trigger the sample and hold circuitry, while the strong optical pulse is actually sampled.
- the pulse amplitude is
- PSDs are extremely accurate in sensing and measuring the position of light on their photosensitive surfaces. They are inexpensive and require very little power consumption. The PSD implementation is also simpler than the CMOS one.
- FIG 23 shows a system timing diagram. When no trace is being made, the 1.0 microcontroller enters the idle mode, and after an arbitrary period of time, the sleep mode.
- the microcontroller is awakened from idle mode or sleep mode by either an interrupt or polling (TBD) of the following inputs: one of the four analog channels, when the signal at its modulation 15 frequency exceeds a certain threshold of the comparator; an interrupt from a USB port when presence of activity from a host is detected; one of its key buttons is pushed.
- TBD interrupt or polling
- the processor When the RAM becomes full or/and the boundary of a flash memory page is reached, the processor writes data from RAM to 20 flash memory. If acquisition continues and the page is full, the microcontroller start writing to flash. However, most of the flash operations should be done during idle cycles when there is no writing.
- Each PSD has two channels of analog signal processing.
- Each 25 channel has a cunent-to-voltage converter whose output is AC
- the signal is chopped with the modulated frequency of the pulsing IR LED (on pen), cunently 1kHz. .
- the chopper When LED emits light, the chopper has a gain of +1. When there is 5 no light, the gain is -1, therefore the signal is synchronously demodulated.
- the last stage is an integrator, whose output is close to DC. More precisely, it is a saw-tooth waveform due to charging and discharging of the integrating capacitor in the feedback of the amplifier.
- the A/D converter either a PC-based DAQ or an A/D of the microcontroller, samples the output at specified time intervals synchronously with the modulation frequency to cancel enors due to saw-tooth waveforms.
- a dynamic 15 change in reference voltage for the converter is used.
- the microcontroller always starts reading the A/D channels with the highest range and then divides it in half until the range is the most optimum for the signal.
- the chopper amplifier uses a replica of the modulation frequency 20 detected with an analog circuitry on each channel (four channels altogether). This signal is taken after the second gain stage, processed for detection of signal transitions, and then the recovered modulation pulses pass through OR gate to drive the chopper amplifier analog switch to change its gain between +1 and -1.
- the rotating light on the pen tip can be realized using several (e.g., eight) LEDs 623 that are triggered at times spaced apart by T/N, where T is the overall time period of the LED cycle, and N (e.g., 8) 10 is the number of LEDs.
- the signal source is at some location on an X-Y plane.
- Two signal detectors 619, 620 are located at two other fixed locations on the same plane. If the signal source has a radiation pattern such that the signal radiated in the positive X direction is in phase
- a second angle A2 will be formed as three lines intersect at the signal source. Again, the angle A2 between the lines at the intersection will be the same as the phase difference of the signal measured between i the detectors.
- the rotating light on the pen tip can be realized using several (e.g., eight) LEDs 623 that are triggered at times spaced apart by T/N, 5 where T is the overall time period of the LED cycle, and N is the number of LEDs.
- T is the overall time period of the LED cycle
- N is the number of LEDs.
- LED light emitting diodes
- eight light emitting diodes (LED) could be ananged in a circle pointing outward, spaced 45 degrees apart and driven by an signal oscillator with a 45 degree phase difference between adjacent LEDs.
- the three detectors 641 could be Positive Intrinsic region Negative (PIN) diode optical detectors driving a signal processing chain 642 consisting of a trans- impedance amplifier 643 and a high gain limiter 645 to remove any amplitude modulation in the detected signals.
- PIN Positive Intrinsic region Negative
- Phase detection could be accomplished with two edge-triggered one bit Up-Down counter type phase detectors 649, two binary counters and a clock running several decades above the signal frequency. If the counters are connected such that they count up with every clock cycle where the one sensor leads the phase of the
- a microprocessor can periodically read and reset all the counters, scaling the reading from the two counters connected to the phase detectors (dividing by) by the reading from the continuously running counter.
- Different pen light activation methods can be used, including conductive rubber, pressure sensitive materials or strain gauges.
- Pressure sensitive material allows for a variable pressure threshold and coordination of the switching point with the ink flow.
- Optical fibers could be used to collect light from an LED 20 and emit it in a 360° pattern around the tip of the pen. Individual LED chips could be located around the tip of the pen and emit light through a half reflective lens/window, such that 50% of light is emitted and the other 50% is reflected internally to be mixed with
- Light could be mixed from a single LED using special rings that redistribute the light for uniformity.
- the pen may be completely passive if the IR light source is located next to the sensor.
- a reflective surface would be provided near or at the tip of pen. The sensors would see reflection of IR light from the tip of the pen and compute angles as described above.
- the tip of the pen must be reflective only when pressed against 10 paper and ink is forming traces. Otherwise there will be enoneous traces in digital form with no conesponding traces on paper.
- Activation of the reflective mechanism can be mechanical or electrical.
- pressure on the tip will open up a sheath and expose reflective surface around the tip.
- pressure on the tip will activate liquid crystals or other photo technology that will make that material reflective to light. Reflection from other objects, like fingernails and rings, can be handled by using polarized IR light.
- the holder may have two reflectors, while the pen both emits light and receives reflections.
- the sensing element on the pen could be a 2-dimensional PSD or CMOS array. If flat 2-D sensors are used, the pen would not be omni-directional, but it
- the pen described above can be used to replace standard PC input devices such as a mouse and a keyboard.
- the sheet of paper, plastic or other flat surface may bear a printed keyboard pattern that will serve as a keyboard and mouse pad for, e.g., PC's, 10 handheld computers, and cellular phones.
- An electromc pen can be used in a mode that provides a highly reliable method for inputting text characters in addition to 5 recording handwritten images and lines.
- a sheet of paper or any other surface is all that users need to type with a pen.
- the spatial transcription capabilities of the electronic pen together with the keyboard template are used to substitute for the mouse or 10 keyboard.
- the paper keyboard can be multiple sizes based on the user's needs.
- the size can range from an 8 l A x 11 sheet of paper to the size of a cell phone cover. The user first selects the size of keyboard he desires and then calibrates by touching the pen to
- the paper can also be used in other modes to record drawings and handwritten notes and images. When done, the paper keyboard can 5 be discarded or folded for future use.
- the keyboard can also contain shortcut keys and function keys that enable more efficient interaction with a small device.
- Short cut keys can minimize the number of keystrokes required to enable commands.
- Short cut keys can be 10 customized based on the type of device being input into.
- the keyboard can also incorporate a section that serves as a paper mouse pad.
- the user can move the pen within a designated square space on the paper that in turn moves a pointer on the screen of a 15 device.
- the paper mouse thus serves as an alternative to keys and styli on screens as a means for navigating on the screen of a handheld or cellular phone.
- the paper keyboard also enables flexibility in inputting foreign characters. Keyboards can be created for several languages such as 20 Japanese, Korean, Spanish, French and Russian. A user can simply print out a new keyboard if they desire to input a different alphabet.
- the positions at which the pen is touched are tracked by tracker 707, converted to text and then sent to a portable device such as a cellular phone 711 or a palm computer 709.
- the keyboard can be folded or discarded after the use.
- the mouse and keyboard 10 can continue to be used and the pen serves as an alternative.
- the pen can operate in either a "pointing device" (mouse) or a "character input” (keyboard) mode.
- the mode can be selected by a dedicated hardware switch or button, on a pen holder or pen, or by sending a command from PC to a holder..
- pen operation is indistinguishable from that of a secondary (USB) mouse. It is a relative positioning pointing device moving the cursor on the screen.
- pen input can be received by a specially designed application using an available character recognizer to convert graphical input (strokes) into
- the pen is the only input device to the system.
- a software driver stack is modified to provide
- Mouse-mode operation is not affected and is identical to the first-described approach.
- pen input is recognized by available handwriting recognition software built into a keyboard filter driver 5 and then delivered to system input queue in a similar way to traditional keyboard input.
- Switching from mouse to pen input mode could be done by retractable pen refill action.
- the refill is inside the pen (pen cannot write), it is used as a mouse.
- writing is activated, the pen acts as a keyboard.
- the holder need not be of the kind that includes an inkwell as described above but can be any kind of device that can hold the sensors.
- the holder can be a simpler pen cap, as shown earlier, or could be any other kind of device whether or not it mates with or 5 caps the pen and whether or not it includes a clip or not.
- the holder could be inco ⁇ orated into a clipboard or a notebook, for example.
- the light in the pen can be fiber optics that deliver light to the tip and convey it in all directions around the pen in a disk-like pattern.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Position Input By Displaying (AREA)
- Length Measuring Devices With Unspecified Measuring Means (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
AU13510/01A AU1351001A (en) | 1999-10-27 | 2000-10-27 | Tracking motion of a writing instrument |
JP2001534078A JP2003515800A (ja) | 1999-10-27 | 2000-10-27 | 筆記用具の動きの追跡 |
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16175299P | 1999-10-27 | 1999-10-27 | |
US60/161,752 | 1999-10-27 | ||
US19549100P | 2000-04-10 | 2000-04-10 | |
US60/195,491 | 2000-04-10 | ||
US23091200P | 2000-09-13 | 2000-09-13 | |
US60/230,912 | 2000-09-13 |
Publications (3)
Publication Number | Publication Date |
---|---|
WO2001031570A2 WO2001031570A2 (fr) | 2001-05-03 |
WO2001031570A3 WO2001031570A3 (fr) | 2002-03-07 |
WO2001031570A9 true WO2001031570A9 (fr) | 2002-08-15 |
Family
ID=27388667
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2000/029758 WO2001031570A2 (fr) | 1999-10-27 | 2000-10-27 | Suivi du deplacement d'un instrument d'ecriture |
Country Status (3)
Country | Link |
---|---|
JP (2) | JP2003515800A (fr) |
AU (1) | AU1351001A (fr) |
WO (1) | WO2001031570A2 (fr) |
Families Citing this family (35)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6803906B1 (en) | 2000-07-05 | 2004-10-12 | Smart Technologies, Inc. | Passive touch system and method of detecting user input |
US6545175B1 (en) | 2000-10-19 | 2003-04-08 | Battelle Memorial Institute | Ester compounds and their use in forming acrylates |
US6954197B2 (en) | 2002-11-15 | 2005-10-11 | Smart Technologies Inc. | Size/scale and orientation determination of a pointer in a camera-based touch system |
US7532206B2 (en) | 2003-03-11 | 2009-05-12 | Smart Technologies Ulc | System and method for differentiating between pointers used to contact touch surface |
CN1293455C (zh) * | 2003-05-09 | 2007-01-03 | 凌阳科技股份有限公司 | 可判断反射面性质的光学输入装置 |
US7411575B2 (en) | 2003-09-16 | 2008-08-12 | Smart Technologies Ulc | Gesture recognition method and touch system incorporating the same |
US7274356B2 (en) | 2003-10-09 | 2007-09-25 | Smart Technologies Inc. | Apparatus for determining the location of a pointer within a region of interest |
US7355593B2 (en) | 2004-01-02 | 2008-04-08 | Smart Technologies, Inc. | Pointer tracking across multiple overlapping coordinate input sub-regions defining a generally contiguous input region |
US7460110B2 (en) | 2004-04-29 | 2008-12-02 | Smart Technologies Ulc | Dual mode touch system |
DE102004021325A1 (de) * | 2004-04-30 | 2005-12-08 | Siemens Ag | Optisches Eingabegerät |
US8120596B2 (en) | 2004-05-21 | 2012-02-21 | Smart Technologies Ulc | Tiled touch system |
EP1844388A2 (fr) * | 2005-02-04 | 2007-10-17 | PolyVision Corporation | Appareil et procede pour le montage d'une unite interactive a un ecran d'affichage |
US8094139B2 (en) | 2005-02-23 | 2012-01-10 | Anoto Ab | Method in electronic pen, computer program product, and electronic pen |
US20070165007A1 (en) * | 2006-01-13 | 2007-07-19 | Gerald Morrison | Interactive input system |
RU2457532C2 (ru) * | 2006-03-10 | 2012-07-27 | Кенджи Йошида | Система обработки ввода для устройства обработки информации |
US9442607B2 (en) | 2006-12-04 | 2016-09-13 | Smart Technologies Inc. | Interactive input system and method |
US8094137B2 (en) | 2007-07-23 | 2012-01-10 | Smart Technologies Ulc | System and method of detecting contact on a display |
JP2009045175A (ja) * | 2007-08-17 | 2009-03-05 | Topcon Corp | 眼科検査システム |
JP5119834B2 (ja) * | 2007-09-27 | 2013-01-16 | ぺんてる株式会社 | 筆跡入力システム |
US8902193B2 (en) | 2008-05-09 | 2014-12-02 | Smart Technologies Ulc | Interactive input system and bezel therefor |
US8339378B2 (en) | 2008-11-05 | 2012-12-25 | Smart Technologies Ulc | Interactive input system with multi-angle reflector |
JP2010257089A (ja) * | 2009-04-22 | 2010-11-11 | Xiroku:Kk | 光学式位置検出装置 |
JP2010277122A (ja) * | 2009-05-26 | 2010-12-09 | Xiroku:Kk | 光学式位置検出装置 |
JP5530809B2 (ja) * | 2010-06-01 | 2014-06-25 | 株式会社日立ソリューションズ | 位置検出装置、及び画像処理システム |
US8872800B2 (en) * | 2011-11-02 | 2014-10-28 | Microsoft Corporation | Optical tablet stylus and indoor navigation system |
CN102566797A (zh) * | 2011-12-07 | 2012-07-11 | 深圳易方数码科技股份有限公司 | 电子笔及跟踪器 |
US9129180B2 (en) | 2012-08-17 | 2015-09-08 | Blackberry Limited | Combination pen refill cartridge and active stylus |
US9096093B2 (en) | 2012-10-04 | 2015-08-04 | Blackberry Limited | Camera-equipped stylus pen cap |
EP2717122B1 (fr) * | 2012-10-04 | 2018-12-05 | BlackBerry Limited | Capuchon de stylet équipé d'une caméra |
US10401498B2 (en) * | 2013-06-06 | 2019-09-03 | Ams Sensors Singapore Pte. Ltd. | Sensor system with active illumination |
US10496226B2 (en) | 2015-01-08 | 2019-12-03 | Sharp Kabushiki Kaisha | Optical sensing unit and touch panel device including the same |
CN115050106A (zh) | 2017-03-13 | 2022-09-13 | 田谷圭司 | 电子设备以及信息处理方法 |
DE102017006740A1 (de) * | 2017-06-22 | 2018-12-27 | Staedtler Mars Gmbh & Co. Kg | Elektronisches Gerät zur Erzeugung analoger Abstriche und zur digitalen Speicherung der analogen Abstriche, sowie Eingabesystem und Verfahren zur Digitalisierung analoger Aufzeichnungen |
CN111324217A (zh) * | 2018-12-13 | 2020-06-23 | 鸿合科技股份有限公司 | 电子白板及板书处理方法、粉笔扩展装置、电子设备 |
CN111397510B (zh) * | 2020-03-24 | 2021-05-18 | 青岛罗博智慧教育技术有限公司 | 一种数字轨迹记录装置及坐标校准方法 |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE4111710C2 (de) * | 1991-04-10 | 1995-01-12 | Data Stream Corp | Drahtlose Eingabevorrichtung für Computer |
WO1995014286A1 (fr) * | 1993-11-17 | 1995-05-26 | Microsoft Corporation | Systeme de saisie informatique a crayon sans fil |
TW256900B (fr) * | 1993-11-25 | 1995-09-11 | Alps Electric Co Ltd | |
US6153836A (en) * | 1997-04-02 | 2000-11-28 | Goszyk; Kurt A. | Adjustable area coordinate position data-capture system |
US5900943A (en) * | 1997-08-29 | 1999-05-04 | Hewlett-Packard Company | Page identification by detection of optical characteristics |
JP2002523830A (ja) * | 1998-08-18 | 2002-07-30 | デジタル インク インコーポレーテッド | 絶対的および相対的な位置決めのための検出センサを有する手書き用デバイス |
-
2000
- 2000-10-27 JP JP2001534078A patent/JP2003515800A/ja active Pending
- 2000-10-27 WO PCT/US2000/029758 patent/WO2001031570A2/fr active Application Filing
- 2000-10-27 AU AU13510/01A patent/AU1351001A/en not_active Abandoned
-
2009
- 2009-04-02 JP JP2009090063A patent/JP2009187565A/ja active Pending
Also Published As
Publication number | Publication date |
---|---|
AU1351001A (en) | 2001-05-08 |
WO2001031570A3 (fr) | 2002-03-07 |
WO2001031570A2 (fr) | 2001-05-03 |
JP2003515800A (ja) | 2003-05-07 |
JP2009187565A (ja) | 2009-08-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7268774B2 (en) | Tracking motion of a writing instrument | |
WO2001031570A9 (fr) | Suivi du deplacement d'un instrument d'ecriture | |
KR100257368B1 (ko) | 펜컴퓨터 | |
US6573887B1 (en) | Combined writing instrument and digital documentor | |
CN100422915C (zh) | 通用计算设备 | |
US9207813B2 (en) | Determining the location of the tip of an electronic stylus | |
EP3420437B1 (fr) | Crayon de positionnement sans fil doté d'une pointe sensible à la pression | |
US20090115744A1 (en) | Electronic freeboard writing system | |
US20060028457A1 (en) | Stylus-Based Computer Input System | |
US20110199305A1 (en) | Mouse controlled by movements of fingers in the air | |
KR20170096214A (ko) | 전자 펜 | |
JP2011521364A (ja) | 格納式ペン先及び力センサを備える電子ペン | |
WO1995014286A1 (fr) | Systeme de saisie informatique a crayon sans fil | |
KR102338351B1 (ko) | 아날로그 스트로크들을 생성하고, 아날로그 스트로크들을 디지털 방식으로 저장하기 위한 전자 디바이스, 및 아날로그 기록들을 디지털화하기 위한 입력 시스템 및 방법 | |
CN201562264U (zh) | 自适应型智能互动投影机 | |
CN1310122C (zh) | 跟踪一书写工具位置的设备 | |
KR100360477B1 (ko) | 무선전자펜 | |
WO2007046604A1 (fr) | Dispositif de saisie d’informations numeriques | |
CN101071351A (zh) | 跟踪一书写工具位置的设备 | |
CN112649049B (zh) | 一种数码笔及握姿检测方法 | |
CN209803741U (zh) | 一种笔迹记录传输笔 | |
JP5272401B2 (ja) | 手書き筆跡入力システム | |
JP2000347806A (ja) | 座標入力ペン | |
EP1048017A1 (fr) | Crayon electronique | |
CN113296620A (zh) | 电子手写笔和绘图系统 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AK | Designated states |
Kind code of ref document: A2 Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CR CU CZ DE DK DM DZ EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG US US US UZ VN YU ZA ZW |
|
AL | Designated countries for regional patents |
Kind code of ref document: A2 Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
DFPE | Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101) | ||
AK | Designated states |
Kind code of ref document: A3 Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CR CU CZ DE DK DM DZ EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG US US US UZ VN YU ZA ZW |
|
AL | Designated countries for regional patents |
Kind code of ref document: A3 Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG |
|
WWE | Wipo information: entry into national phase |
Ref document number: 00814981X Country of ref document: CN |
|
ENP | Entry into the national phase |
Ref country code: JP Ref document number: 2001 534078 Kind code of ref document: A Format of ref document f/p: F |
|
COP | Corrected version of pamphlet |
Free format text: PAGES 1/33-33/33, DRAWINGS, REPLACED BY NEW PAGES 1/30-30/30; DUE TO LATE TRANSMITTAL BY THE RECEIVING OFFICE |
|
REG | Reference to national code |
Ref country code: DE Ref legal event code: 8642 |
|
122 | Ep: pct application non-entry in european phase | ||
122 | Ep: pct application non-entry in european phase |