WO2020039460A1

WO2020039460A1 - Electronic writing system

Info

Publication number: WO2020039460A1
Application number: PCT/IN2019/050613
Authority: WO
Inventors: Govinda Rajulu ITHAM
Original assignee: Durjeya Solutions Private Limited
Priority date: 2018-08-22
Filing date: 2019-08-22
Publication date: 2020-02-27

Abstract

An electronic writing system comprising: an electronic input device; a substrate comprising uniquely identifiable markers for interaction by said electronic input device; and a central computing unit for processing data generated during interaction of said substrate and said device.

Description

ELECTRONIC WRITING SYSTEM

FIELD OF THE INVENTION

[0001] The invention relates generally to an electronic writing device for inputting figures, symbols, characters and the like into a data processing device and a substrate thereof.

BACKGROUND OF THE INVENTION

[0002] A wide variety of electronic writing systems have been proposed. In general, an electronic writing system allows a user to interface with a computer or other electronic device using a writing instmment (often referred to as a“digital pen” or a“stylus”). The electronic writing system includes means for recording the positions of the writing instmment across a substrate over time. These positions are recorded as strokes or traces. The recorded strokes may be stored in the form of a virtual copy of the original movements of the writing instrument or they may be interpreted by the electronic writing system.

[0003] In one approach, there is provided an electronic writing instrument to be used along with a stationary means for detecting the writing instrument position. The said means is fixed at a corner of the substrate, so that when the writing instrument writes/draws on the substrate, real time coordinates of the instrument are calculated, based on the physical distance of the writing instrument from the stationary means. In some implementations of the writing instrument, the stationary means may be a detachable embodiment of the writing instmment. However, such instrument is unable to differentiate multiple pages. This disadvantage has been tried to be addressed by mandating a user to enter a unique page number on the paper and/or pressing a button on the writing instrument or stationary means indicating that the user had entered a page number. The entered number can be digitized and associated with the digital copies of the documents, while storing them. Even so, the instmment is disadvantageous to the extent that the stationary means must be fixed at one position and any displacement thereof, including any inadvertent displacement, may lead the writings to get placed far away or get overlapped with whatever was written/drawn prior to the stationary means’ displaced position. In yet another limitation, if the user wants to return to the same page (after finishing writing/drawing on it and moving on to another page) it would be almost an impossible task to position the stationary means or detachable embodiment of the writing instrument exactly back at the same location. [0004] In yet another approach, US2005243656A1 teaches a writing instrument having one or two“2- axis accelerometers” incorporated therein for detecting pen motion. However, the said prior art is unable to capture the user’s handwriting movements in an accurate manner. US2005243656A1 teaches that the data points for a square are plotted as a two-dimensional parallelogram, with two sides being much longer than the other two, and the parallelogram rotated about 45 degrees from the original, written position. The rotation of the parallelogram is a result of the orientation of the writing instrument in the user's hand. Every inadvertent rotation of the instrument in the user’ s hand will affect the data points in such a way that the data points when plotted together (digitally) to form strokes (of the user’ s hand-writing/hand drawing) may overlap with each other, may get separated from each other or may get distorted/elongated/compressed. This results in inaccurate representation of user’s hand writing/hand-drawing movements.

[0005] In one approach, there is provided an electronic writing instrument that works in conjunction with a specially printed paper (or substrate), each page being capable of being uniquely identified and each being incorporated with location coordinates at each and every part of the page, by a fine -print code. This code is spread from edge-to-edge on the page and gives a greyish shade to the entire surface of the page. The writing instrument has a camera or any image capturing device mounted thereon which keeps on capturing images when the instrument is in contact with the paper and in motion. A processor in the instrument body determines the motion based on the captured images. Code on each such captured image, when processed, yields the unique identity of the page and at least one set of coordinates representing the instantaneous position of the nib (writing tip) of the writing instrument with respect to a fixed reference point (normally top-left corner of the page). The strokes of the user’s handwriting/hand-drawing are then digitally reconstructed into a digital document by mapping the deduced coordinates in sequence according to the date-time stamps at which the images were captured. However, one limitation with this approach is to create uniquely identifiable pages specific to a user or a uniquely identifiable instrument specific to a user. For a set of uniquely identifiable pages, it would be necessary to print universally unique pages for each user and store those page identifiers against the user’s unique identifier (and/or the unique instrument ID) in the central database. In that scenario, the fine-print code on each page must be distinct from other pages (within the user base) and the printing must be customized to that page only and therefore, printing cost of such pages surge. Additionally, such pages cannot be printed on mass printing technologies like offset printing which can print huge number of prints of same content at a very low-cost; because output from such printing may contain blemishes which would conflict with the fine -print code. Offset printing technology comes with its own set of quality problems like sporadic smudges and other blemishes. When the fine print is mixed up with print-blemishes, the user’s hand -writing/hand-drawing written/drawn on such areas cannot be captured.

[0006] In view of the above, there is a need for a versatile and effective writing instrument, whose orientation parameters on a substrate may be accurately determined. Such instrument must be cost-effective and therefore, be available to the masses.

SUMMARY OF THE INVENTION

[0007] In view of the shortcomings of the prior art, it is an object of this invention to provide an electronic writing system comprising, an electronic input device; a substrate comprising uniquely identifiable markers for interaction by said electronic input device; and a central computing unit for processing data generated during interaction between said substrate and said device.

[0008] The electronic input device comprises of a longitudinally extended pen-shaped housing comprising a proximal end and distal end; a cartridge having a nib for contacting said substrate; an electronic module placed within the housing; and at least one battery. . The electronic module comprises of 3-axis accelerometer, 3-axis gyroscope and 3-axis magnetometer or any microelectromechanical system (MEMS) to measure the relative movement of the pen in the X, Y and Z axes with respect to the substrate, which in turn may be used to provide three- dimensional co-ordinates representing the movement of writing device. In a specific embodiment, the accelerometer provides acceleration and deceleration of the writing device as the device moves on the substrate, the gyroscope gives the readings of instrument’s tilt, rotation and inclination as well as the orientation of the instrument in the user’ s hand, the magnetometer compensates and corrects the drift in the readings. In a specific embodiment, the electronic module is preferably placed within the housing of the mounted on the instrument within a cavity, so that the writing experience of the user is not altered in anyway. [0009] It is yet another object of this invention to accurately capture a user’s identity while the user is writing or drawing on the substrate, thereby making the electronic input device re-usable for a plurality of users. The electronic module is configured to provide data, which can be used to identify user strokes and categorize and process each stroke as a set of points having plurality of attributes such as - X, Y and Z-coordinates, date and time stamp, substrate number (page number), stroke-width and at least one unique identity of the user, remotely within the electronic input device. The stroke width is directly proportional to the pressure exerted by the person on the substrate via the electronic input device. The unique identifiers to distinguish a user from the other users may be obtained from scanning fiducial markers, fingerprint recognition, facial recognition etc.

[0010] In addition to the motion sensor in the electronic module, the module also comprises a plurality of sensors, ICs and other electronic components including an antenna mounted inside the module for transmitting the set of points to a computing device, for reconstruction of the user strokes.

[0011] It is yet another object of this invention to provide a substrate for use by the electronic input device. The substrate comprises of non-obtrusive uniquely identifiable markers (i.e. not printed all over the area of the substrate) printed thereon, and therefore does not distract the user in any way; unlike prior art as discussed at paragraph [0005] above. The written strokes of the user on the substrate whether visible or invisible, are deduced by combining instantaneous coordinates of the nib of electronic input device which are calculated by running image- processing algorithms on images of the substrate (captured at high speed) combined with or without the help of MEMS, which is discussed at paragraph [0008] above.

BRIEF DESCRIPTION OF THE FIGURES

[0012] A more complete appreciation of the present invention and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:

Fig. 1 illustrates a perspective view of an electronic input device as an embodiment according to the present invention. Fig. 2 illustrates a perspective view of a substrate for the electronic input device as an embodiment according to the present invention.

DETAILED DESCRIPTION

[0013] While the invention is susceptible to various modifications and alternative forms, an embodiment thereof has been shown by way of example in the drawings and will be described here below. It should be understood, however that it is not intended to limit the invention to the particular forms disclosed, but on the contrary, the invention is to cover all modifications, equivalents, and alternatives falling within the spirit and the scope of the invention.

[0014] The term“comprises”, comprising, or any other variations thereof, are intended to cover a non exclusive inclusion, such that a setup, structure or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or structure or method. In other words, one or more elements in a system or apparatus proceeded by“comprises... a” does not, without more constraints, preclude the existence of other elements or additional elements in the system or apparatus.

[0015] Fig. 1 illustrates a perspective view of an electronic input device as an embodiment according to the present invention. The electronic input device comprising a longitudinally extended pen shaped housing comprising a proximal end and distal end. The electronic input device also comprises of an electronic module (1) placed within the housing, a cartridge having a nib at the proximal end, a battery (3), an antenna (4), an image sensor (5), a capacitance sensor (6), a pressure sensor (7), a touch sensor (8), a fingerprint sensor (9), a microphone (10), an image and/or video capturing device (11), a speaker unit (12), an infrared light emitting diode (13), status indicator light emitting diodes (14), charging socket (15), a piezo transducer (16) for energy harvesting.

[0016] In an embodiment of the present invention, the electronic module (1) comprises of a digital processor, a memory unit, a reset switch, a real-time clock, a power supply unit, a battery charging unit, USB to UART bridge, various individual ICs to augment the array of sensors and a buzzer, crypto authentication device and supporting components like LDO regulators, filters, capacitors, resistors, transistors, inductors, fuses, diodes etc. Further, the electronic module also incorporates a microelectromechanical system (MEMS). In an exemplary embodiment, the MEMS unit in the electronic module may comprise of 3 -axis accelerometer, a 3-axis gyroscope and a 3-axis magneto meter. In an exemplary embodiment, the accelerometer is respectively disposed in the directions of X-axis, Y-axis, and Z-axis, of a three-dimensional co-ordinate system, all of which are intersected perpendicularly to each other, wherein the sensors in the accelerometer detect, respectively, the accelerations in the X- axis, Y-axis, and Z-axis directions at the proximal end of the writing instrument. The gyroscope detects the angular velocities of the writing instmment around the X-axis, the Y-axis, and the Z-axis. Also, the gyroscope helps sense linear acceleration separately from gravity and track small, quick movements which are typical in a hand-writing/hand-drawing context. Furthermore, the movement distance of the proximal end in the scenario where the substrate is not horizontal is calculated by combining the readings from the accelerometer and the gyroscope together. Compensation of undesired drift in the position signal of the writing instrument around the X-axis, Y-axis and the Z axis is precisely determined by the magnetometer.

[0017] Fig. 2 illustrates a perspective view of a substrate for the writing instrument as an embodiment according to the present invention. The length and width of the substrate maybe standard or customizable and the substrate intends to indicate a surface on which a user can create strokes and thereby write/draw thereon, visibly or otherwise. The substrate is printed with one or more non-obtrusive markers.

[0018] In an exemplary embodiment, such marker may be selected from a group consisting of fiducial marker, a one dimensional or a two-dimensional barcode or any custom-developed code or any like, preferably a custom-developed code. In an exemplary embodiment, a marker can also be a finite area (with clear boundaries) pre-filled by printing one pre-defined colour uniformly within the boundaries. In an embodiment, markers of one or both of the above-mentioned types can be used.

[0019] The placement and size of the markers are non-obtrusive and are created on the substrate in such a way as to neither distract the user’s attention nor give any discomfort to the user’s eyes. In an exemplary embodiment, a marker merely represents the identity of a finite area on the substrate. A marker may be of any shape, regular or irregular, such as a 10mm x 10mm square area or area of an irregular shape bounded by a contour.

[0020] The page illustrated in Fig. 2 appears in the form of a standard ruled paper, available at large.

However, the lines printed on this paper are not single contagious lines. They are, rather, comprised of different markers printed consecutively one after the other. However, the markers are printed in fine print and hence appears as a standard ruled paper for any user. Fig. 2 also illustrates magnified lines and how one marker may look under a magnifying lens.

[0021] In another exemplary embodiment of the present invention, the substrate may look like a typical form such as a user fillable form which contains fillable boxes wherein users write a letter, digit or fill-in a mark like check-mark, cross-mark etc. In such an embodiment, a tiny marker is printed near each fillable box in a non-obtrusive way.

[0022] In another exemplary embodiment, the markers are printed in the substrate (i) consecutively representing ruled lines as in standard ruled paper and/or (b) singularly only at places of interest. A place of interest may represent any shape, regular or irregular bounded by a contour (visible or invisible). In a preferred embodiment of the present invention, the custom- developed code is made of a series of consecutive spaces of varying widths separated by printed blobs having same width and height, wherein a predefined set of consecutive spaces constitute an individual marker, as shown in FIG. 2. The markers are printed on the substrate either singularly or in a linear arrangement forming a chain (looking like a line), in which each marker is bounded by another marker either at preceding end or succeeding end or both. That is, each marker is surrounded by other markers only on two sides and not on all sides. To the contrary, in the prior art as discussed at paragraph [0005] above, where, the, codes are surrounded by other similar markers on all eight sides. Further, the blobs that lie between consecutive spaces in the custom developed code do not involve bars of different thicknesses or bars of different heights. Rather, the custom-developed code is made by placing the bars of same width and height with varying spaces between individual bars.

[0023] When the nib of writing instrument placed in the proximal end of the writing instrument comes in contact with the substrate, the capacitance sensor (6) and/or the pressure sensor (7), which is configured to sense the said contact, sends a signal to the electronic module (1). The image sensor (5) located adjacent to the nib of the writing instrument in the proximal end captures image of the substrate surface at a very high speed. In an exemplary embodiment, the image sensor (5) is adapted to capture an image every 20 milliseconds or less. The processor then analyses the captured image and decodes the marker on the substrate. In an exemplary embodiment, the processor identifies the finite area on the substrate associated to that marker. The MEMS unit provides the‘orientation related data’ including the, acceleration, tilt and rotation of the writing instrument with respect to the surface of the substrate. Based on the marker location combined with or without the help of‘orientation related data’ provided by the MEMS unit which as discussed at paragraph [0008] above, the processor determines the exact coordinates of the nib of the writing instrument vis-a-vis any one comer of the substrate. Thereafter and until the nib lifts up or breaks contact with the surface of the substrate, all the motion data of the writing instrument are captured, in a similar fashion, by the instrument.

[0024] In cases where the printing of the marker on the substrate is errant or in the scenario, when the print of a marker is missing or is defective, the processor is unable to analyse the image captured by the image sensor (5). U ndcr such circumstances, the MEMS unit will continue to log data on the movement and orientation of the electronic input device until the device encounters a properly printed marker. On encountering an accurate marker or a properly printed marker, the processor is configured to analyse the coordinates of the said properly printed marker and deduce the correct coordinates of the previously encountered missing or defective markers. In this way, the writing instrument is tolerant to sporadic absence or defects in marker prints.

[0025] Simultaneously, the processor processes each stroke of the user as plurality of attributes such as - X, Y and Z-coordinates, substrate number (page number), date and time stamp provided by the real-time clock, stroke- width provided by the capacitance sensor (6) and/or the pressure sensor (7) and at least one unique identifier of the user, provided by scanning a fiducial marker by the image sensor (9), scanning the fingerprint by the fingerprint sensor (9), or facial- recognition algorithms applied to the input image/video captured by the image/video capturing device (11), while the user is writing/drawing using the writing instrument. The unique identifiers of the user may also include style of handling the instrument, to distinguish a user from the other users, whereby for example, samples of the instrument’s tilt, orientation, acceleration, deceleration etc. are captured for fixed time spans at different intervals and analysed to yield a set of‘instrument handling traits’ attributable to the user. The plurality of attributes and the user information are transmitted to a computing device or a central database via the antenna module (4) of the writing instrument. In an exemplary embodiment, the processor transmits the said signal through a Bluetooth-paired mobile device or a Wi-Fi connection to a Wi-Fi Receiver. Technologies/protocols used to transmit the said signal may include (but not limited to) Bluetooth, Wi-Fi, LoRaWAN, NB-IoT, LTE-M, Cellular 2G/3G/4G or 5G.

[0026] In yet another embodiment, where the marker is a pre-filled color, the substrate may be of the nature of a comic book/illustrated-story book, printed in colour. When the nib of writing instrument touches an area, the writing instrument is adapted to recognize the color on the substrate and in some cases, both the shape and color of the set of marker objects. In this embodiment, different areas of the illustrations are pre-printed with pre-defined colours, which are mapped to identify different parts of the illustration. Pre-recorded voices tagged to each identifiable area are played accordingly. For example, on the nib of writing instmment touching the body of ‘Donald Duck’ in a colored illustration, the speaker unit (12) on the writing instrument provides a voice output announcing -“This is Mr. Donald Duck”. The electronic module on the writing instrument is configured to recognize the colour of the body of‘Donald Duck’ and is pre-recorded with a message to identify a tagged area. The processor runs colour- recognition-algorithms on the image captured by the image sensor (5) and then retrieves the matching voice-recording to play it via the speaker unit (12). In an exemplary embodiment, the sequence of touching each identifiable area (by the nib of writing instrument) and/or the associated pauses and intervals may be used to analyse the user’s soft skills comprising cognition, response -time, analytical, problem-solving, flexibility, adaptability, aptitude, vocabulary etc.

[0027] Within a central computing unit, the digital strokes are reconstructed from the plurality of attributes and compiled into unique pages and stored against the identity of the user. In an exemplary embodiment, the user information and the reconstructed user writings (or drawings) are archived in his/her secure account in the central database (like cloud storage). In an embodiment, the processor is configured to auto-deliver the documents having the reconstructed user writings to user-chosen email IDs and the like. Yet in another embodiment, a digitized copy of the user’s hand writing is also stored against the identity of the user. The digitization of user’s hand writing may use one or more technologies which comprise of digital ink character recognition, machine learning etc. [0028] In yet another embodiment, the markers alone (without depending on output from MEMS) may be used to calculate instantaneous coordinates of the nib of the electronic input device with respect to the substrate’s surface at a speed of 30 or more coordinates per second. The above- mentioned calculation of instantaneous coordinates may be done within the electronic input device and then transmitted to a computing device or a central database via the antenna module. The above-mentioned calculation of instantaneous coordinates is done in a central computing unit upon receiving raw data from the electronic writing device via the antenna module. The raw data is the identity of the marker and its relative position to the nib of the electronic input device obtained by running image-processing algorithms on images of the substrate (captured at high speed). The raw data may also include the instantaneous outputs from the MEMS.

[0029] In this way, the electronic writing system is not only cost-effective but also re-usable with any user, such that the digital copies of any specific user will securely get stored against his/her unique identifier. Owing to the simplistic design of the substrate, the substrate is capable of mass production and will be of the same price as any standard ruled-paper or ruled-book, which can be printed via offset printing. Any user can use any writing instrument using any substrate and there is no requirement of a writing instrument or a substrate associated to a single user. The writing system envisaged herein, can confidently and reliably leverage the low-cost benefit of mass-duplicate-printing technologies like offset printing and thus make itself available to the better usage of‘common man’.

Claims

We Claim:

1) An electronic writing system comprising: an electronic input device;

a substrate comprising uniquely identifiable markers for interaction by said electronic input device; and

a central computing unit for processing data generated during interaction of said substrate and said device; said electronic input device comprising: a longitudinally extended pen-shaped housing comprising a proximal end and distal end; a cartridge having a nib for contacting said substrate;

an electronic module placed within the housing; and

at least one battery, wherein said uniquely identifiable markers are printed over a finite area on the substrate.

2) The system as claimed in claim 1, wherein said marker is selected from a group consisting of fiducial marker, 1- dimensional barcode, 2-dimensional barcode, April-tag or a custom developed uniquely identifiable marker, preferably a custom developed marker.

3) The system as claimed in claim 2, wherein the custom developed marker is made of a series of consecutive spaces of varying widths separated by printed blobs having same width and height, wherein, a predefined set of consecutive spaces constitute an individual marker.

4) The system as claimed in claim 3, wherein the custom-developed codes are printed on substrate in a singular and/or linear arrangement.

5) The system as claimed in any of preceding claims, wherein said custom-developed marker in the substrate is bounded by other custom developed marker either at preceding end or succeeding end or both. 6) The system as claimed in any of preceding claims, wherein said uniquely identifiable marker on the substrate is either visible or non- visible.

7) The system as claimed in claim 1, optionally comprising, a retraction mechanism for slidably moving said cartridge between a retracted position wherein said nib is retracted within the housing of said electronic input device and an extended position wherein said nib protrudes from housing of said electronic input device.

8) The system as claimed in claim 1, wherein said electronic input device further comprises of: at least one antenna

at least one image sensor,

at least one capacitance sensor along with or without one pressure sensor,

at least one light emitting diode,

at least one charging socket, and

and, optionally, may include:

at least one touch sensor,

at least one fingerprint sensor,

at least one audio input/output device,

at least one image or video recording device,

at least one piezo transducer.

9) The system as claimed in claim 1, wherein said electronic module comprises of: at least one digital processor,

at least one memory unit,

a reset switch,

a real-time clock,

at least one battery recharging unit,

at least one power supply unit, and

additional supporting components.

10) The system as claimed in claim 1, wherein said electronic module further comprises of a microelectromechanical system to collect data on acceleration, tilt and rotation of electronic input device with respect to surface of substrate. 11) The system as claimed in claim 8, wherein said microelectromechanical system comprises of at least one 3-axis accelerometer, at least one 3-axis gyroscope and at least one 3-axis magnetometer.

12) The system as claimed in claim 6, wherein said image sensor is configured to generate data by capturing images of the uniquely identifiable markers in the substrate indicating the interaction of electronic input device with the substrate.

13) The system as claimed in claim 7, wherein said digital processor configured to analyze the data generated by said image sensor.

14) The system as claimed in claim 6, wherein the antenna transmits the data to the said central computing device where the data is reconstmcted thereby obtaining a digitalized copy of a user’s writing.

15) A substrate comprising, uniquely identifiable markers which individually comprise of a series of consecutive spaces of varying widths separated by printed blobs having same width and height, wherein a predefined set of consecutive spaces constitute an individual marker, wherein the uniquely identifiable markers are printed over a finite area on the substrate in a singular and/or linear arrangement, and wherein, individual uniquely identifiable marker in the substrate is bounded by other individual uniquely identifiable marker either at preceding end or succeeding end or both.

16) The substrate as claimed in claim 15, wherein said uniquely identifiable markers in the substrate is either visible or non- visible.