WO2012049592A2

WO2012049592A2 - Electronic signature apparatus and method

Info

Publication number: WO2012049592A2
Application number: PCT/IB2011/054390
Authority: WO
Inventors: Jacob Bridger; Belal Lehwany
Original assignee: Vpsign, Ltd.
Priority date: 2010-10-10
Filing date: 2011-10-05
Publication date: 2012-04-19
Also published as: WO2012049592A3

Abstract

Embodiments relate to methods and apparatus for facilitating the protection from tampering of an electronic document to which an electronic signature is applied. In non- limiting examples, techniques may relate to the handling of document appearance data, dynamic signature biometric data, digital footprints data, pixel history data, and camera- acquired image data.

Description

ELECTRONIC SIGNATURE APPARATUS AND METHOD BACKGROUND AND RELATED ART

Wikipedia defines a signature as "a stylized script associated with a person." This definition refers to the specific case of a 'handwriting signature. '

Wikipedia defines an "electronic signature" as any legally recognized electronic means that indicates that a person adopts the contents of an electronic message. The U.S. Code defines an electronic signature as "an electronic sound, symbol, or process, attached to or logically associated with a contract or other record and executed or adopted by a person with the intent to sign the record."

The Canadian Personal Information Protection and Electronic Documents Act

(PIPED A ) defines an electronic signature as "a signature that consists of one or more letters, characters, numbers or other symbols in digital form incorporated in, attached to or associated with an electronic document." According to PIPEDA, a secure electronic signature is as an electronic signature that:

(a) is unique to the person making the signature;

(b) the technology or process used to make the signature is under the sole control of the person making the signature;

(c) the technology or process can be used to identify the person using the technology or process; and

(d) the electronic signature can be linked with an electronic document in such a way that it can be used to determine whether the electronic document has been changed since the electronic signature was incorporated in, attached to or associated with the electronic document.

FIG. 1 (PRIOR ART) is a flowchart of routines for signing documents with a digital signature (left hand side) and for verifying the authenticity of the digital signature according to one prior art implementation. FIG. 1 has been copied (with slight modifications) from the Wikipedia article entitled "Digital Signatures." As is shown on the left hand side of FIG. 1, in order to digitally sign data (also referred to as a 'message' in this background section), it is possible to first compute a message digest of a particular message (for example, by computing a hash function of the data/message. Typically, the size of the message digest (for example, the 'hashed data') is much smaller than the size of the original data. The message digest may then be encrypted using a verifiable, certified private key to 'sign the message.' As illustrated in FIG. 1, this private-key- encrypted message digest may be attached to the original data to verify the data's authenticity.

As shown on the right hand side of FIG. 1, when the data is verified at a later time, it is possible to: (i) compute the message digest or hash function of the original data; and (ii) to decrypt the encrypted message digest/hash using the published public key that corresponds to the private key earlier used to encrypt the message digest/hash.

If the computed digest of 'candidate data' (i.e. data whose authenticity is being examined) corresponds to the public-key-decrypted message digest of the 'original data' (see the left hand side of FIG. 1), this indicates that the candidate data is 'authentic' inasmuch as it is identical to the original data.

The aforementioned Wikipedia article entitled "Digital Signatures" notes that it is known to store private key data (i.e. which may be used on the 'left hand side' of FIG. 1 to encrypt the message digest) on a tamper-resistant smart card. When the smartcard is read by a smartcard reader, data processing digital circuitry (for example, ASIC circuitry or CPU(s)) of the smart card may encrypt the message digest of the data to be signed using the stored private key that is stored within the smart card. This message digest may be attached to the 'digitally-signed data.

At a later time, it is possible to (i) decrypt the encrypted message digest attached to the data using a published public key corresponding to the secret private key of the smart card; (ii) compare the decrypted message digest with the original data. If the decrypted message digest matches the original data, this may indicate that a person in possession of the smart card (often but not necessarily the 'legitimate owner') agreed to 'digitally sign' the data.

According to the article "Digital Signature" of Wikipedia,

Digital signature schemes share basic prerequisites that— regardless of cryptographic theory or legal provision— they need to have meaning:

1 . Quality algorithms Some public-key algorithms are known to be insecure, practicable attacks against them having been discovered.

2. Quality implementations

An implementation of a good algorithm (or protocol) with mistake(s) will not work.

3. The private key must remain private if it becomes known to any other party, that party can produce perfeci digital signatures of anything whatsoever.

4. The public key owner must be verifiable

A public key associated with Bob actually came from Bob. This is commonly done using a public key lnfrassruciu;e and the public key<~»user association is attested by the operator of the PKI (called a certificate authority). For 'open' PKIs in which anyone can request such an attestation (universally embodied in a cryptographically protected identify certificate), the possibility of mistaken attestation is non trivial. Commercial PKI operators have suffered several publicly known problems. Such mistakes could lead to falsely signed, and thus wrongly attributed, documents, 'closed' PKI systems are more expensive, but less easily subverted in this way.

5. Users (and their software) must carry out the signature protocol properly.

Wikipedia defines a "cryptographic hash function" as a "deterministic procedure that takes an arbitrary block of data and returns a fixed-size bit string, the (cryptographic) hash value, such that an accidental or intentional change to the data will change the hash value. The data to be encoded is often called the 'message,' and the hash value is sometimes called the message digest or simply digest."

The hash function is one example of a One-way function.' When data is subjected to a one-way function (e.g. a hash function) to obtain a 'message digest,' it is infeasible to modify the original data in such as way so that the digest value does not change.

Digital signature devices are described in various patent documents - for example US 2009/0031132 of one of the present inventors, incorporated herein by reference in its entirety.

There is an ongoing need for methods and apparatus which facilitate the acquisition and handling of secure electronic signatures. The following US patents and patent publications provide potentially relevant background material, and are all incorporated by reference in their entirety: US 6,694,045, US 6,459,424, US 7,481,363.

SUMMARY OF EMBOIDMENTS

A method of facilitating the protection from tampering of a handwriting-signed electronic document that is generated when an electronic handwriting signature is applied to an electronic document displayed on a display screen of a document display device is now disclosed. The method comprises the steps of: a) as an object is moved over the display screen to apply the electronic handwriting signature to the electronic document, acquiring dynamic signature biometric data of the applied electronic handwriting signature; b) acquiring handwriting-signed-document visual appearance data describing a static post-signature appearance of the handwriting-signed electronic document; c) effecting a private key encryption of the signed-document visual appearance data or a message digest thereof; and d) effecting a public key encryption of the dynamic signature biometric data of the user-applied handwriting electronic signature or a message digest thereof.

In some embodiments, the acquiring of the handwriting-signed-document visual appearance data includes at least one of: i) effecting a correlation between the dynamic signature biometric data and visual content of the electronic document according to offset data; and ii) interpolating signature points derived from the dynamic signature biometric data

In some embodiments, the public key encryption is applied to a hybrid data object or a message digest thereof, the hybrid data object comprising dynamic signature biometric data and handwriting-signed-document visual appearance data.

It is that the 'message digest thereof may refer to either a message digest of the entire hybrid data or to some combination (e.g. a concatenation or any other combination) of message digests of 'constitutive' or 'component data' of the hybrid data.

In some embodiments, the hybrid data object further comprises digital footprint data.

In some embodiments, the method further comprises the step of: e) causing the private- key-encrypted handwriting-signed document visual appearance data or a message digest thereof and the public-key-encrypted dynamic signature biometric data of a message digest thereof to co-reside in a single container data object.

In some embodiments, the container data object is an image object including both a viewable image and metadata encapsulated within the image object, and wherein both the private-key-encrypted visual appearance data or message digest thereof as well as the public-key-encrypted dynamic signature biometric data or message digest thereof are embedded within the image object as metadata.

In some embodiments, the container data object is a file.

In some embodiments, the file is selected from the group consisting of a single page .tiff file, a multi-page .tiff file, a single page .pdf file, a multi-page .pdf file, an .xml file and a .zip file.

In some embodiments, step (e) is carried out within the document display device. In some embodiments, step (c) and/or step (d) is carried out within the document display device.

In some embodiments, the private key encryption of step (c) employs at least one private key selected from the group consisting of: i) a private key that is specific to the document display device to which the user applies a signature; and ii) an external private key that is external to the user appliance.

In some embodiments, the public key encryption of step (d) is carried out to a message digest of handwriting-signed-document visual appearance data or to hybrid data thereof.

In some embodiments, the public key encryption is carried out to a hybrid data object including both the dynamic signature biometric data and at least one of (i.e. any combination - i.e. including combinations explicitly described and combinations not explicitly described): i) time-stamp data (e.g. in any format - for example, day/hour/second/millisecond format or in terms of CPU counter tick data or in any other manner); ii) unique device indication data (e.g. tamper resistant data) iii) document appearance data (e.g. word count, number of pages ; a subset of text); iv) identifying data describing the user who applies the electronic signature (e.g. social security number or credit card number or phone number); v) identifying data describing a customer service agent (for example, where a teller is a customer service agent and has an identifier); vi) branch information; vii) location information describing a location where the document is signed.

In some embodiments, the object moved over the displayed screen is a finger or a stylus/electronic pen or any other object.

In some embodiments, i) the private key encryption is carried out to a first hybrid data object or message digest thereof, the hybrid data object including first time stamp data; and ii) the public key encryption is carried out to a second hybrid data object or message digest thereof, the hybrid data object including second time stamp data which matches the first time stamp data.

Apparatus for facilitating the protection from tampering of an electronic document is now disclosed. The apparatus comprising: a) electronic circuitry including memory for storing the electronic document; and b) a display screen configured to display information responsive to signals received from the electronic circuitry, the display screen configured to display the electronic document as modified by an electronic signature in response to moving an object over the display screen, wherein the electronic circuitry is configured to: i) acquire or handle dynamic signature biometric data of the applied electronic handwriting signature; ii) acquire or handle handwriting-signed- document visual appearance data describing a static post-signature appearance of the handwriting-signed electronic document; iii) effect a private key encryption of the signed-document visual appearance data or a message digest thereof; and iv) effect a public key encryption of the dynamic signature biometric data of the user-applied handwriting electronic signature or a message digest thereof.

In some embodiments, the electronic circuitry includes any combination of hardware, software and firmware.

In some embodiments, the display screen is configured as a touch screen.

In some embodiments, the apparatus further comprises a digital pen in communication with the electronic circuitry.

In some embodiments, the apparatus is configured to effect any method disclosed herein.

A method of operating a document display device including a display screen to facilitate the acquisition of an electronic handwriting signature of an electronic document displayed on the electronic appliance is now disclosed. The method comprises: a) monitoring (i.e. in any manner - i.e. explicit or implicit) content displayed on a display screen of the electronic appliance along with display times to acquire digital footprints data describing a sequence of display states of the electronic document correlated with respective display times; and b) acquiring at least one type of digital signature data describing a digital signature applied to the electronic document using the electronic appliance at a time after at least some of the display states (i.e. but within the same 'session'), the digital signature data selected from the group consisting of: i) handwriting-signed-document visual appearance data describing a static post-signature appearance of an electronic-handwriting-signed electronic document; and ii) smartcard PKI data describing the PKI encryption of the electronic document or a message digest thereof; c) subjecting the digital footprints data or a message digest thereof to a PKI encryption, wherein the digital signature data is time-stamped data having a time stamp that matches one or more of the display times of the digital footprints data and/or the PKI encryption of step (c) is carried out to hybrid data comprising both the digital signature data and the digital footprints data or any combination of message digests thereof and/or both the PKI-encrypted digital footprints data and the encrypted or unencrypted digital signature data or any combination of message digests thereof are embedded into a single container data object.

In some embodiments, the monitoring of the display content is carried out according to at least one of: i) an image of the display screen acquired by an observer electronic camera; ii) logged browsing commands; and iii) an internal data structure of the document display device.

In some embodiments, the logged browsing commands are applied to one or more of: i) the document display device; and ii) a document monitor device in communication with the document display device.

In some embodiments, the digital display data describes both document display device browsing commands as well as the document monitor browsing commands.

Apparatus for facilitating the protection from tampering of an electronic document is now disclosed. The apparatus comprises: a) electronic circuitry including memory for storing the electronic document; and b) a display screen configured to display information responsive to signals received from the electronic circuitry, the electronic circuitry being configured to: i) acquire or handle digital footprints data describing a sequence of display states of the electronic document correlated with respective display times; ii) acquire or handle at least one type of digital signature data describing a digital signature applied to the electronic document using the electronic appliance at a time after at least some of the display states (i.e. but within the same 'session'), the digital signature data selected from the group consisting of: A) handwriting-signed-document visual appearance data describing a static post-signature appearance of an electronic-handwriting-signed electronic document; and B) smartcard PKI data describing the PKI encryption of the electronic document or a message digest thereof; c) subjecting the digital footprints data or a message digest thereof to a PKI encryption, wherein the digital signature data is time-stamped data having a time stamp that matches one or more of the display times of the digital footprints data and/or the digital footprints PKI encryption of is carried out to hybrid data comprising both the digital signature data and the digital footprints data or any combination of message digests thereof and/or both the PKI-encrypted digital footprints data and the encrypted or unencrypted digital signature data or any combination of message digests thereof are embedded into a single container data object.

The electronic circuitry may include any combination of hardware (i.e. analogue or digital hardware), software and firmware.

In some embodiments, the display screen is configured as touch screen.

In some embodiments, the apparatus is pen-less and lacks a digital pen.

A method of facilitating the protection from tampering to a handwriting-signed electronic document that is generated when an electronic handwriting signature is applied to an electronic document displayed on a display screen of a document display device is now disclosed. In some embodiments, the method comprises: effecting PKI encryption(s) to both: i) later-time appearance data or a message digest thereof, the later-time appearance data describing a post-signature appearance electronic document; and ii) earlier-time appearance data or a message digest thereof, the earlier-time appearance data describing an appearance of the electronic document at an earlier time before the electronic handwriting signature has been completely applied to the electronic document.

In some embodiments, steps (i) and (ii) are carried out by a single PKI encryption to hybrid data comprising any combination of later-time appearance data and earlier-time appearance data or one or more message digest(s) thereof.

In some embodiments, steps (i) and (ii) are carried out so that PKI encryptions of the later-time appearance data or a message digest thereof and the time appearance data or a message digest thereof are respectively performed as separated PKI encryptions.

In some embodiments, the method is carried out so that both the sealed later-time appearance data and the sealed earlier-time appearance data co-reside in a single data object.

In some embodiments, the earlier-time appearance data describes a partially- signed state of the document when only a partial handwriting signature appears in the document.

In some embodiments, the earlier-time appearance data describes an unsigned state of the document when no handwriting signature appears in the document.

In some embodiments, i) first and second electronic handwriting signatures of first and second person are respectively and chronologically applied to the electronic document; and ii) the term post-signature state applies specifically to the second electronic handwriting signature such that: A) the earlier-time appearance data describes an appearance of the document after the first electronic signature has been applied to the document but before the second electronic signature has been applied to the document; and B) the later-time appearance data describes an appearance of the document after both the first and second electronic signatures have been applied to the document.

Apparatus for facilitating the protection from tampering of an electronic document is now disclosed. The apparatus comprises: a) electronic circuitry including memory for storing an electronic document; and b) a display screen configured to display information responsive to signals received from the electronic circuitry, the display screen being configured to display the electronic document as modified by an electronic signature in response to moving an object over the display screen, wherein the electronic circuitry is configured to effect PKI encryption(s) to both: i) later-time appearance data or a message digest thereof, the later-time appearance data describing a post-signature appearance electronic document; and ii) earlier-time appearance data or a message digest thereof, the earlier-time appearance data describing an appearance of the electronic document at an earlier time before the electronic handwriting signature has been completely applied to the electronic document.

A method of facilitating the protection from tampering of a handwriting-signed electronic document that is generated when an electronic handwriting signature is applied to an electronic document displayed on a display screen of a document display device is now disclosed. The method comprises: a) acquiring at least one type of digital signature data describing a digital signature applied to the electronic document using the electronic appliance, the digital signature data selected from the group consisting of: i) handwriting- signed-document visual appearance data describing a static post-signature appearance of an electronic -handwriting-signed electronic document; and ii) smartcard PKI data describing the PKI encryption of the electronic document or a message digest thereof; b) obtaining digital image data by causing a digital camera to acquire at least one digital camera image selected from the group consisting of: i) an image of a scene including the user; and ii) an image of a scene including a visual credential of the user; c) subjecting the digital image data or a message digest thereof to a PKI encryption, wherein the digital image data is time-stamped data having a time stamp that matches a time stamp of the digital signature data and/or the PKI encryption of step (c) is carried out to hybrid data comprising both the digital signature data and the digital image data or any combination of message digests thereof and/or both the PKI-encrypted digital image data and the encrypted or unencrypted digital signature data or any combination of message digests thereof are embedded into a single container data object.

Apparatus for facilitating the protection from tampering of an electronic document is now disclosed. The apparatus comprises: a) electronic circuitry including memory for storing the electronic document; and b) a display screen configured to display information responsive to signals received from the electronic circuitry, the electronic circuitry being configured to: i) acquire or handle at least one type of digital signature data describing a digital signature applied to the electronic document using the electronic appliance at a time after at least some of the display states (i.e. but within the same session), the digital signature data selected from the group consisting of: A) handwriting-signed-document visual appearance data describing a static post-signature appearance of an electronic-handwriting-signed electronic document; and B) smartcard PKI data describing the PKI encryption of the electronic document or a message digest thereof; ii) acquire or handle digital image data by causing a digital camera to acquire at least one digital camera image selected from the group consisting of: i) an image of a scene including the user; and ii) an image of a scene including a visual credential of the user; c) subjecting the digital image data or a message digest thereof to a PKI encryption, wherein the digital image data is time-stamped data having a time stamp that matches a time stamp of the digital signature data and/or the PKI encryption of step (c) is carried out to hybrid data comprising both the digital signature data and the digital image data or any combination of message digests thereof and/or both the PKI-encrypted digital image data and the encrypted or unencrypted digital signature data or any combination of message digests thereof are embedded into a single container data object.

A method of operating a document display device including a display screen to facilitate the acquisition of an electronic handwriting signature or of a smartcard PKI signature of an electronic document displayed on the electronic appliance is now disclosed. The method comprises: a) for each location within the electronic document of a plurality of locations, making the location available for respective acquisition of a respective digital signature selected from the group consisting of a respective smartcard PKI signature and a respective handwriting signature; and b) in the event that handwriting signatures are acquired in step (a), subjecting visual appearance data or a message digest thereof to a PKI encryption, the visual appearance data describing the document including all handwriting signature(s) of step (a) wherein the method is carried out such that at least one condition selected from the group consisting of a first condition, a second condition and a third condition is true, the first, second and third conditions being defined as follows: (i) according to the first condition, for at least one of the locations, a respective signature acquisition or PKI encoding is contingent upon previous signature acquisitions in other locations of the plurality of locations within the electronic document; (ii) according to the second condition, soft user controls for completing the signature process made available in a manner that is contingent upon signature acquisition for all locations of the plurality of locations; (iii) according to the third condition, the user is only provided with a visual or audio indication that the document is sealed if signatures have been acquired at all locations of the plurality of locations; and (iv) according to the fourth condition, a mode transition from a first mode to a second mode is carried out in a manner that is contingent acquisition of digital signatures for all locations of the plurality of locations, such that in the first mode the electronic document is available for signature and a in the second mode a different electronic document is available for signature.

A method of operating a document display device including a display screen to facilitate the acquisition of an electronic handwriting signature or of a smartcard PKI signature of an electronic document displayed on the document display device, the method comprising: a) monitoring content displayed on a display screen of the document display device along with display times to acquire digital footprints data describing a sequence of display states of the electronic document correlated with respective display times; b) for a set of points in time comprising one or more points in time, for each time point of the time point set, respectively analyzing the digital footprints data to respectively determine if one or more non-minimal historical browsing operations(s) that are not minimal for the respective instantaneous display state of the point in time have been carried out, thereby respectively effecting a positive or negative determination for the time point; c) acquiring at least one type of digital signature data describing a digital signature applied to the electronic document using the document display device at a time after at least some of the display states, the digital signature data selected from the group consisting of: i) handwriting-signed-document visual appearance data describing a static post-signature appearance of an electronic -handwriting-signed electronic document; and ii) smartcard PKI data describing the PKI encryption of the electronic document or a message digest thereof; d) in the event that handwriting signatures are acquired in step (b), subjecting visual appearance data or a message digest thereof to a PKI encryption, the visual appearance data describing the document including all handwriting signature(s) of step (a), wherein the method is carried out such that at least one signature operation selected from a signature operation set is contingent upon a positive determination being made in step (b), the signature operation set consisting of the following operations: (i) one or more signature acquisitions of step (c) (ii) providing soft user controls display on the display screen document display device for completing the signature process; (iii) effecting a mode transition from a first mode to a second mode where in the first mode the electronic document is available for signature and in the second mode a different electronic document is available for signature.

DETAILED DESCRIPTION OF EMBODIMENTS

The claims below will be better understood by referring to the present detailed description of example embodiments with reference to the figures. The description, embodiments and figures are not to be taken as limiting the scope of the claims. It should be understood that not every feature of the presently disclosed methods and apparatuses is necessary in every implementation. It should also be understood that throughout this disclosure, where a process or method is shown or described, the steps of the method may be performed in any order or simultaneously, unless it is clear from the context that one step depends on another being performed first. As used throughout this application, the word "may" is used in a permissive sense (i.e., meaning "having the potential to'), rather than the mandatory sense (i.e. meaning "must").

It is appreciated that certain features of the invention, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the invention, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable sub-combination.

Embodiments of the present invention relate to apparatus and methods for acquiring and handling electronic signatures and related data. One example of an electronic signature is a handwriting electronic signature (see FIG. 2). Another example of an electronic signature is a smartcard PKI electronic signature (see FIG. 3).

FIGS. 2A-2B illustrate a signature acquisition device 10 for capturing electronic signatures according to some embodiments. In both FIG. 2A and FIG. 2B, the document (or the portion of the electronic document being displayed) includes four lines of text and a signature box. In FIG. 2A, the portion of the document requires an electronic handwriting signature, and this portion of the electronic document is in a pre-signature state. In FIG. 2B, the required electronic handwriting signature has been applied to the document, and this portion of the electronic document is in a post-signature state.

The example signature acquisition device illustrated in FIGS. 2A-2B happens to have a 'tablet form factor' (though this is not a limitation) and includes both a stylus (or electronic pen 18) and a display screen 16. One salient feature of the device 10 of FIGS. 2A-2B is the "What You See Is What You Sign" (or WYSIWYS) functionality. Signature acquisition device 10 displays a 'target electronic document' (or portion of the document) to be signed on display screen 16, and the user moves stylus/electronic pen 18 on or over display screen 18 to electronically sign his/her name "on the actual document" thereby virtually 'modifying' the visual appearance of the electronic document. This is similar to a traditional ink-and-paper signature where the user's ink signature on the document modifies the visual appearance of the document not in order to modify the content of the document per se but rather in order to show agreement with the document.

When the user signs the document (for example, by moving stylus 18 over the surface of screen 16), his/her electronic handwriting signature is captured and the 'post- signature state' of the document (see FIG. 2B) is displayed on screen 16. Furthermore, 'handwriting-signed-document visual appearance data' describing the 'post-signature state' (and/or a message digest thereof) of the document (for example, pixel data) may be subjected to a PKI encryption (typically a private key encryption) to obtain 'encryption data.'

In some embodiments, this 'encryption data' is physically or logically associated with the digital image (in one non-limiting example, a 'graphic image file' such as a single page or multipage tiff file or a single or multi-page pdf file or any other graphic image file) that describes the visual appearance of the handwriting-signed electronic document may be generated. In one example, the aforementioned 'digital image' describing the appearance of the signed document is a 'pure graphics file.' In another example, the 'digital image' may be a combination of text (for example, searchable text) and graphics - thus, the 'digital image' is at least partially graphical and describes the visual appearance of the handwriting-signed electronic document.

In many jurisdictions, this 'encryption data' may be presented as evidence that the digital image object (e.g. tiff file or pdf file or other digital image) of the static post- signature appearance of the signed electronic document (i.e. how the electronic document appears after the user signature - see, for example, what is displayed on display screen 16 in FIG. 2B) is genuine and has not been tampered with.

Furthermore, some embodiments of the present invention relate to one or more of the following features (i.e. any combination of the features may be provided, including combinations explicitly listed or any other combination):

(i) a first set of 'multi-tiered encryption' feature(s) where 'dynamic signature biometric data' of the electronic handwriting signature that is applied to the electronic document is encrypted by a public key rather than a private key. The 'static data' describing visual post-signature appearance of the handwriting-signed electronic document is encrypted by a private key. Certain aspects of this first set of features are discussed below with reference to FIGS. 4A, 6A-6B.

(ii) a second set of feature(s) referred to informally as the capturing of 'Digital Footprints." Certain aspects of the second set of feature(s) are discussed below with reference to FIGS. 4A, 7-8.

In one non-limiting example, signature device 10 maintains a real-time record of what portions of an electronic contract document (for example, a rental contract or a mortgage contract or a service contract such as a cell phone service contract) or other electronic document to be signed (e.g. a government form or any other document)) are actually viewed by the consumer browsing and/or how the document is viewed. For example, a record may be kept of the order in which content is presented on the screen 16 of device 10, or the 'zoom factor' applied for viewing any object of the electronic document such as 'fine print.' These 'digital footprints' may be electronically associated with the signed-document visual appearance data describing the static post-signature visual appearance of the document (for example, a graphic image file) and/or with 'authenticating' data verifying the authenticity of the static post-signature visual appearance of the document (e.g. a PKI encryption of the signed-document visual appearance data or of a message digest thereof).

As will be discussed below, in some embodiments, one or more of the feature(s) of the second set of feature(s) are implemented in the context of an electronic handwriting signature (for example, see FIGS. 2A-2B). In other embodiments, one or more of the feature(s) of the second set of feature(s) are implemented in the context of a 'smartcard PKI signature.' (discussed below - see for example, FIG. 3)

(iii) a third set of feature(s) whereby a 'pixel history' of a 'target document' (i.e. to which an electronic handwriting is applied by the user) is tracked. Certain aspects of the third set of feature(s) are discussed below with reference to FIGS. 2A-2B, 4, and 9.

Thus, in some embodiments, snapshots at different points in time of the static visual appearance of the electronic document that has been displayed on screen 16 of signature-handling device 10 are acquired.

This visual appearance data (i.e. of the document as displayed on screen 16 at given point in time) of the document changes as the user applies his/her electronic handwriting signature. Nevertheless, the 'visual appearance data' is often referred to as 'static visual appearance data' because it relates to the appearance of the document rather than the 'dynamic' application of a signature.

In one example, this pixel history includes a 'pre-signature state' (for example, see FIG. 2A) as well as a 'post-signature state' (for example, see FIG. 2B). In another example, this pixel history may include 'intermediate states' when an electronic handwriting signature has been partially applied (for example, where the word 'John" has been signed but "Hancock" has not yet been signed).

Each 'snapshot' (i.e. static visual appearance describing how the document appears and displayed at a moment in time) or a message digest thereof is subjected to a PKI encryption - optionally and preferably with a time stamp describing a 'snapshot time.'

(iv) a fourth set of feature(s) whereby an electronic camera (e.g. a digital camera) acquires an image of a scene in which the customer/user is signing the electronic document and/or an image of a traditional visual credential (e.g. an identification document such as a passport or a drivers license) presented by the user upon signing. Certain aspects of the fourth set of feature(s) are discussed below with reference to FIGS. 4A, 9-10.

In some embodiments, the camera-acquired image data (or a message digest thereof), and preferably a time stamp of the time of image acquisition by the camera, is subjected to a PKI encryption using a public or private key. In one embodiment, the camera-acquired image data of the signing user (or of his/her visual credential) may be specifically subjected to a public key encryption

Similar to the second set of features, the fourth set of feature(s) may be implemented in the context of an electronic handwriting signatures and/or in the context of a 'smartcard PKI signature.';

(v) a fifth set of features whereby the ability of the user to complete the process of signing a document is contingent upon one or more conditions and/or detected events. The conditions/events may relate to the presence or application of electronic signatures to certain signature field. Alternatively or additionally, they may relate to the user's browsing activity and/or the 'history' of displayed content on display screen 16 - e.g. the use must view certain content (or view it in a certain manner) of the electronic document to 'cleanly' complete the signing process.

Certain aspects of the fifth set of feature(s) are discussed below with reference to FIGS. 12A-12B.

One non-limiting example/use case relates to embodiments where the electronic document includes a plurality of signature fields and the one or more (i.e. any combination of) 'document electronic signature completion operation(s)' is only carried out if the user has applied an electronic signature to all field and/or applied an electronic signature to the fields in a specific order and/or according to a specific timing scheme. According to this example/use case, an electronic contract requires three user signatures - e.g. the user/signer must sign the body of the contract, the user must signed 'Annex A,' 'Annex B' and 'Annex C In this example, in order for one or more 'document electronic signature completion operation(s)' to be carried out (i.e. for the user to 'cleanly' complete the signature process and/or to be notified that the document has successfully been sealed), all three electronic signatures would need to be present. In one variation of this example, it is not enough to supply all three signatures - rather, the electronic signature (i.e. handwriting signature or smartcard PKI signature) to the body section must be applied before the signature to any appendix). In yet another variation, all three signatures must be applied within a five minute 'time window' in order for the one or more 'document electronic signature completion operation(s)' to be carried out.

Examples of 'document electronic signature completion operation(s)' include but are not limited to electronically sealing the signed document, visually signaling to the user (or providing an audio signal) that the signature process is complete, providing to the user a printed or electronic receipt, providing user interface controls (for example, 'soft buttons') that allow the user to manually complete the signature process, or making a 'mode transition' from a first mode where digital signatures are applied to a first electronic document (i.e. the 'current document') and a second mode where digital signatures are applied to a second electronic document (i.e. the 'next document'). As far as the user of the signature acquisition device to which signatures are applied is concerned, this transition be 'informative' where the user is explicitly informed of the transition or can be 'silent' where the transition is carried out without providing any explicit signal to the user

In one use case relating to 'browsing operations' and/or requirement that certain content is displayed to cleanly' complete the signature process, an electronic contract has five pages, only two of which require an electronic signature (i.e. handwriting signature or smartcard PKI signature) - the other three pages are defined as 'non-signature pages.' It is possible to track the user's browsing patterns and only if the user has viewed one or some or all the 'non-signature pages' on screen 16 will the combination of one or more 'document electronic signature completion operation(s)' be carried out. In another use case, the user must 'zoom in' on 'fine print' for the combination of one or more 'document electronic signature completion operation(s)' to be carried out.

Similar to the second and fourth set of features, the fifth set of feature(s) may be implemented in the context of an electronic handwriting signatures and/or in the context of a 'smartcard PKI signature.';

A Discussion FIG. 3 FIGS. 2A-2B related to the specific case where an electronic handwriting signature is applied to an electronic document displayed on document display device 10. In another example, the user is in possession of a portable electronic device referred to as a PKI smartcard 94 on which a tamper-resistant private key is stored in volatile or non-volatile (preferred) memory (for example, flash memory). A data representation of the electronic document (or a message digest thereof) is encrypted (for example, using electronic circuitry residing within smartcard 94) using the private key data stored on the smartcard. In many jurisdictions, this encrypted data serves as legal evidence that the holder of the smartcard has agreed to the contents of the electronic document.

For the present disclosure, the term 'smartcard' (or electronic 'token') is used generically to relate to any electronic device housing private-key data stored in a tamper- resistant manner on computer memory/storage (including but not limited to flash memory and magnetic storage) so that electronic circuitry (i.e. any combination of hardware and executable code) can encrypt the electronic document (or a message digest thereof) using the private key - this electronic circuitry for utilizing the tamper-resistant PKI data may reside within the smartcard 94 itself or on an optional smartcard reader 96 or in any other location. In the present disclosure, such an electronic device is referred to as a 'smartcard' only because this term is often used in the art of electronic signatures. If the electronic circuitry resides outside of smartcard 94 itself, there may be some sort of secure interface or protocol so that only an authorized device/application may access the private key data (i.e. for the purpose of encrypting the electronic document or a message digest thereof).

It is noted that the term 'smartcard' is an extremely common term used in a number of contexts. In order to avoid ambiguity, it is noted that there is no additional requirement of a smartcard 94 or 'smartcard device' - for example, there is no need for the 'smartcard' to provide any mechanical or interface properties often associated with 'smartcards' - i.e. form factors, smartcard interfaces and the like.

In some embodiments, it is possible to utilize the smartcard 94 to apply to a specific location within the electronic document (e.g. a signature box or line), as was the case for handwriting electronic signatures. For example, a specific location may be marked (e.g. by a color code or flashing characters or any other way to indicate an 'active' signature field) as displayed on screen 16 - when the user applies his/her smartcard signature, not only is the actual document/portion of the document (or a message digest thereof) encoded by the private key data residing on smartcard 94, but some indication of the 'signature field' or 'signature location' within the document is also encoded. For example, if a document includes N signature fields (where N is a whole number), an 'identification number' of the specific signature field (i.e. the 'active' field indicated on screen 16) may also be encoded.

In this manner, it is possible for the user to apply multiple electronic signatures to a single document (each signature to a different location), in a manner similar to how a user signs a paper/ink contract or how a user applies multiple signatures to an electronic document. In order to view different positions in the document, document display device 10 may include 'soft' browse controls 23 (e.g. using touch-screen technology) or 'hard' browse controls (e.g. depressible buttons). \

In FIG. 3, smartcard device 94 is illustrated as a separate device that is separate from document display device 10 - in some embodiments, these two devices do indeed have separate housing and are separate devices, and may be in wired or wireless communication. In other embodiments (not illustrated), smartcard 94 is provided as a part of document display device 10 (i.e. as a 'combined' device that includes both smartcard functionality and document display functionality).

A Discussion FIG. 4A

As noted earlier (see the discussion with reference to FIGS. 2A-2B), when an electronic image of the document in its 'post-signature state' (i.e. after the user applies the handwriting signature) or a message digest thereof is PKI-encrypted (for example, private-key encrypted), this provides evidence of authenticity of the handwriting-signed electronic document. The result of the PKI-encryption is labeled in FIG. 4A as element 640.

One salient feature provided by some embodiments is that this private-key- encrypted signed document appearance data (or the encrypted message digest) 640 co- resides in the same data object 600 as one or more of the following:: (i) public-key- encrypted dynamic signature biometric data 644 (see the above discussion of the 'first set of features and the discussion below with reference to FIG. 6); and/or (ii) PKI-encrypted digital footprints 648 (see the above discussion of the 'second set of features and the discussion below with reference to FIGS. 7-8); and/or (iii) PKI-encrypted historical document appearance data 652 describing how the document appears at an earlier time that is before the time of the 'post-signature state' (i.e. before the user applies an electronic handwriting signature or when such a signature has only been partially applied) (see the above discussion of the 'third set of features and the discussion below with reference to FIGS. 2 and 9); and/or (iv) PKI-encrypted camera-acquired image data 656 (see the above discussion of the 'fourth set of features' and the discussion below with reference to FIGS. 10-11).

Although certainly not a requirement, in some embodiments the 'containing data object' 600 is a file and/or an image data object such as an image file - for example, a pdf file or tiff file including image data 620 (or any other image data object ) of the actual electronic document (for example, the electronic-handwriting-signed electronic document or the 'presented electronic document' that was presented on display screen 16). In one particular example, the file may include 'meta-data' and one or more of data (i.e. any combination) 640-656 is embedded in the file/data object as metadata.

In another example, the file may be a zip file or any other type of file. The zip file may include any combination of data 640-656 and/or the image 620.

It is noted that some embodiments provide the generation and/or distribution of a single data object that includes both the 'less private' data describing the user's signature applied to the electronic document as well as the 'more private data' describing for example a user's dynamic biometric handwriting parameters or a user's appearance at a certain point in time.

The encapsulating data object 600 may be generated in any location. In one preferred embodiment, the encapsulating data object 600 is generated on the signature acquisition device 10.

In some embodiments, encapsulating data object 600 may include some sort of directory or index of the various data contained within - for example, a list of metadata location offsets of a .tiff file. Although not a limitation, in one preferred embodiment,, a message digest of a 'hybrid data object' comprising the visual appearance data and the digital footprint data may be computed. This message digest may be subjected to a PKI encryption to obtain 648. This is true for any of the 'auxiliary data objects" objects 644, 648, 652 and/or 656.

It is noted that the examples of 'pre-signature state' and 'post-signature states' in FIGS. 2A-2B relate to the situations where a single electronic signature is applied to a single electronic document. In some embodiments, a plurality of signatures are sequentially applied - for example, first a signature is applied in "Location A" (SIGNATURE A); later, a signature is applied to "Location B" (SIGNATURE B); later, a signature is applied to "Location C" (SIGNATURE C).

In this case, at the point in time after SIGNATURE A is applied but before SIGNATURE B is applied, the document is in a 'pre-signature state' relative to SIGNATURE B but in a 'post-signature state' relative to SIGNATURE A. In some embodiments, the signatures are all applied by the same user. In other embodiments, it is possible for two different users/signers to sign - for example, the husband may applied SIGNATURE A and his wife may 'counter-sign' the document to apply SIGNATURE B. A Discussion of 'Static visual appearance data' and 'Dynamic Signature Biometric Data' Some embodiments relate to electronic handwriting signatures. When the user applies a handwriting signature to an electronic document (for example, as illustrated in FIGS. 2A-2B) the displayed appearance of the document (or portion thereof where the signature is applied) on display screen 16 changes to indicate the user's handwriting signature. Inspection of FIGS. 2A and 2B indicate that the appearance of the electronic document changes when the user applies his/her handwriting signature (this is not necessarily the case for PKI smartcard signatures as will be discussed below). The appearance of the document after the user applies the electronic signature is referred to as the 'visual post-signature appearance describing the static post-signature appearance of the handwriting-signed electronic document.'

As the user applies his/her handwriting signature, it is possible to track (i.e. in 'real time' ) one or more biometric parameters about how the applies the signature (for example, pen speed, pen direction, amount of pressure applied, tilt angle, acceleration, hovering (pen-up/pen-down) etc) - in the present disclosure, this referred to as the as 'dynamic biometric data.'

'Static visual appearance data' describes a snapshot in a static point in time of how the electronic document displayed on screen 16 appears - this may be at any time when the document is in any state - for example, a 'pre-signature state' before the handwriting signature is applied, a 'partial-signature state' when a portion of the handwriting signature has been applied, and a 'post-signature state' when the entirety or substantially the entirety of the handwriting signature has been applied.

For situations where more than one handwriting signature is applied (for example, multiple pages or multiple fields require a signature), a document may be in a 'post- signature state' relative to one signature and in a pre-signature state relative to another signature.

In Fig. 2B, it is possible that the user began to sign his name from the upper part of the letter "J" of "John Hancock" - the spatial relationship between locations within the electronic document and the where the user effected a signature operation (for example, a pen-down to begin the letter 'J') is referred to as a signature-document 'spatial offset' 9. In FIG. 2B, this spatial offset 9 is within the signature box. As is illustrated in FIG. 4B (see the lower arrow from right to left), it may be possible to derive 'static visual appearance data' of the document in a 'post-signature state' from the combination of (i) the 'offset data' 9 and (ii) the dynamic signature data. For example, if the trajectory data (e.g. velocity, acceleration) of stylus 18 is known it may be possible to utilize some sort of interpolation function (e.g. splines, Bezier functions or any other interpolation functions) to determine the geometric shape of the signature itself. The combination of the offset data 9 and the geometric shape of the signature is generally enough to derive the 'static visual appearance data' of the document in a 'post-signature state.'

In contrast (see the upper arrow in FIG. 2B from left to right), even forensic handwriting analysts may not be able to derive (or to derive with any sort of reasonable accuracy) certain dynamic signature biometric data' from the static appearance of the document alone. It is noted that this may also be true in the ink-and-paper world for the case of a photocopy, a scan or fax of an ink-and-paper signature where 'pressure imprints data' describing pressure imprints on the paper and the darkness of the ink are unavailable..'

For the present disclosure, the term 'non-derivable dynamic signature biometric data' refers to dynamic signature biometric data that cannot, according to techniques known in the art, be derived (or be derived with any sort of reasonable accuracy) from the static appearance of the user's signature (for example, cannot be derived from the visual appearance data of the document in the post-signature state).

In general, 'dynamic signature biometric data' includes 'non-derivable dynamic signature biometric data.'

In order to provide proof of authenticity of 'static visual appearance data' describing the visual appearance of the handwriting-signed document in a post-signature state (for example, in order to provide proof of authenticity of object 620 of FIG. 4A), it is possible to subject the 'static visual appearance data' (or a message digest thereof) to a PKI encryption (for example, using a private key) to obtain encrypted data (for example, see 640 of FIG. 4A).

As noted with reference to FIG. 4B, in some embodiments, it is possible to derive handwriting-signed-document visual appearance data from the combination of dynamic signature biometric data - thus, there may be no need to separately store handwriting- signed-document visual appearance data and it may only be necessary to store dynamic signature biometric data and offset data 9.

This is not a limitation, and in other embodiments, it may be possible to acquire handwriting-signed-document visual appearance data in another manner - for example, by determining only points-of-contact between stylus 18 and screen 16 or in any other manner.

As noted above, in many jurisdictions, this 'encryption data' 640 may be presented as evidence that the digital image object (e.g. tiff file or pdf file or other digital image) of the 'static visual appearance data' describing the post-signature appearance of the signed electronic document is genuine and has not been tampered with.

Some embodiments of the present invention relate to supplementing this encryption data 640 with additional biometric data (see 644 of FIG. 4A) describing the user's signature (i.e. to reduce the likelihood of a signature forgery). Although this additional biometric data might be sensitive, it still may be possible to distribute this 'more sensitive data' in a single encapsulating data object (e.g. 600 of FIG. 4A) where both the less sensitive static visual appearance data and the more sensitive dynamic signature biometric data co-reside because the biometric data (or a message digest therof) has been encrypted with the public key. Even though the more sensitive data 644 is more freely distributed, only holders of the private key may access its content.

As noted earlier, data object 644 may be a 'hybrid data object' (i.e. any combination of objects and/or message digests thereof) comprising the visual data and the biometric data that is subjected to a PKI encryption. For the specific case of object 644 (and possibly object 656), it is preferable that this PKI encryption is carried out using a public key.

Some embodiments of the present invention relate to supplementing this encryption data 640 with additional data (i) providing evidence that the user has indeed viewed pertinent sections of the document that s/he signed with an electronic handwriting signature or a smartcard PKI signature (see 648 of FIG. 4A; see the 'second set' of feature(s) discussed above and FIGS. 7-8) and/or (ii) providing evidence of what the user actually signed - for example, in the event that the user's signature blocks out content or obscures a portion of content of the original electronic document (see 652 of FIG. 4A; see the 'third set of features' discussed above and FIG. 9A) and/or (iii) providing evidence that the actual user who signed the document with an electronic handwriting signature or smartcard PKI signature was present in a certain location (or present having a certain visual appearance) and/or that the user presented some sort of visual credential such as passport or license (see 656 of FIG. 4 and the fourth set of feature FIGS. 9-10).

Any of 644 or 648 or 652 or 656 may be referred to as 'signature-supplementary data' which supplements the more conventional encrypted document appearance data 640 that merely describes the appearance of the viewed document or handwriting-signed document.

Definitions

For convenience, in the context of the description herein, various terms are presented here. Other terms may be described and/or defined at other locations in the present disclosure. For terms defined here or anywhere and to the extent that definitions are provided, explicitly or implicitly, here or elsewhere in this application, such definitions are understood to be consistent with the usage of the defined terms by those of skill in the pertinent art(s). Furthermore, such definitions are to be construed in the broadest possible sense consistent with such usage.

Subjecting data to a "PKI encryption" refers to a public key infrastructure cryptography operation(s). One example of subjecting data to a "PKI encryption" is where data is encrypted by a private key. Another example of subjecting data to a "PKI encryption" is where data is encrypted by a public key.

As will be discussed below, for the present disclosure, any PKI encryption of data can be carried out by effecting a PKI encryption of a message digest of data.

The private key can be any private key - for example, a unique private key of signature acquisition device 10 or a private key of a user's smartcard device or a private key of a service agent or a private key of document monitor device (e.g. used by a service agent to assist the user) 12 that electronically communicates with signature acquisition device 10 or any other private key.

Subjecting a data describing a visual appearance of an electronic document to a PKI encryption (for example, using a smartcard PKI key or by subjecting visual appearance data describing a static appearance of the document - for example, a post- signature appearance) is useful for protecting the electronic document (e.g. the signed document) from tampering. Embodiments of the present invention relate to 'facilitating the protection from tampering of a signed electronic document. Optionally but preferably, the visual appearance data is hybrid data including time stamp data (or any message digest combination thereof).

The term 'facilitating the protection' is not intended as limiting whatsoever - instead, the term relates to operations that may be useful to carry out in the context of acquisition of an electronic signature. Non-limiting examples of techniques which 'facilitate the protection' includedigital footprints techniques (see FIGS. 7-8, 12A) and 'signature wizard' methods (see FIG. 12B) are not per-se for protection from tampering but to rather to provide evidence that a document was reviewed, understood and executed properly. Additional examples are described herein. Operations that 'facilitate the protection' may, in some non-limiting embodiments, be useful for acquiring and/or obtaining and/or sealing data describing the circumstance in which the signature was applied. For example, (i) an appearance of the person applying the electronic signature (or his/her visual credentials) as in FIG. 10 (and element 656 of FIG. 4A) or the data display and/or (ii) the browsing circumstances of the display device and electronic document at a matching time that matches the 'signature application time' when the electronic signature is applied (see element 648). Subjecting a data describing a visual appearance of an electronic document to a PKI encryption (for example, using a smartcard PKI key or by subjecting visual appearance data describing a static appearance of the document - for example, a post-signature appearance) is useful for protecting the electronic document (e.g. the signed document) from tampering. Embodiments of the present invention relate to 'facilitating the protection from tampering of a signed electronic document. Optionally but preferably, the visual appearance data is hybrid data including time stamp data (or any message digest combination thereof).

The phrase 'facilitating the protection from tampering' is not intended as limiting whatsoever and does not require any steps that 'seal' the contents of the document. Instead, the phrase 'facilitating the protection from tampering' relates to operations that may be useful to carry out in the context of the acquisition of an electronic signature(s). For example, the operations that 'facilitate the protection' may, in some non-limiting embodiments, be useful for acquiring and/or obtaining and/or sealing data describing the circumstance in which the signature was applied. For example, (i) an appearance of the person applying the electronic signature or the visual appearance of his/her visual credentials (see FIG. 10 and element 656 of FIG. 4A) or (ii) the data display circumstance or browsing circumstances (see FIG. 8 and element 648 of FIG. 4A) and/or the (iii) dynamic signature circumstances when an electronic handwriting signature is applied (see FIG. 6 and element 644 of FIG. 4A) and/or pixel history data (see FIG 9 and element 652 of FIG. 4A).

The terms 'signature acquisition device' 10 and 'document display device' 10 are used interchangeably. The terms 'signature acquisition device' 10 and 'document display device' 10 relate to any device of any size, shape or form factor that includes a screen and electronic circuitry (for example, associated with each other by common device housing). The application of the electronic signature (i.e. electronic handwriting signature or PKI smartcard signature) to the electronic document is carried out at least in part using the signature acquisition device' 10 or 'document display device' 10.

In a first example, it is possible to track in any possible way movement of an object (e.g. a stylus/pen or finger or any other object) over the screen 16 to acquire a handwriting signature. In a second example, it is possible to effect a data exchange operation between signature acquisition device and a smartcard device - for example, via a wired or wireless interface. For example, electronic document data describing content of the electronic data is sent from signature acquisition device to the smartcard device which returns PKI encrypted electronic document data (the actual data or a message digest thereof) to the signature acquisition device. In both examples, the signature is document display device 10 plays a role in acquiring the digital signature - in the first example, the signature is applied to the screen 16 of the document display device 10 while in the second example, the signature display display device 10 sends to PKI smartcard (either an 'internal' that is part of device 10 or an 'external' smartcard) data describing document content displayed on the screen 16.

'Encrypting data' may relate to encrypting the actual data (i.e. document visual appearance data, biometric data, digital footprints data, pixel history data, or any other data - the data may be in any form - 'pure' data or hybrid data) or encrypting a message digest thereof. The definition of 'message digest' for the present disclosure is the one used in the art of cryptography. The 'message digest' refers to the result of computing a 'one-way function' (e.g. a hash function) to obtain the 'message digest.'

According to the definition of 'one-way function' and 'message digest,' it is infeasible to modify the original data in such as way so that the resulting message digest value (i.e. obtained by subjecting the original data to the one-way function) does not change. One salient feature of the one-way function used to obtain a message digest is that it is a so-called 'lossy transformation.' For the present disclosure, any reference to 'encrypting data' (e.g. PKI-encryption) may refer to encrypting the data or a message- digest thereof.

When data or a message digest of the data is encrypted, this refers to the case where (i) the actual data is encrypted (or a message digest thereof) 'by itself ; or (ii) the combination of the actual data along with any additional data (or a message digest thereof) is encrypted. FIG. 4C illustrates 'hybrid data 1100' that is the combination of 'base data 1110' and 'additional data 1120.' Although hybrid data is illustrated with 'one type' of additional data in FIG. 4C, it is noted that the hybrid data may include multiple 'chunks' or types of additional data.

The encrypting of 'hybrid data' may relate encrypting the hybrid data, encrypting a message digest of the hybrid data, encrypting some combination components of the hybrid data (i.e. base data 1110 and any additional data 1120

Thus, for the present disclosure whenever data (including data components) is encrypted, this may also refer to any combination of a message digest of the entire data, and message digests of any combination of 'data components.'

It is noted that for a particular type of data (e.g. visual appearance data, digital footprints data, etc), hybrid data including the particular type of data is a specific example of the particular type of data. For example, the combination of digital footprints data and other data (as 'encryption-additional data 1120) is a specific case of digital footprints data.

The phrase 'encryption-additional data' 1120 is not intended as limiting and merely refers to additional data which may be encrypted with the 'base data' (i.e. any combination of data or message digests).

Although not a limitation, in preferred embodiments PKI encryption of message digests (i.e. any combination of message digests) is preferable to PKI encryption of the actual data because message digests tend to be 'lighter weight' and hence easier to encrypt and decrypt. This principle is known in the 'protecting data from tampering' - i.e. the point is not to encrypt the data as a whole (which may be freely available except for the case of the public-key encrypted data) but rather to provide some sort of authentication that the freely-available data has not been tampered with.

In one preferred but non-limiting embodiment related to object 644 of FIG. 4A and related to FIG. 6, the biometric dynamic signature data (i.e. either 'pure data' or some sort of hybrid data) may be subjected to a PKI encryption, instead of or in addition to a PKI encryption of a message digest of biometric dynamic signature data. In some embodiments, the 'sealing of data of a document' or simply the 'sealing of a document' may refer to encrypting data or a message digest (or combination of message digests, etc - as noted above ,encrypting message digests is preferable in some embodiments especially for private-key encrypted data and/or visual appearance data).

Thus, the PKI encryption of 'visual appearance data' or 'dynamic biometric data' or 'footprints data' or any other 'base data' may actually refer to the PKI encryption of the combination (i.e. actual data or a message digest thereofO of (i) the base data 1110 and (ii) any encryption-additional data 1120 (for example and/or time-stamp data and/or a CPU-cycle counter of the CPU of device 10 and/or unique-derive-identifier data uniquely identifying signature acquisition device 10 and/or GPS or cell-phone-derived location data or any other data).

For the present disclosure, any PKI encryption of target data (i.e. static or dynamic signature data, footprints, etc) may also refer to the situation where any of the aforementioned additional data 1120 (or other additional data) is encrypted together with the target data which is base data.

In some embodiments, there may be a need to link together the 'static visual appearance data' (see 640 of FIG. 4A) and the 'dynamic signature biometric data' (see 644 of FIG. 4A) (or any other data - e.g. the footprint data or the camera data or the pixel data or any other data) in order to provide evidence of linkage between the 'more conventional' private-key encrypted static visual appearance data and the signature- supplementary data (e.g. 644 or 648 or 652 or 656).

In one example, (i) the more conventional private-key encrypted static visual appearance data is encrypted along with its time stamp (i.e. any combination of the data or message digests thereof) for example, to obtain 640 and as (ii) the additional' or 'supplementary data' (e.g. 644 or 648 or 652 or 656) is separately encrypted (i.e. any combination of data and message digests) along with its time stamp. At a later time when these objects are decrypted (for example, during litigation relating to the authenticity of visual appearance data 620 is in question and/or relating to the situation where the user signed the document to provide 620), matching of the time stamps (or of the location data or the CPU cycle counter data or the device identifier data) would provide evidence of linkage between the two types of data. In another example, it is possible when PKI-encrypting any 'signature- supplementary data' (e.g. 644 or 648 or 652 or 656) to utilize a portion of the visual appearance data 620 as 'encryption-additional data 1120' even if the 'price' is redundant encryption. In this example, the visual appearance data 620 may be encrypted a first time to generate 640 and a second time to generate encrypted 'signature- supplementary data' . 644 or 648 or 652 or 656 ).

For the present disclosure, the 'user' is the person who applies a handwriting electronic signature or a smartcard PKI electronic signature to a document. Thus, the term 'user' is synonymous with 'signer.'

A 'signature acquisition device' 10 including display screen 16 is the device where the electronic document is displayed at a time when the user applies the electronic handwriting signature (see FIGS. 2A-2B) or the smartcard PKI signature (see FIG. 3). Although the 'signature acquisition device' 10 is often illustrated as a 'tablet device,' it can have any form factor. Generally, 'signature acquisition device' 10 includes at least a display screen and electronic circuitry (for example, configured to perform any function disclosed herein). Additional features of signature acquisition device 10 are discussed below.

The term 'electronic circuitry is not limited to hardware but rather refers to any combination of hardware, firmware and software.

The phrase 'acquiring data' may include any of data generation and/or data computation of data and/or effecting a measurement to obtain measurement data and/or receiving data and/or handling data. The phrase 'acquiring data' may, in some non- limiting examples, sending a command to a component to take a measurement - for example, a command to a digital camera to obtain a digital image of a scene. The phrase 'handling data' may relate to internally obtaining and/or generating data (i.e. with no need to receive this data from any external device) or to externally receiving the data from another device. In another example, 'handling data' .is merely storing or providing the data in volatile or non-volatile memory.

Some embodiments relate to 'time matching' or 'matching of times' - for example, it is possible to time stamp both the visual appearance data as well other data (e.g. footprints data or any other data described herein). In the event that the times are substantially the same (but not necessarily exactly the same), they correlate or match. The skilled artisan from the field of electronic document tamper-protection would know if two times 'match' or 'correlate' - for example, having the same value within some sort of tolerance (e.g. within some number of hours, or preferably within some number of minutes (i.e. <1 hour or <30 minutes or <15 minutes or <5 minutes) or within less than a minute or within a tolerance of some number of seconds (<30 seconds or <15 seconds) or within a tolerance that is less than a second (e.g. some number within some of milliseconds). In different embodiments, any tolerance described herein may be used. In some preferred embodiments, the skilled artisan would know how to select a specific tolerance.

A 'non-signature page' of a 'document' is a page of a multi-page electronic document to be signed which does not include a signature location/field or require a signature. Signatures are only required on other pages of the multi-page document. In some embodiments, even though there is no requirement for a signature to be applied to the 'non-signature page,' there may be a requirement that the content of the 'non- signature page' is displayed on screen 16 of signature acquisition device 10 (e.g. as a result of a browsing operation of the user or a service agent or anyone else).

Discussion of FIGS. 2-3 and 5

FIGS. 2-3 illustrate a device for incorporating handwriting digital signatures into electronic documents in accordance with one non-limiting example. As illustrated in the figure, electronic-signature acquisition device 10 includes a display screen 16, an electronic stylus or pen 18, and a wired or wireless data port 8 (for example, USB, infrared, Bluetooth, Ethernet, WiFi, cellular and so on).

Optionally, in one non-limiting embodiment discussed below with reference to FIG. 5, electronic-signature acquisition device 10 may be in communication with document monitor device 12 (for example, operated by a customer service agent) or any other digital computing device via port 8 at the time that the user browses an electronic document on electronic- signature acquisition device 10 and/or applies a handwriting signature to the electronic document. In one particular use case, electronic-signature acquisition device 10 is configured as a printing device (for example, a USB printing device). FIG. 5B just illustrates one possible physical configuration and is not limiting. The arrow between the devices 10 and 12 illustrate that logically the two devices are in electronic communication with each other. In one non-limiting example, a wired connection is provided, for example, via port 8 (this configuration is not shown in the figure) or in any other manner. Alternatively or additionally, devices 10 and 12 are in wireless communication.

Electronic-signature acquisition device 10 (or any other device disclosed herein - for example, device 12 of FIG 5) may include any combination of digital or analog hardware (for example, including a microprocessor and optionally volatile memory such as RAM or registers), firmware and/or software (for example, computer code which is stored in volatile and/or non-volatile memory and is executable by a a computer data process such as a microprocessor). Electronic-signature acquisition device 10 may include any software and/or firmware and/or hardware element(s) including but not limited to programmable array logic (PAL) element(s), hard-wired logic element(s), field programmable gate array (FPGA) element(s), and application-specific integrated circuit (ASIC) element(s). Any instruction set architecture may be used in electronic circuitry (for example, control circuitry or any other circuitry) of signature acquisition device 10 (i.e. 'display device' 10) including but not limited to reduced instruction set computer (RISC) architecture and/or complex instruction set computer (CISC) architecture.

For the present disclosure, 'memory' refers to any combination of volatile memory (e.g. RAM or registers or any other volatile memory) and non-volatile memory (for example, flash memory, magnetic storage, disk storage, optical storage or any other kind of non-volatile memory.

As illustrated in FIGS. 2-3, device 10 (or any other device disclosed herein) includes a screen such as a flat-panel display implemented by liquid crystal display (LCD) technology and/or plasma display technology and/or organic light-emitting diode displays (OLEDs) technology and/or any other technology known in the art. In one non- limiting embodiment, device 10 may be a so-called touchscreen display employing any known touchscreen technology including but not limited to resistive touchscreen technology and/or surface acoustic wave touchscreen technology and/or capacitive technology and/or resistive touchscreen technology and/or any other touchscreeen technology known to the skilled artisan.

In yet another non-limiting example, there is no explicit requirement for touch screen technology, and it is possible to track the motion of stylus/electronic pen 18 for example, using an IR/ultrasound electronic pen and/or electromagnetic inductance tracking technique and/or any other any other stylus/electronic pen technology known to the skilled artisan.

Electronic stylus/pen 18 may be provided as a portion of device 10 or may be a separate device. In some embodiments, electronic stylus/pen 18 may communicate with device 10 using wired and/or wireless communication technology. There is no explicit requirement to provide a electronic stylus/pen 18 - in another example, the user may apply a handwriting signature to the screen of device 10 using by moving his/her fingertip across screen 16 where the position (and/or pressure) the user's fingertip is monitored to record the handwriting signature.

Control buttons (for any embodiment, either touchscreen 'virtual soft controls' 22 as illustrated in FIG. 3 and/or depressable 'hard controls' 23 as illustrated in FIG. 5) may be used) may be present in or on device 10 for browsing content (e.g. to provide translation operations and/or rotation operations and/or page transition operations and/or zoom operations and/or any other browsing operations known in the art for viewing electronic content). Alternatively or additionally, browsing operations can be carried out by document monitoring device 12 (e.g. operated by the service agent).

Referring to FIG. 5, it is noted that in some embodiments, the electronic-signature acquisition device 10 is in a wired or wireless communication with a so-called 'document monitoring device 12. For the present disclosure, a 'document monitoring device 12' is (i) configured to see the content display on electronic-signature acquisition device 10 (e.g. in a substantially synchronous manner - i.e. whatever is displayed electronic- signature acquisition device 10 is also displayed (in whole or in part or other form of representation such as thumbnails) on document monitoring device 12) while (ii) lacks the ability to apply 'user signatures' to the electronic document (though the service agent or anyone else operating document monitoring device 12 may be able to countersign the document. Thus, in one example, a so-called 'customer service agent' operates monitoring device 12 to see exactly what the consumer (or the 'user' of device 10) sees on the signature acquisition device 10 or to provide assistance or to carry out any other function. The customer service agent may then assist the consumer during the signing process.

In embodiments of the invention, monitoring device 12 is operated so that it is directly or indirectly in electronic communication with (one way in either direction or two way) with electronic-signature acquisition device 10. There is no limitation on how they can communicate- in one example, the communication may be a wired communication such as a 'USB tether' where electronic-signature acquisition device 10 is a peripheral device of monitoring device 12. Other examples may relate to wired or wireless LAN or WAN communication employing WIFI or Ethernet or cellular or Bluetooth or any other technology in which the monitoring device is associated with the electronic signature device through a logical or physical network connection

In one example, signature device 10 includes has an Ethernet connection, while both the signature device 10 and monitoring devicel 12 reside on a network but are not physically tethered to each other. The association is done by virtue of software/logical association.

Discussion of FIG. 6

FIG. 6 is flow chart of a routine for acquiring and handling electronic handwriting-signature related data.

As with any routine described herein, the order of steps should not be seen a s limiting, and not every step is required in every embodiment.

In step S101 of FIG. 6, the electronic document for signing (see for example FIG. 2A the document including Text Lines 1-4 and the signature both) ) is presented on the display screen 16 of signature acquisition device 10. The user then applies a handwriting signature to the signature acquisition device 10 (for example, using stylus 18). In step S103 at signature time, the technique of how the user actually signs his/her name is monitored to obtain 'dynamic signature biometric data' describing the manner in which the handwriting signature is applied to the electronic document. This 'dynamic signature biometric data' includes data other than is derivable from the visual appearance of the handwriting signature (for example, according to the art of handwriting biometrics/signature dynamics).

In step S107, the visual appearance data describing a static post-signature appearance of the handwriting-signed electronic document is obtained/acquired - either from the dynamic signature biometric data and offset data 9 as discussed above with reference to FIG. 4C or in any other manner.

Dynamic signature biometric data (i.e. measured/acquired in step S103and encrypted in step SI 13) includes but is not limited to character application order, handwriting velocity, handwriting direction, handwriting pressure, and pen angle data, pen up, pen down, acceleration, hovering or on- surface)

In step S109, the handwriting-signed-document visual appearance data is encrypted (for example, by electronic circuitry of signature acquisition device 10 or monitor device 12) using a private key (for example, a private key of device 10 or any other private key). As with any encryption disclosed herein, the encryption of step S109 may refer to hybrid data 1100 (where, in this particular case, the handwriting-signed- document visual appearance data is base data) or encryption of a message digest of the base data or hybrid data to obtain private-key-encrypted signed document appearance data (see 640 of FIG. 4A).

In step S113, the dynamic signature biometric data is encrypted (for example, by electronic circuitry of signature acquisition device 10 or monitor device 12 - in some preferred embodiments, specifically by electronic circuitry of signature acquisition device 10) using a public key. As with any encryption disclosed herein, the encryption of step S113 may refer to hybrid data 1100 (where, in this particular case, the handwriting- signed-document visual appearance data is base data) or encryption of a message digest of the base data or hybrid data to obtain public-key-encrypted dynamic signature biometric data (see 644 of FIG. 4A).

As was noted above (not illustrated in FIG. 6), the dynamic signature biometric data and handwriting-signed-document visual appearance data may be physically or logically associated with each other - for example, co-residing in the same data object, or 'linked' via time-stamp or CPU counter data or encrypted together as a hybrid data object. Discussion of FIGS. 7-8

Another feature discussed briefly (i.e. the 'second set' of features) above is the "Digital Footprints" feature. According to this feature, signature device 10 may maintain a real-time record of what portions of a contract document (or other electronic document to be signed) are actually viewed by the consumer browsing and/or how the document is viewed, and who controlled what was presented on the screen (signatory/user who applies an electronic or handwriting signature via device 10 or a user of monitoring device 12 such as a service agent). For example, a record may be kept of the order in which content is presented on the screen 16 of device 10, or the 'zoom factor' applied for viewing any object of the electronic document such as 'fine print.' These 'digital footprints' may be stored as meta-data along with the signed and sealed document and retrieved at a later.

FIGS. 7A-7E present a non-limiting use case relating to Digital Footprints. This feature may relate to either the handwriting signature (see FIG. 2) or the smartcard PKI signature (see FIG. 3).

In the non-limiting example of FIGS. 7A-7E, the electronic document to be signed is a three page document - for example, a contract. It is appreciated that the Digital Footprints technique may be employed when acquiring and handling electronic signatures of any other electronic document.

In the example of FIGS. 7A-7E, the main body of the document is on pages 1-2 of the contract displayed in the figures, while the contract appendix is located on page 3. For the non-limiting example of FIGS. 7A-7E, the user is only required to sign at the 'signature line' on page 2.

FIGS. 7A-7E illustrate five different frames, where each frame is a 'snapshot' of the display state of the electronic signature device 10 at a different time - the frame of FIG. 7A occurs at time tl, frame of FIG. 7B occurs at time t2, and so on. Thus, FIGS. 7A-7E describes how a user might browse (the browsing may be user controlled or controlled automatically or controlled by another person - for example, controlled by a service agent operating document monitoring device 12) through the electronic document document, In one example, the user signing the document by employing one or more user controls such as buttons 22A-22C to browse (either depressable 'hard buttons' or touchscreen 'soft buttons' or any other user control).

In the example of FIGS. 7A-7B according to the 'zoom factor' of 120% zoom, page 1 is 'too big' for all content to simultaneously appear on the screen. In FIG7A, the screen 16 displays at time tl only paragraphs 1-3 of page 1 and picture 1 of page 1 at 120% zoom - this is referred to as 'display state DSL ' In FIG. 7B, the screen 16 displays at time t2 only paragraphs 2-4 of page 1 and picture 1 of page 1 at 120% zoom - this is referred to as 'display state DS2. ' In FIG. 7C, the screen 16 displays at time t3 paragraphs 1 of page 2 along with the signature line at 100% zoom - this is referred to as 'display state DS3. The user may apply his/her handwriting signature to the signature line when the device 10 is in display state DS3.

In FIG. 7D, the screen 16 displays at time t4 paragraphs 1-2 of page 3 at 100% zoom - this is referred to as 'display state DS4. ' In FIG. 7E, the screen 16 displays at time t4 paragraphs 1-2 of page 3. However, a portion of paragraph 1 of page 3 (for example, 'fine print' text) is displayed at 200% zoom - the collective image illustrated in FIG. 7E is referred to as 'display state DS5. '

Because the document requires a signature only on page 2, page 2 is referred to as a 'signature page' of the document. Pages 1 and 3 are referred to as 'non-signature pages' of the document.

Because the document only requires a signature on page 2, page 2 is referred to as a 'signature page' of the document. Pages 1 and 3 are referred to as 'non-signature pages' of the document.

According to some embodiments of the present invention (see FIG. 12B below), in order to 'complete the electronic signature acquisition process' whereby the electronic signature is encrypted and/or the user is provided with an indication that 'electronic signature acquisition process' is complete and/or a data object of the electronic handwriting-signed document is transferred (for example, via port 8), it may be a requirement for the user to browse certain locations of the documents (and/or with certain zoom factors). This 'browsing' may be carried out via controls of signature acquisition device 10 or by controls of document monitor device 12 depending on the embodiments. In one example (see FIG. 12B below), the user may be required to browse a 'non- signature page' or a portion thereof. In one example, device 10 may be configured to request the user to browse unbrowsed portions of the document if an attempt is made to sign the document (or to instruct the device to seal the document) before requisite portions of the electronic document have been displayed on the display screen.

In yet another embodiment , so-called 'digital footprints' of what the user browses and/or what is displayed on the screen 16 and/or how it is displayed may be recorded in a display log (see step S201 of FIG. 8). This may be done in any manner - for example, it may be possible to access internal rendering or display data structures of device 10 or to monitor browse commands or an 'observing camera' observing the scene in which the user views information displayed on the screen may be employed.

FIG. 8 is a flow chart of a routine for acquiring and handling digital footprint data. In some embodiments, one result of the routine of FIG. 8 is data 648 illustrated in FIG. 4.

Step S201 is described above. In step S205, the handwriting electronic signature

(or other electronic signature such as a thumbprint, application of a PKI smartcard signature , etc) to the electronic document is acquired. In step S209, a handwriting- signed document visual appearance data (for the case where the document is signed with an electronic handwriting signature - see FIG. 2B) and/or visual appearance data of the document as displayed on screen (without necessarily including any handwriting signature - for example, for the case of the smartcard signature - see FIG. 2) is subjected to a PKI encryption. In step S213, state data of the display log (for example, describing one or more display states of the electronic document as displayed on screen 16- for example, multiple historical display states where each display state is optionally and preferably associated with a respective time stamp) is subjected to a PKI encryption

The digital footprints data (see 648 of FIG. 4A and the results of FIG. 8) may be useful in a number of situations. In one example, the fact that certain portions of the document were displayed (even non-signature pages or portions thereof) may be electronically associated with the signed document itself. In the event that the user disputes that a certain portion of the document (for example, non-signature pages or a portion thereof) was viewed by the user, or if the user complains of 'fine print' that was difficult to read, it may be possible to employ the results of the routine of FIG. 8 as evidence that the user did, indeed, view those portions and/or view those portions in a 'readable manner.'

For the use case of FIG. 7, the results of steps S201 and S213 may be used at a later time to prove that the user did indeed view, for example, page 3 of the document and the user did view certain fine print at 200% zoom.

The digital footprints may also include 'time stamp' information - this may be useful if a disputing signatory complains that even though certain content might have been displayed, it was only displayed very briefly, not affording any opportunity for proper review of the content. This may be useful (for the use case of FIG. 5) for proving that the time duration at which content of the page 3 was displayed was at least t5-t4.

As will be discussed below with reference to FIG. 12B, in some embodiments, the digital footprints data may be used to 'enforce a regime' whereby certain browsing operations are required to 'seal the document' or 'complete the signature process'

A Discussion of FIG. 9A

FIG. 9A is a flow chart of a routine for acquiring and handling pixel history data for situations where one or more electronic handwriting signatures are applied to a electronic document. In step S301, for multiple points in time, data describing the 'visual appearance' of the document as it is display on screen 16 is acquired. This describes a 'pixel history' of the document. The multiple points in time include (i) an earlier time before a specific electronic handwriting signature has been applied in full and (ii) a later signature describing the 'post-signature state' of the document - i.e. the appearance of the document that is displayed on screen 16 after the electronic handwriting signature is applied. It is appreciated that 'post signature state' and 'earlier state' (i.e. before the signature is applied or when it has been partially applied) relate to a specific signature - for example, if several signatures are applied in sequence (e.g. first SIGNATURE A and then SIGNATURE B and then SIGNATURE C), at the time after application of SIGNATURE B but before application of SIGNATURE C the document is in a 'post- signature state' relative to SIGNATURE A and SIGNATURE B but in a early state (in in this case a pre-signature state) relative to SIGNATURE C. In step S301, for each point in time, a 'pixel snapshop' describing the respective appearance of the document for the particular point in time is subjected to a PKI encryption with a private or public key. The visual appearance data or hybrid data including the visual appearance data (which is a specific case of visual appearance data) or any combinations of message digests may be subjected to the encryption.

A Discussion of FIG. 9B

According to various implementations of the routine of FIG. 9A and/or techniques for generating components of the object 600 of FIG. 4 A, any data storage scheme that permits, at a later time, reconstruction of the appearance of the digital document at the different points in time may be used. For simplicity, FIG. 9B will be described for the simple case where the 'digital history' refers to only two points in time - a later point in time when the electronic document is in a 'post-signature' state (i.e. relative to a particular handwriting signature) after the particular electronic handwriting signature and an earlier point in time before the electronic handwriting signature has been applied (or when it has only been partly applied).

According to Example 1 of FIG. 9B, it is possible to store two images (e.g. two tiff files or using any other data representation) - a first image 1512 describing the appearance of the document at the 'earlier time' before the time of the 'post-signature' state and a second image 1516 describing the appearance of the document at the 'later time' when the document appearance is in the 'post-signature' signature state.

In other scheme (Examples 2-3), it is possible to save merely the 'delta information' describing how the appearance of the data has changed between an earlier point in time (i.e. before the 'post-signature state') and the later point in time (i.e. when the document is in the 'post-signature state').

FIG. 9B is not intended as comprehensive and any other scheme or combination of schemes may be used, as long as it is possible to subsequently compute from the data objects of the scheme what the appearance of the document as displayed on screen 16 was at various time including the 'post-signature time' and the earlier time.

To protect the 'pixel history data' from tampering, it is possible to subject one or more data objects (or any combination of message digests thereof - also the 'data objects' may be hybrid data objects combined with each other and/or with other 'additional data' such as time stamp data) to a PKI encryption(s).

In one preferred but optional embodiment, object 640 of FIG. 4A includes PKI- encrypted later-time image data 1516 (i.e. any combination of the data or hybrid data or message digests thereof) and object 652 includes PKI-encrypted appearance change data 1522.

Thus, it is noted that the example of FIG. 9B may relate to different techniques of 'appearance data tracking' where image 1 1512 or image 2 1516 may serve as a baseline while appearance change data 1522 describes deviations of the document appearance relative to any baseline.

A Discussion of FIGS. 10-11

FIGS. 10A-10B illustrate systems including a digital camera 32 for acquiring an image of the user 42 (or of a physical credential of the user - e.g. passport, driver's license, etc) at a time that the user applies one or more electronic handwriting signatures and/or smartcard PKI signature to a document. The digital camera may be associated with (or with the housing of) of display device 10 as in FIG. 10A. or may be deployed in another location as in FIG. 10B.

Camera 32 may be a 'still camera' or a video camera.

Digital camera images the scene including the user (or a portion thereof) or his/her credentials from any angle. Optionally but preferably, a time stamp of the time of this imaging is captured and associated with the image of the scene. It is possible to match this time stamp with other time stamps - for example, time stamps of when the user applies an electronic signature.

In step S401, an electronic signature (e.g. handwriting signature of smart card PKI signature) is applied to a document (optionally but preferably time-stamped). In step S405, an image of a scene including the user and/or his/her credential is acquired at a time which 'matches' the time of the signature (e.g. substantially at the same time) when the user is physically located in the same scene as the device 10 displaying the signed document. In step S413, the scene image data is obtained using electronic camera 32 is subjected to a PKI encryption using a public or private key. In some preferred embodiments, a public key is used. Electronic camera may include any mechanical or electrical (i.e. digital or analog) or software component known in the art of digital photography - for example, sensors such as CCDs or CMOS sensors or any other kind of sensor and/or a mechanical or electrical shutter or computer memory or any other component.

In some embodiments, a command is sent to camera 32 in response to user actions relating to applying an electronic signature.

A Discussion of FIGS. 12A-12B

In some embodiments, multiple signature fields are provided on a single page or across multiple pages of the electronic document. In one example, an income tax form includes multiple schedules each of which requires a signature. In another example, several declarations appear on a single page (for example, a person needs to separately apply a signature to each declaration). In another example, a contract includes multiple appendixes, each of which requires a signature. In another example, signatures from multiple (e.g. two) people are required - for example, a husband and a wife.

Application of a handwriting signature to multiple fields would require the physical application of the signature (e.g. by moving the object over the appropriate location on the screen) for each signature. For a smartcard signature, it is possible for each signature field to have a visual cue (for example, a blinking cue or a color cue such as appearing in red). This cue indicates the 'active field' to which the smartcard PKI signature is applied. After the user signs the 'active' field, then a new field may become active (i.e. either immediately or at a later time - for example, after the user browses to another page).

In the routine of FIG. 12A, the user can browse content in step S501. In step S505, an electric signature is applied to the document.

According to step S509, only if all signatures have been applied (i.e. in general all signatures are applied in a specific order or with a specific timing - e.g. within a certain number of seconds or minutes or within in a single 'session') is it possible, in step S513, to effect an operation to complete the signatures to the electronic document - for example, to send an audio or visual indication to the user that the 'signature process' has been completed or to make certain 'soft controls' (e.g. buttons) available to a user or to make it possible for the same user (or another user) to sign a 'different' electronic document (i.e. to 'move on' to the next document).

Steps S501, S505 and S513 of FIG. 12B are as in FIG. 12. In step S521, it is a requirement to complete the signature process for one or more browsing operations to have occurred in order to reach S513. These browsing operations are more than the 'minimum browsing operations' associated with scrolling or moving to each signature location/field. These 'non-minimal' browsing operations required in S521 may include browsing a non-signature page, or 'zooming in' on certain text or visiting a certain location in the document more than one or any other 'non-browsing operation.'

In FIGS. 12A-12B there are two 'NO branches' leaving respectively from steps

S509 or S521. In one example (BRANCH 1), the user can apply an additional signature without any need for further browsing operations. In another example, (BRANCH 2) the user (and/or the operator of document monitor device 12) can choose to continue browsing the document. One or both options may be available when the all fields have not been correctly signed (FIGS. 12A) and/or when required browsing operations (i.e. beyond the 'minimum browsing operations) have not been carried out.

In some embodiments, to carry out step S521, for each point in time of a set of one or more points in time, it is possible to analyze digital footprints data describing how the content states of the display screen of device 10 have changed in a 'historical time period' up to the point in time - for example, there may be a set of points of time tl, t2, t3, (referred to as ί,)· This point in time i, is associated with a matching 'display state' DSi describing the content displayed (and how it is display) on screen 16 of device 10 at the point in time ί,.

For each point in time i„ it is possible to determine if non-minimal historical browsing operations that are beyond the minimum browsing requirements for the 'current display state' DSi associated with the point in time. For example, if a document has 10 pages, for a display state DSi where content of page 3 is displayed, it may be required to 'scroll down' from page to page 3 in order to view page 3. Thus, the minimum browsing operations for this example may simply be a 'page down' operation from page 1 to page 3. However, if the user first views page 9 before viewing page 3 (thus, the 'view time' for page 9 is before the time ¾ DSi ), this entails non-minimal browse operations (i.e. which are 'historical' relative to time ί,).

In some embodiments of the invention, in order to acquire a signature in step S505 or to effect a document signature completion operation in step S513, it may be required that first one or more non-minimal browsing operations (i.e. non-minimal relative to the display state of the 'candidate' signature acquisition or completion operation).

Towards this end, it may be possible for each point in time i„ to analyze digital footprints data and determine if required 'non-minimal historical browsing operations' not required for the instantaneous display state DSi of the point in time ¾ have been historically carried out in a historical time period up to time ¾. In the event that these browsing operations have been carried out (for example, a zoom operation or viewing a certain page such as a non-signature page or viewing a certain page for a certain amount of time or viewing pages in a certain order or any other 'non- minimal historical browsing operations,' then it is possible to refuse acquisition of a signature and/or in step S521 to refuse any signature completion operation. Thus, step S505 or S513 may be contingent upon a 'positive determining' that the browsing operations have been carried out in a historical time period that is. historical relative to time ¾.

In some embodiments, it is enough that the display screen 16 of device 10 has, during the 'historical time period,' exhibited the requisite content states associated with the 'required browsing operations.'

In other embodiments , active browsing operations by controls of signature acquisition device 10 or document management device 10 may be required during the a historical time period that is. historical relative to time t

It is further noted that any of the embodiments described above may further include receiving, sending or storing instructions and/or data that implement the operations described above in conjunction with the figures upon a computer readable medium. Generally speaking, a computer readable medium may include storage media or memory media such as magnetic or flash or optical media, e.g. disk or CD-ROM, volatile or non-volatile media such as RAM, ROM, etc. as well as transmission media or signals such as electrical, electromagnetic or digital signals conveyed via a communication medium such as network and/or wireless links.

Having thus described the foregoing exemplary embodiments it will be apparent to those skilled in the art that various equivalents, alterations, modifications, and improvements thereof are possible without departing from the scope and spirit of the claims as hereafter recited. In particular, different embodiments may include combinations of features other than those described herein. Accordingly, the claims are not limited to the foregoing discussion.

Claims

WHAT IS CLAIMED IS:

1) A method of facilitating the protection from tampering of a handwriting-signed electronic document that is generated when an electronic handwriting signature is applied to an electronic document displayed on a display screen of a document display device, the method comprising:

a) as an object is moved over the display screen to apply the electronic handwriting signature to the electronic document, acquiring dynamic signature biometric data of the applied electronic handwriting signature;

b) acquiring handwriting-signed-document visual appearance data describing a static post-signature appearance of the handwriting-signed electronic document; c) effecting a private key encryption of the signed-document visual appearance data or a message digest thereof; and

d) effecting a public key encryption of the dynamic signature biometric data of the user-applied handwriting electronic signature or a message digest thereof.

2) The method of claim 1 wherein the acquiring of the handwriting-signed-document visual appearance data includes at least one of:

i) effecting a correlation between the dynamic signature biometric data and visual content of the electronic document according to offset data; and

ii) interpolating signature points derived from the dynamic signature biometric data

3) The method of claim 1 wherein the public key encryption is applied to a hybrid data object or a message digest thereof, the hybrid data object comprising dynamic signature biometric data and handwriting-signed-document visual appearance data.

4) The method of claim 3 wherein the hybrid data object further comprises digital footprint data.

5) The method of any preceding claim further comprising:

e) causing the private-key-encrypted handwriting-signed document visual appearance data or a message digest thereof and the public-key-encrypted dynamic signature biometric data of a message digest thereof to co-reside in a single container data object. 6) The method of claim 5 wherein the container data object is an image object including both a viewable image and metadata encapsulated within the image object, and wherein both the private-key-encrypted visual appearance data or message digest thereof as well as the public -key-encrypted dynamic signature biometric data or message digest thereof are embedded within the image object as metadata.

7) The method of any of claims 5-6 wherein the container data object is a file.

8) The method of claim 7 wherein the file is selected from the group consisting of a single page .tiff file, a multi-page .tiff file, a single page .pdf file, a multi-page .pdf file, an .xml file and a .zip file.

9) The method of any of claims 5-9 wherein step (e) is carried out within the document display device.

10) The method of any preceding claim wherein step (c) and/or step (d) is carried out within the document display device.

11) The method of any of the preceding claims wherein the private key encryption of step (c) employs at least one private key selected from the group consisting of:

i) a private key that is specific to the document display device to which the user applies an electronic signature; and

ii) an external private key that is external to the user document display device.

12) The method of any preceding claim wherein the public key encryption of step (d) is carried out to a message digest of handwriting-signed-document visual appearance data or to hybrid data thereof.

13) The method of any of preceding claim wherein the public key encryption is carried out to a hybrid data object including both the dynamic signature biometric data and at least one of:

i) time-stamp data;

ii) unique device indication data;

iii) document appearance data;

iv) identifying data describing the user who applies the electronic signature;

v) identifying data describing a customer service agent;

vi) branch information;

vii) location information describing a location where the document is signed. 14) The method of any preceding claim wherein the object moved over the displayed screen is a finger or a stylus.

15) The method of any preceding claim wherein:

i) the private key encryption is carried out to a first hybrid data object or message digest thereof, the hybrid data object including first time stamp data; and ii) the public key encryption is carried out to a second hybrid data object or message digest thereof, the hybrid data object including second time stamp data which matches the first time stamp data.

16) Apparatus for facilitating the protection from tampering of an electronic document, the apparatus comprising:

a) electronic circuitry including memory for storing the electronic document; and b) a display screen configured to display information responsive to signals received from the electronic circuitry, the display screen configured to display the electronic document as modified by an electronic signature in response to moving an object over the display screen, wherein the electronic circuitry is configured to: i) acquire or handle dynamic signature biometric data of the applied electronic handwriting signature;

ii) acquire or handle handwriting-signed-document visual appearance data describing a static post-signature appearance of the handwriting-signed electronic document;

iii) effect a private key encryption of the signed-document visual appearance data or a message digest thereof; and

iv) effect a public key encryption of the dynamic signature biometric data of the user-applied handwriting electronic signature or a message digest thereof.

17) The apparatus of claim 16 wherein the electronic circuitry includes any combination of hardware, software and firmware.

18) The apparatus of any of claims 16-17 wherein the display screen is configured as touch screen. 19) The apparatus of any of claims 16-18 further comprising a digital pen in communication with the electronic circuitry.

20) The apparatus of any of claim 16-19 wherein the apparatus is configured to effect any method of any of claims 1-15.

21) A method of operating a document display device including a display screen to facilitate the acquisition of an electronic handwriting signature of an electronic document displayed on the document display device, the method comprising:

a) monitoring content displayed on a display screen of the document display device along with display times to acquire digital footprints data describing a sequence of display states of the electronic document correlated with respective display times; and

b) acquiring at least one type of digital signature data describing a digital signature applied to the electronic document using the document display device at a time after at least some of the display states, the digital signature data selected from the group consisting of:

i) handwriting-signed-document visual appearance data describing a static post-signature appearance of an electronic-handwriting-signed electronic document; and

ii) smartcard PKI data describing the PKI encryption of the electronic document or a message digest thereof;

c) subjecting the digital footprints data or a message digest thereof to a PKI encryption,

wherein the digital signature data is time-stamped data having a time stamp that matches one or more of the display times of the digital footprints data and/or the PKI encryption of step (c) is carried out to hybrid data comprising both the digital signature data and the digital footprints data or any combination of message digests thereof and/or both the PKI-encrypted digital footprints data and the and the encrypted or unencrypted digital signature data or any combination of message digests thereof are embedded into a single container data object.

22) The method of claim 21 wherein the monitoring of the display content is carried out according to at least one of: i) an image of the display screen acquired by an observer electronic camera;

ii) logged browsing commands; and

iii) an internal data structure of the document display device.

23) The method of claim 22 wherein the logged browsing commands are applied to one or more of:

i) the document display device; and

ii) a document monitor device in communication with the document display device.

24) The method of 23 wherein the digital display data describes both document display device browsing commands as well as the document monitor browsing commands.

25) Apparatus for facilitating the protection from tampering of an electronic document, the apparatus comprising:

a) electronic circuitry including memory for storing the electronic document; and b) a display screen configured to display information responsive to signals received from the electronic circuitry, the electronic circuitry being configured to: i) acquire or handle digital footprints data describing a sequence of display states of the electronic document correlated with respective display times; ii) acquire or handle at least one type of digital signature data describing a digital signature applied to the electronic document using the document display device at a time after at least some of the display states, the digital signature data selected from the group consisting of:

A) handwriting-signed-document visual appearance data describing a static post-signature appearance of an electronic- handwriting-signed electronic document; and

B) smartcard PKI data describing the PKI encryption of the electronic document or a message digest thereof;

wherein the digital signature data is time-stamped data having a time stamp that matches one or more of the display times of the digital footprints data and/or the digital footprints PKI encryption of is carried out to hybrid data comprising both the digital signature data and the digital footprints data or any combination of message digests thereof and/or both the PKI-encrypted digital footprints data and the encrypted or unencrypted digital signature data or any combination of message digests thereof are embedded into a single container data object.

26) The apparatus of claim 25 wherein the electronic circuitry includes any combination of hardware, software and firmware.

27) The apparatus of any of claims 25-26 wherein the display screen is configured as touch screen.

28) The apparatus of any of claims 25-28 further comprising a digital pen in communication with the electronic circuitry.

29) The apparatus of any of claims 25-28 wherein the apparatus is pen-less and lacks a digital pen.

30) The apparatus of any of claim 25-29 wherein the apparatus is configured to effect any method of any of claims 21-24.

31) A method of facilitating the protection from tampering to a handwriting-signed electronic document that is generated when an electronic handwriting signature is applied to an electronic document displayed on a display screen of a document display device, the method comprising:

effecting PKI encryption(s) to both:

i) later-time appearance data or a message digest thereof, the later-time appearance data describing a post-signature appearance electronic document; and

ii) earlier-time appearance data or a message digest thereof, the earlier- time appearance data describing an appearance of the electronic document at an earlier time before the electronic handwriting signature has been completely applied to the electronic document.

32) The method of claim 31 wherein steps (i) and (ii) are carried out by a single PKI encryption to hybrid data comprising any combination of later-time appearance data and earlier-time appearance data or one or more message digest(s) thereof. 33) The method of claim 31 wherein steps (i) and (ii) are carried out so that PKI encryptions of the later-time appearance data or a message digest thereof and the time appearance data or a message digest thereof are respectively performed as separated PKI encryptions.

34) The method of any of claims 31-33 wherein the method is carried out so that both the sealed later-time appearance data and the sealed earlier-time appearance data co-reside in a single data object.

35) The method of any of claims 31-34 wherein the earlier-time appearance data describes a partially-signed state of the document when only a partial handwriting signature appears in the document.

36) The method of any of claims 31-35 wherein the earlier-time appearance data describes an unsigned state of the document when no handwriting signature appears in the document.

37) The method of any of claims 31-36 wherein:

i) first and second electronic handwriting signatures of first and second persons are respectively and chronologically applied to the electronic document; and ii) term post-signature state applies specifically to the second electronic handwriting signature such that:

A) the earlier-time appearance data describes an appearance of the document after the first electronic signature has been applied to the document but before the second electronic signature has been applied to the document; and

B) the later-time appearance data describes an appearance of the document after both the first and second electronic signatures have been applied to the document.

38) Apparatus for facilitating the protection from tampering of an electronic document, the apparatus comprising:

a) electronic circuitry including memory for storing an electronic document; and b) a display screen configured to display information responsive to signals received from the electronic circuitry, the display screen being configured to display the electronic document as modified by an electronic signature in response to moving an object over the display screen, wherein the electronic circuitry is configured to effect PKI encryption(s) to both: i) later-time appearance data or a message digest thereof, the later-time appearance data describing a post-signature appearance electronic document; and

39) The apparatus of claim 38 wherein the electronic circuitry includes any combination of hardware, software and firmware.

40) The apparatus of any of claims 38-39 wherein the display screen is configured as touch screen.

41) The apparatus of any of claims 38-40 further comprising a digital pen in communication with the electronic circuitry.

42) The apparatus of any of claim 38-41 wherein the apparatus is configured to effect any method of any of claims 31-36.

43) A method of facilitating the protection from tampering of a handwriting-signed electronic document that is generated when an electronic handwriting signature is applied to an electronic document displayed on a display screen of a document display device, the method comprising:

a) acquiring at least one type of digital signature data describing a digital signature applied to the electronic document using the document display device, the digital signature data selected from the group consisting of:

b) obtaining digital image data by causing a digital camera to acquire at least one digital camera image selected from the group consisting of:

i) an image of a scene including the user; and

ii) an image of a scene including a visual credential of the user; c) subjecting the digital image data or a message digest thereof to a PKI encryption,

wherein the digital image data is time-stamped data having a time stamp that matches a time stamp of the digital signature data and/or the PKI encryption of step (c) is carried out to hybrid data comprising both the digital signature data and the digital image data or any combination of message digests thereof and/or both the PKI-encrypted digital image data and the encrypted or unencrypted digital signature data or any combination of message digests thereof are embedded into a single container data object.

44) Apparatus for facilitating the protection from tampering of an electronic document, the apparatus comprising:

a) electronic circuitry including memory for storing the electronic document; and b) a display screen configured to display information responsive to signals received from the electronic circuitry, the electronic circuitry being configured to: i) acquire or handle at least one type of digital signature data describing a digital signature applied to the electronic document using the document display device at a time after at least some of the display states, the digital signature data selected from the group consisting of:

ii) acquire or handle digital image data by causing a digital camera to acquire at least one digital camera image selected from the group consisting of:

i) an image of a scene including the user; and ii) an image of a scene including a visual credential of the user; c) subjecting the digital image data or a message digest thereof to a PKI encryption, wherein the digital image data is time-stamped data having a time stamp that matches a time stamp of the digital signature data and/or the PKI encryption of step (c) is carried out to hybrid data comprising both the digital signature data and the digital image data or any combination of message digests thereof and/or both the PKI-encrypted digital image data and the encrypted or unencrypted digital signature data or any combination of message digests thereof are embedded into a single container data object.

45) A method of operating a document display device including a display screen to facilitate the acquisition of an electronic handwriting signature or of a smartcard PKI signature of an electronic document displayed on the document display device, the method comprising:

a) for each location within the electronic document of a plurality of locations, making the location available for respective acquisition of a respective digital signature selected from the group consisting of a respective smartcard PKI signature and a respective handwriting signature; and

b) in the event that handwriting signatures are acquired in step (a), subjecting visual appearance data or a message digest thereof to a PKI encryption, the visual appearance data describing the document including all handwriting signature(s) of step (a)

wherein the method is carried out such that at least one condition selected from the group consisting of a first condition, a second condition and a third condition is true, the first, second and third conditions being defined as follows:

according to the first condition, for at least one of the locations, a respective signature acquisition or PKI encoding is contingent upon previous signature acquisitions in other locations of the plurality of locations within the electronic document;

(i) according to the second condition, soft user controls for

completing the signature process made available in a manner that is contingent upon signature acquisition for all locations of the plurality of locations;

(ii) according to the third condition, the user is only provided with a visual or audio indication that the document is sealed if signatures have been acquired at all locations of the plurality of locations; and (iii) according to the fourth condition, a mode transition from a first mode to a second mode is carried out in a manner that is contingent acquisition of digital signatures for all locations of the plurality of locations, such that in the first mode the electronic document is available for signature and a in the second mode a different electronic document is available for signature.

46) A method of operating a document display device including a display screen to facilitate the acquisition of an electronic handwriting signature or of a smartcard PKI signature of an electronic document displayed on the document display device, the method comprising:

a) monitoring content displayed on a display screen of the document display device along with display times to acquire digital footprints data describing a sequence of display states of the electronic document correlated with respective display times;

b) for a set of points in time comprising one or more points in time, for each time point of the time point set, respectively analyzing the digital footprints data to respectively determine if one or more non-minimal historical browsing operations(s) that are not minimal for the respective instantaneous display state of the point in time have been carried out, thereby respectively effecting a positive or negative determination for the time point;

c) acquiring at least one type of digital signature data describing a digital signature applied to the electronic document using the document display device at a time after at least some of the display states, the digital signature data selected from the group consisting of:

d) in the event that handwriting signatures are acquired in step (b), subjecting visual appearance data or a message digest thereof to a PKI encryption, the visual appearance data describing the document including all handwriting signature(s) of step (a),

wherein the method is carried out such that at least one signature operation selected from a signature operation set is contingent upon a positive determination being made in step (b), the signature operation set consisting of the following operations:

i) one or more signature acquisitions of step (c);

ii) providing soft user controls display on the display screen

document display device for completing the signature process; and iii) effecting a mode transition from a first mode to a second mode where in the first mode the electronic document is available for signature and in the second mode a different electronic document is available for signature.