WO2022197436A1 - Révélation et sélection de groupement d'encres - Google Patents

Révélation et sélection de groupement d'encres Download PDF

Info

Publication number
WO2022197436A1
WO2022197436A1 PCT/US2022/018399 US2022018399W WO2022197436A1 WO 2022197436 A1 WO2022197436 A1 WO 2022197436A1 US 2022018399 W US2022018399 W US 2022018399W WO 2022197436 A1 WO2022197436 A1 WO 2022197436A1
Authority
WO
WIPO (PCT)
Prior art keywords
digital
canvas
instance
ink
digital ink
Prior art date
Application number
PCT/US2022/018399
Other languages
English (en)
Inventor
Irina Smolyanskaya
Yibo SUN
Julio Estrada
Arcadio García Salvadores
Original Assignee
Microsoft Technology Licensing, Llc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing, Llc filed Critical Microsoft Technology Licensing, Llc
Publication of WO2022197436A1 publication Critical patent/WO2022197436A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/0482—Interaction with lists of selectable items, e.g. menus
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04842—Selection of displayed objects or displayed text elements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/10—Text processing
    • G06F40/166—Editing, e.g. inserting or deleting
    • G06F40/171—Editing, e.g. inserting or deleting by use of digital ink
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/041—Indexing scheme relating to G06F3/041 - G06F3/045
    • G06F2203/04101—2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup

Definitions

  • Embodiments described herein provide systems and methods for digital inking and digital ink editing.
  • Digital inking allows handwriting and drawings to be added electronically to electronic documents.
  • users can use a pen to perform digital inking within a canvas provided within an electronic document for journaling (e.g., note-taking, generating lists, adding captions or annotations to photos, or the like), planning (e.g., calendaring, goal or habit tracking, to-dos, or the like), and content-creating (e.g., diagramming, mind mapping, storytelling, and the like).
  • This pen can be referred to as a digital pen as it can be used to create digital ink within a canvas.
  • the digital pen can include a passive pen (e.g., a stylus) or an active pen.
  • Inking applications which provide a user with a canvas for performing digital inking, also provide tools for editing digital ink. For example, using these tools, a user may delete a word or portion of existing digital ink or may change the position of digital ink within a canvas (e.g., to change the content layout).
  • Modification or editing of digital ink described herein is referred to as digital ink editing and includes changing the physical appearance or location of digital ink or changing other metadata of digital ink, which may not change the physical appearance of the digital ink within the canvas.
  • Editing digital ink within an inking application often requires a user to switch from an inking mode to an editing mode, which may require the user to switch from using the digital pen on the canvas to using menus or other selection mechanisms or input mechanisms within the inking application to navigate and select an appropriate editing tool.
  • This switching between an inking mode and an editing mode creates inefficiencies. For example, to move digital ink to a new position within the canvas, a user may need to navigate to a menu and select a lasso tool (e.g., using the digital, a cursor-controlled device, a keyboard command, their finger, or the like), use the lasso tool to select digital ink, and then drag the selected digital ink to a new location within the canvas.
  • a lasso tool e.g., using the digital, a cursor-controlled device, a keyboard command, their finger, or the like
  • embodiments described herein provide methods and systems for performing digital ink editing using pen commands, which may invoked using digital ink or other input using the digital pen and without requiring a user to manually switch from an inking mode to a digital ink editing mode.
  • some of the methods and systems described herein translate digital ink received through a canvas to a particular command, such as an editing command.
  • This translation or identification of commands can be based on one or more strokes of the digital ink (e.g., whether one or more strokes match one or more predetermined signifiers associated with a particular command), the context of the digital ink (e.g., other digital ink within the canvas, where the digital ink is added, a sequence or process for how the digital ink was added, etc.), or a combination thereof.
  • the digital ink can represent natural gestures for particular commands. For example, in one embodiment described in more detail below, a user can draw a connecting line between two digital ink instances within a canvas to establish a link (e.g., a hyperlink) between the instances.
  • a user can use digital ink added to the canvas to invoke the “insert link” command as well as designate what instances should be linked.
  • some of the methods and systems described herein use input other than digital ink (i.e., digital strokes) provided via the digital pen within the canvas, such as tilt position, hover position, or the like, to invoke a particular command.
  • digital ink i.e., digital strokes
  • using input from the digital pen within the canvas to invoke various commands or editing functionality enables a user to efficiently create and manage digital content within a canvas without having to change input modes or access menus or toolbars, which can break the user’s focus or otherwise discourage a user from using digital ink.
  • one embodiment provides a system for linking digital ink instances within an electronic document.
  • the system includes a memory configured to store instructions and an electronic processor coupled to the memory.
  • the electronic processor through execution of the instructions stored in the memory, is configured to receive detected interactions between a touchscreen and a digital pen representing one or more digital strokes within a canvas and determine whether the one or more digital strokes match a predetermined signifier, the predetermined signifier including a predetermined mark and a line extending from the predetermined mark.
  • the electronic processor is also configured to, in response to determining that the one or more digital strokes match the predetermined signifier, determine a first digital ink instance and a second digital ink instance included in the canvas based on a position of the one or more digital strokes within the canvas and update metadata of the first digital ink instance to include an identifier of the second digital ink instance.
  • the electronic processor is further configured to, in response to receiving a selection of the first digital ink instance within the canvas, navigate to the second digital ink instance within the canvas based on the identifier of the second digital ink instance included in the metadata of the first digital ink instance.
  • Another embodiment provides a method of linking digital ink instances within an electronic document.
  • the method includes receiving detected interactions between a touchscreen and a digital pen representing one or more digital strokes within a canvas and determining, with an electronic processor, whether the one or more digital strokes match a predetermined signifier, the predetermined signifier including a predetermined mark and a line extending from the predetermined mark.
  • the method also includes, in response to determining that the one or more digital strokes match the predetermined signifier, determining, with the electronic processor, a first digital ink instance and a second digital ink instance included in the canvas based on a position of the one or more digital strokes within the canvas and updating, with the electronic processor, metadata of the first digital ink instance to include an identifier of the second digital ink instance.
  • the method further includes, in response to receiving a selection of the first digital ink instance within the canvas, navigating to the second digital ink instance within the canvas based on the identifier of the second digital ink instance included in the metadata of the first digital ink instance.
  • a further embodiment provides non-transitory computer readable medium storing instructions that, when executed by an electronic processor, perform a set of functions.
  • the set of functions includes receiving detected interactions between a touchscreen and a digital pen representing one or more digital strokes within a canvas and determining whether the one or more digital strokes match a predetermined signifier, the predetermined signifier including a predetermined mark and a line extending from the predetermined mark.
  • the set of functions also includes, in response to determining that the one or more digital strokes match the predetermined signifier, determining a first digital ink instance and a second digital ink instance included in the canvas based on a position of the one or more digital strokes within the canvas and updating metadata of the first digital ink instance to include an identifier of the second digital ink instance.
  • the set of functions includes, in response to receiving a selection of the first digital ink instance within the canvas, navigating to the second digital ink instance within the canvas based on the identifier of the second digital ink instance included in the metadata of the first digital ink instance.
  • Another embodiment provides a system for providing answers to digitally inked questions within an electronic document.
  • the system includes a memory configured to store instructions and an electronic processor coupled to the memory.
  • the electronic processor through execution of the instructions stored in the memory, is configured to receive detected interactions between a touchscreen and a digital pen representing one or more digital strokes within a canvas and determine whether the one or more digital strokes match a predetermined signifier.
  • the electronic processor is also configured to, in response to determining that the one or more digital strokes match the predetermined signifier, determine a digital ink instance included in the canvas based on a position of the one or more digital strokes within the canvas, extract text from the digital ink instance, submit the extracted text to an answer service as a query, receive a text-based answer to the query from the answer service, convert the text-based answer to digital ink, and add the digital ink to the canvas.
  • Embodiments described herein also provide a method of providing answers to digitally inked questions within an electronic document.
  • the method includes receiving detected interactions between a touchscreen and a digital pen representing one or more digital strokes within a canvas, and determining, with an electronic processor, whether the one or more digital strokes match a predetermined signifier.
  • the method also includes, in response to determining that the one or more digital strokes match the predetermined signifier, determining, with the electronic processor, a digital ink instance included in the canvas based on a position of the one or more digital strokes within the canvas, extracting, with the electronic processor, text from the digital ink instance, submitting, with the electronic processor, the extracted text to an answer service as a query, receiving, with the electronic processor, a text-based answer to the query from the answer service, converting, with the electronic processor, the text-based answer to digital ink, and adding, with the electronic processor, the digital ink to the canvas.
  • Yet another embodiment provides non-transitory computer readable medium storing instructions that, when executed by an electronic processor, perform a set of functions.
  • the set of functions includes receiving detected interactions between a touchscreen and a digital pen representing one or more digital strokes within a canvas and determining whether the one or more digital strokes match a predetermined signifier.
  • the set of functions also includes, in response to determining that the one or more digital strokes match the predetermined signifier, determining a digital ink instance included in the canvas based on a position of the one or more digital strokes within the canvas, extracting text from the digital ink instance, submitting the extracted text to an answer service as a query, receiving a text-based answer to the query from the answer service, converting the text-based answer to digital ink, and adding the digital ink to the canvas.
  • An additional embodiment provides a system for converting text to digital ink.
  • the system includes a memory configured to store instructions and an electronic processor coupled to the memory.
  • the electronic processor through execution of the instructions stored in the memory, is configured to generate an image of text, trace characters within the generated image to establish a plurality of ink points and a plurality of digital strokes, generate a digital ink instance based on the plurality of ink points and the plurality of digital strokes, and provide the digital ink instance within a user interface.
  • Another embodiment provides a method of converting text to digital ink.
  • the method includes generating an image of text, tracing characters within the generated image to establish a plurality of ink points and a plurality of digital strokes, generating, with an electronic processor, a digital ink instance based on the plurality of ink points and the plurality of digital strokes, and providing, with the electronic processor, the digital ink instance within a user interface.
  • a further embodiment provides non-transitory computer readable medium storing instructions that, when executed by an electronic processor, perform a set of functions.
  • the set of functions includes generating an image of text, tracing characters within the generated image to establish a plurality of ink points and a plurality of digital strokes, generating a digital ink instance based on the plurality of ink points and the plurality of digital strokes, and providing the digital ink instance within a user interface.
  • One embodiment also provides a system for aggregating digital ink instances within an electronic document.
  • the system includes a memory configured to store instructions and an electronic processor coupled to the memory.
  • the electronic processor through execution of the instructions stored in the memory, is configured to receive detected interactions between a touchscreen and a digital pen representing one or more digital strokes within a canvas and determine whether the one or more digital strokes match a predetermined signifier.
  • the electronic processor is also configured to, in response to determining that the one or more digital strokes match the predetermined signifier, determine a first digital ink instance included in the electronic document based on a position of the one or more digital strokes within the canvas, duplicate the first digital ink instance to create a second digital ink instance, and add the second digital ink instance to a designated portion of the electronic document.
  • Another embodiment provides a method of aggregating digital ink instances within an electronic document.
  • the method includes receiving, with an electronic processor, detected interactions between a touchscreen and a digital pen representing one or more digital strokes within a canvas and determining, with the electronic processor, whether the one or more digital strokes match a predetermined signifier.
  • the method also includes, in response to determining that the one or more digital strokes match the predetermined signifier, determining, with the electronic processor, a first digital ink instance included in the electronic document based on a position of the one or more digital strokes within the canvas, duplicating, with the electronic processor, the first digital ink instance to create a second digital ink instance, and adding, with the electronic processor, the second digital ink instance to a designated portion of the electronic document.
  • Yet another embodiment provides non-transitory computer readable medium storing instructions that, when executed by an electronic processor, perform a set of functions.
  • the set of functions includes receiving detected interactions between a touchscreen and a digital pen representing one or more digital strokes within a canvas and determining whether the one or more digital strokes match a predetermined signifier.
  • the set of functions also includes, in response to determining that the one or more digital strokes match the predetermined signifier, determining, within an electronic document, a first digital ink instance included in the canvas based on a position of the one or more digital strokes within the canvas, wherein the first digital ink instance has a position within the canvas adjacent to the one or more digital strokes in a predetermined direction, duplicating the first digital ink instance to create a second digital ink instance, and adding the second digital ink instance to a designated portion of the electronic document.
  • Another embodiment provides a system for switching an input mode of a digital pen used with an electronic document.
  • the system includes a memory configured to store instructions and an electronic processor coupled to the memory.
  • the electronic processor in conjunction with the instructions stored in the memory, is configured to receive a tilt angle of the digital pen while the digital pen is operating in a first input mode, the first input mode being an inking mode, compare the tilt angle of the digital pen to a predetermined threshold associated with a second input mode, and, in response to the tilt angle satisfying the predetermined threshold, setting the input mode of the digital pen to the second input mode.
  • a further embodiment provides a method for switching an input mode of a digital pen used with an electronic document.
  • the method includes receiving, with an electronic processor, a tilt angle of the digital pen while the digital pen is operating in a first input mode, the first input mode being an inking mode, comparing, with the electronic processor, the tilt angle of the digital pen to a predetermined threshold associated with a second input mode, and, in response to the tilt angle satisfying the predetermined threshold, setting, with the electronic processor, the input mode of the digital pen to the second input mode, the second input mode including a highlighting mode.
  • Yet another embodiment provides non-transitory computer readable medium storing instructions that, when executed by an electronic processor, perform a set of functions.
  • the set of functions including receiving a tilt angle of a digital pen while the digital pen is operating in a first input mode, the first input mode being an inking mode, comparing the tilt angle of the digital pen to a predetermined threshold associated with a second input mode, and, in response to the tilt angle satisfying the predetermined threshold, setting an input mode of the digital pen to the second input mode, the second input mode including a highlighting mode.
  • a still further embodiment provides a system for revealing boundaries of a digital ink instance within an electronic document.
  • the system includes a memory configured to store instructions and an electronic processor coupled to the memory.
  • the electronic processor through execution of the instructions stored in the memory, is configured to receive a hover position of a digital pen over a canvas displayed via a touchscreen, determine whether the hover position of the digital pen is adjacent to a position of the digital ink instance within the canvas; and, in response to the hover position of the digital pen being adjacent to the position of the digital ink instance within the canvas, reveal the boundaries of the digital ink instance within the canvas via a boundary graphic added to the canvas.
  • Other embodiments also provide a method of revealing boundaries of a digital ink instance within an electronic document.
  • the method includes receiving, with an electronic processor, a hover position of a digital pen over a canvas displayed via a touchscreen and determining, with the electronic processor, whether the hover position of the digital pen is adjacent to a position of the digital ink instance within the canvas.
  • the method also includes, in response to the hover position of the digital pen being adjacent to the position of the digital ink instance within the canvas, revealing, with the electronic processor, the boundaries of the digital ink instance within the canvas via a boundary graphic added to the canvas.
  • Another embodiment provides non-transitory computer readable medium storing instructions that, when executed by an electronic processor, perform a set of functions.
  • the set of functions includes receiving a hover position of a digital pen over a canvas displayed via a touchscreen and determining whether the hover position of the digital pen is adjacent to a position of a digital ink instance within the canvas.
  • the set of functions also includes, in response to the hover position of the digital pen being adjacent to the position of the digital ink instance within the canvas, revealing boundaries of the digital ink instance within the canvas via a boundary graphic added to the canvas.
  • FIG. l is a block diagram of a digital inking system according to some embodiments.
  • FIG. 2 is a block diagram of a user device and a digital pen included in the digital inking system of FIG. 1 according to some embodiments.
  • FIG. 3 is a block diagram of components included in a digital inking application provided via the digital inking system of FIG. 1 according to some embodiments.
  • FIG. 4 illustrates a user interface provided by the digital inking application including digital ink invoking a “link” command according to some embodiments.
  • FIG. 5 is a flow chart illustrating a method of linking digital ink instances within a canvas according to some embodiments.
  • FIG. 6A illustrates a user interface provided by the digital inking application including digital ink invoking a “question” command according to some embodiments.
  • FIG. 6B illustrates a user interface provided by the digital inking application including digital ink representing an answer to a digitally inked question according to some embodiments.
  • FIG. 7 is a flow chart illustrating a method of submitting a question via digital ink within a canvas according to some embodiments.
  • FIG. 8 illustrates a user interface provided by the digital inking application including digital ink generated from text according to some embodiments.
  • FIG. 9 is a flow chart illustrating a method of converting text to digital ink within a canvas according to some embodiments.
  • FIG. 10A illustrates a user interface provided by the digital inking application including digital ink invoking an “aggregate” command according to some embodiments.
  • FIG. 10B illustrates a user interface provided by the digital inking application including an aggregation page according to some embodiments.
  • FIG. IOC illustrates a user interface provided by the digital inking application including the aggregation page illustrated in FIG. 10B with an update to a target digital ink instance according to some embodiments.
  • FIG. 10D illustrates a user interface provided by the digital inking application including a source digital ink object updated based on the update to the target digital ink instance illustrated in FIG. IOC according to some embodiments.
  • FIG. 11 is a flow chart illustrating a method of aggregating digital ink instances according to some embodiments.
  • FIG. 12A illustrates a user interface provided by the digital inking application including digital ink within a canvas input via a digital pen operated in a first input mode according to some embodiments.
  • FIG. 12B illustrates a user interface provided by the digital inking application including highlighting added to the digital ink illustrated in FIG. 12A via a digital pen operated in a second input mode according to some embodiments.
  • FIG. 13 is a flow chart illustrating a method of switching a digital pen from a first input mode to a second input mode according to some embodiments.
  • FIG. 14 illustrates a user interface provided by the digital inking application including “beautified” within a canvas digital ink according to some embodiments.
  • FIGS. 15 A, 15B, and 15B illustrates user interfaces provided by the digital inking application illustrating reflowing of digital ink within a canvas according to some embodiments.
  • FIG. 16 illustrates a user interface provided by the digital inking application revealing boundaries of a digital ink instance within a canvas according to some embodiments.
  • FIG. 17 illustrates a user interface provided by the digital inking application including a selected digital ink instance within a canvas according to some embodiments.
  • FIG. 18 is a flow chart illustrating a method of revealing boundaries of a digital ink instance according to some embodiments.
  • non-transitory computer-readable medium comprises all computer-readable media but does not consist of a transitory, propagating signal. Accordingly, non-transitory computer-readable medium may include, for example, a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, a ROM (Read Only Memory), a RAM (Random Access Memory), register memory, a processor cache, or any combination thereof.
  • embodiments described in the present application provide methods and systems for performing digital inking and, in particular, creating and editing digital instance within a canvas using commands invoked in response to received digital ink (i.e., input received while in an inking mode) or other digital pen input, which eliminates or reduces context or mode switching.
  • digital ink i.e., input received while in an inking mode
  • other digital pen input which eliminates or reduces context or mode switching.
  • commands are invoked through more natural gestures than offered by existing inking application. Accordingly, embodiments described herein provide a technical solution that results in improved user interfaces and user experiences within an inking application that also results in reduced or more efficient computing resource usage.
  • the improved user interfaces and user experiences enable users to stay “in-the-flow” and remain focused on the content being created within a canvas and minimizes distractions and inefficiencies associated with mode switching and using various user interface elements (e.g., menus, tools, buttons, moving to different locations within a canvas) by reducing a number of steps (and, therefore, input that needs processing) needed to executed a particular command.
  • user interface elements e.g., menus, tools, buttons, moving to different locations within a canvas
  • the technical effects therefore, show an improvement in the functioning of computers, particularly those providing inking applications with a canvas for receiving digital ink.
  • FIG. 1 is a block diagram of a digital inking system 100 according to some embodiments.
  • the system 100 includes a user computing device 105 and a digital pen 115.
  • the user computing device 105 include a laptop computer, a desktop computer, a tablet computer, a mobile telephone, a personal digital assistant (PDA), a mobile email device, or another electronic device configured to perform digital inking or digital ink editing.
  • the user device 105 receives input from the digital pen 115 as operated or controlled by a user 120.
  • the digital pen 115 can include a passive pen (e.g., a stylus) or an active pen.
  • the user device 105 includes a digital inking application 110.
  • the digital inking application 110 provides, within an electronic document, such as, for example, a digital journal, a canvas configured to receive digital ink via the digital pen 115.
  • the digital inking application 110 is a stand-alone application executed by the user device 105 (an electronic processor included in the user device 105) to provide the digital inking and digital ink editing functionality described herein.
  • the digital inking application 110 may access or otherwise communicate with a digital inking service 130 provided by a server 140, which may provide one or more hosted services.
  • the user device 105 is connected to a network 125 to communicate with the server 140.
  • the network 125 can include wired networks, wireless networks, or a combination thereof that enable communications between the various entities in the system 100.
  • the communication network 125 includes cable networks, the Internet, local area networks (LANs), wide area networks (WAN), mobile telephone networks (MTNs), and other types of networks, possibly used in conjunction with one another, to facilitate communication between the user device 105 and the server 140.
  • LANs local area networks
  • WAN wide area networks
  • MTNs mobile telephone networks
  • the digital inking application 110 installed on the user device 105 may be a general purpose browser application configured to access various services and content over the network 125, including the digital inking service 130 provided by the server 140.
  • the digital inking application 110 installed on the user device 105 may be a dedicated application configured to access the digital inking service 130.
  • the functionality described herein as being performed by the digital inking application may be performed by the user device 105, the server 140, or a combination thereof where functionality may be distributed in various manners.
  • system 100 illustrated in FIG. 1 is provided by way of example and the system 100 may include additional or fewer components and may combine components and divide one or more components into additional components.
  • the system 100 may include any number of user devices 105 or networks 125 and various intermediary devices may exist between a user device 105 and the server 140.
  • multiple servers 140 may be used to provide the digital inking service 130, such as within a cloud computing environment.
  • the user device 105 includes an electronic processor 204, a computer-readable memory 206, and a communication interface 208.
  • the electronic processor 204, memory 206, and communication interface 208 communicate wirelessly, over one or more wired communication channels or busses, or a combination thereof (not illustrated).
  • the memory 206 can include non-transitory memory, such as random access memory, read-only memory, or a combination thereof.
  • the electronic processor 204 can include a microprocessor, a microcontroller, a digital signal processor, or any combination thereof configured to execute instructions stored in the memory 206.
  • the memory 206 can also store data used with and generated by execution of the instructions.
  • the communication interface 208 allows the user device 105 to communicate with external networks and devices, including, for example, the network 125.
  • the communication interface 208 may include a wireless transceiver for communicating with the network 125.
  • the user device 105 may include additional or fewer components than those illustrated in FIG. 2 and may include components in various configurations.
  • the user device 105 includes a plurality of electronic processors, a plurality of memories, a plurality of communication interfaces, or a combination thereof.
  • the user device 105 includes additional input devices, output devices, or a combination thereof.
  • the memory 206 stores the digital inking application 110.
  • the digital inking application 110 (as executed by the electronic processor 204) provides a canvas (e.g., a digital journal) for receiving, displaying, and editing digital ink, including the editing functionality described below.
  • the memory 206 also stores one or more electronic documents 210, such as one or more digital journals generated via the digital inking application 110.
  • the electronic documents 210 may be stored in separate memory included in the user device 105, in a memory external to the user device 105 but accessible by the user device 105 (e.g., via the communication interface 208 or a dedicated port or interface), or in a combination thereof.
  • each of the server 140 may include similar components as the user device 105 and, in particular, may include one or more electronic processors for executing applications or instructions that, when executed, provide the digital inking service 130.
  • the user device 105 also includes (or communicates with) a touchscreen 220.
  • the digital inking application 110 (or a separate application or module) is configured to detect when the digital pen 115 touches or otherwise interacts with the touchscreen 220 (e.g., using capacitive technology). These detected positions can be translated to ink points and digital strokes via the digital inking application 110, which can be used to generate and add digital ink to the canvas.
  • the digital pen 115 includes a passive pen.
  • the digital pen 115 includes an active pen.
  • An active pen as compared to a passive pen, includes electronics or circuitry configured to communicate with a digitizer included in the touchscreen 220.
  • This communication allows the digital inking application 110 to provide additional functionality as compared to when the digital pen 115 includes a passive pen.
  • the digital inking application 110 can provide functionality that uses or is responsive to pressure or touch sensitivity applied by the pen 115, a tilt of the pen 115, a position of the pen 115 (such as a position of the pen even when the pen is hovering over but not touching the touchscreen 220), activation of input mechanisms (e.g., buttons) on the pen 115, use of an eraser tip of the pen 115, or the like.
  • the digital inking application 110 (as executed by the one or more electronic processors 204) provides a canvas within an electronic document (e.g., as part of a digital journal) and is configured to detect digital strokes input via the digital pen 115 and process the digital strokes to generate, add, and edit digital ink within the canvas.
  • an electronic document e.g., as part of a digital journal
  • FIG. 3 is a block diagram of example components of the digital inking application 110 according to some embodiments.
  • the digital inking application 110 may include a screen detection module 301, an activity analysis module 303, a decision module 305, a command invoking engine 307, a command execution engine 309, and, optionally, an inking engine 311.
  • the modules and engines described herein with respect to FIG. 3 represent instructions stored in one or more memories (e.g., the memory 206) executable by the electronic processor 204 to perform the functionality described herein. It should be understood that the functionality described herein can be performed by various configurations of modules, engines, or applications and the particular configuration illustrated in FIG. 3 and described herein is provided merely as one example.
  • functionality described as being performed by multiple modules may be performed by a single module or functionality described as being performed by one module may be distributed between two or more modules.
  • the digital inking service 130 may include similar components as the application 110.
  • the digital inking application 110 may provide additional functionality not described herein.
  • the screen detection module 301 detects activities implemented on the touchscreen 220, which displays an electronic document including canvas for receiving digital ink (as generated by the digital inking application 110).
  • the canvas defines an area of a user interface for receiving and displaying digital ink, and the activities detected by the screen detection module 301 includes interactions between the touchscreen 220 and the digital pen 115.
  • the screen detection module 301 also detects other types of activities or input associated with the digital pen 115, such as a position of the digital pen 115 regardless of whether the digital pen 115 is touching or otherwise interacting with the touchscreen 220.
  • the screen detection module 310 detects activities implemented on the touchscreen 220 via a user’s finger, via a cursor-controlled device, or via other input devices or mechanisms.
  • the activity analysis module 303 analyzes the activities detected by the screen detection module 301 to determine, for example, what type of input was received, where the input was received within the canvas, and other parameters of the input.
  • the activity analysis module 303 can be configured to analyze locations of interactions between the digital pen 115 and the touchscreen (as detected via the screen detection module 301) as a group of ink points defining a group of digital strokes, which may define a particular pen-related operation (also referred to as a “gesture”).
  • the activity analysis module 303 can analyze the activities in the current context or state of the canvas.
  • the decision module 305 decides whether to invoke a pen command based on the activity analysis. For example, the decision module 305 may determine that a pen command is triggered in response to one or more digital strokes input via the digital pen 115 matching a predetermined signifier associated with a particular command.
  • detected pen activity that includes digital ink i.e., detected digital strokes
  • “Wet ink” can be displayed with the canvas but, may, in some embodiments, be removed from the canvas when the digital ink is determined to be a pen command.
  • Digital ink determined not to be a pen command is retained in the canvas and referred to as “dry ink.”
  • the command invoking engine 307 invokes a pen command based on the decision made by the decision module 305. For example, in response to the decision module 305 determining that a pen command has been triggered, the command invoking engine 307 may invoke the associated pen command.
  • the associated pen command may involve changing a mode of the pen 115 (e.g., from a digital inking mode to an edit mode) or directly editing existing digital ink included in the canvas.
  • the content on the touchscreen 220 may be further manipulated as part of the pen command and this manipulation may be monitored and handled by the command execution engine 309.
  • the digital inking application 110 also includes an inking engine 311.
  • the inking engine 311 may be configured to perform general inking activities other than a pen command and, in some embodiments, may be configured to add or render digital ink within the canvas.
  • the screen detection module 301 may be configured to detect digital ink strokes applied on the touchscreen 220 as well as certain no-stroke pen activities.
  • the screen detection module 301 may include a canvas event tracking component that tracks activities applied on the touchscreen 220 (i.e., the canvas) by the digital pen 115.
  • the screen detection module 301 may detect that a user is moving digital content displayed on the touchscreen 220.
  • the screen detection module 301 may detect that the digital pen 115 is touching one point of the touchscreen 220 and being moved (e.g., dragged) in a particular direction.
  • the screen detection module 301 may detect movement of a scroll bar or detect digital strokes (e.g., representing new digital ink content or representing a pen command as described below).
  • the screen detection module 301 may include a location identification component to identify a location (e.g., a most recent location) where an ink point or stroke was applied on the touchscreen 220 (e.g., within the displayed canvas).
  • a location e.g., a most recent location
  • the location identification component may collect location information from a digitizer of the user device 105, where the digitizer may use a capacitance technique to sense the location of the digital pen 115.
  • one or more cameras of the user device 105 can be configured to detect the position of the digital pen 115.
  • the cameras can optionally include a depth camera system that uses a time-of-flight technique, a structured light technique, a stereoscopic technique, or the like, to capture a depth image of the user’s hand, the digital pen 115, or both.
  • an inertial measurement unit (IMU) associated with the digital pen 115 can detect the position of the digital pen 115, and, by extension, the position of the hand that holds the digital pen 115.
  • the IMU can include any combination of one or more accelerometers, gyroscopes, magnetometers, or the like. It should be understood that other techniques for detecting the location of the user’s hand, the digital pen 115, or both may be used.
  • the digital pen 115 may identify location information of the digital pen 115 and the performed gesture.
  • the location information may be compared with the existing ink (i.e., contents such as texts, tables, charts, images, etc.) to determine where the strokes or gestures have been applied with respect to the content.
  • characteristics of the digital ink e.g., shape, size, etc.
  • the digital inking application 110 may access one or more stored rules that associate particular signifiers with particular commands.
  • the signifiers may define particular digital strokes (e.g., marks (symbols, shapes, etc.) or lines) and, optionally, particular timing or sequences of such strokes.
  • detected strokes can be compared to the stored rules to identify whether the detected strokes represent new digital ink to be added to the canvas as content or whether the detected strokes are triggering a particular command.
  • the screen detection module 301 also include a pressure determination component to determine the pressure applied to the surface of the touchscreen 220 from the digital pen 115.
  • the pressure determination component may communicate with the digital pen 115 to collect the pressure information.
  • the digital pen 115 may be an active pen that includes an electronic component for registering the pressure at which the user presses a pen tip against the surface of the touchscreen 220 at each moment.
  • a tip displacement sensor may register the force with which the user applies the digital pen 115 against the surface of the digitizer.
  • the digital pen 115 may also include a transmitter that forwards the pressure information to the user device 105 (e.g., to the pressure determination component in the screen detection module 301).
  • the user device 105 may itself include a sensor that measures and determines how much pressure is being exerted on the touchscreen 220.
  • a sensor that measures and determines how much pressure is being exerted on the touchscreen 220.
  • ZnO zinc oxide
  • TFT thin-film transistor
  • the screen detection module 301 may detect other information during an inking or ink editing process.
  • the digital pen 115 is also configured to transmit other data to the user device 105, such as, for example, a tilt angle of the pen 115 and a hover position of the pen 115, which may be detected or determined by the pen 115 using internal electronics or circuitry.
  • the user device 105 (and the digital inking application 110) can be configured to detect this positional information regarding the digital pen 115 directly.
  • the activity analysis module 303 analyzes detected pen activities. For example, in response to detection of a pen activity, the activity analysis module 303 may analyze location information with respect to existing digital ink in the canvas and other relevant information.
  • a Document Object Model (DOM) or similar data structure defines a current state of an electronic document, such as an electronic journal.
  • the DOM can define one or more pages included in an electronic document, wherein each page includes one or more paragraphs (including one or more lines and one or more words), one or more drawings, or a combination thereof.
  • a paragraph and a drawing are both represented within the DOM structure as including a group of digital strokes, which includes a set of ink points and position information for the set of ink points within the page.
  • a group of digital strokes can be associated with more than one set of position information.
  • digital ink can be duplicated within an electronic document and each copy may reference the same set of ink points but different position information.
  • the position information also includes formatting information, such as color, size, etc. However, in other embodiments, this data can be stored as separate metadata within the DOM.
  • a particular digital ink instance included in an electronic document is defined within the DOM as a set of digital strokes, which include a set of ink points located at a particular position within a page.
  • the DOM may also include other metadata for digital ink, such as, for example, an identifier of other digital ink or content linked with the digital ink, a type of the digital ink, or the like. Accordingly, using the current state of the electronic document currently provided within the canvas, the activity analysis module 303 can determine an intent of detected pen activity.
  • the activity analysis module 303 may determine where a digital stroke is applied by the pen 115. For example, in response receiving digital ink input representing a vertical line, the shape and location of the vertical line may be identified and, based on the identified location information, combined with location the existing digital ink, the activity analysis module 303 may determine whether the vertical line is drawn between words, at the end of a line, at the beginning of a line, at the end of multiple lines, at the beginning of multiple lines, etc.
  • the activity analysis module 303 may determine whether any existing digital content is located within the circle (e.g., overlaps positioning with the circle). For example, the activity analysis module 303 may determine that one or more words, an image, an equation, a table, etc., is located within (e.g., fully or partially within) the circle.
  • the decision module 305 receives the results of the analysis performed by the activity analysis module 303 as input and decides whether to invoke a pen command and, in applicable what pen command to invoke.
  • the activity analysis module, the decision module 305, or both apply various rules or thresholds to detected pen activity to process the activity, such as using predetermined signifiers associated with particular commands as described. These rules or thresholds may be default rules or thresholds or, alternatively, may be user- specific, user device-specific, pen-specific, or a combination thereof.
  • the digital inking application 110 includes a learning component that can be trained to understand what pen activity is associated with a particular pen command.
  • the learning component may detect one or more tilt positions of the pen 115 (in response to a prompt to a user to tilt the pen 115 to provide a reference angle) to learn a threshold amount of tilt the user desires to use for a pen command requiring a predetermined tilt angle.
  • the activity analysis module 303, the decision module 305, or both may collect identifying information from the user device 105, the pen 115, the user, or a combination thereof to apply appropriate rules or thresholds.
  • the identifying information may include a MAC address of the user device 105, biometric or log-in information of the user, a hardware or software identifier of the pen 115, or a combination thereof.
  • a transmitter of the pen 115 may transmit a pen identifier to the user device 105.
  • the decision module 305 in addition to determining whether detected pen activity represents new content or a pen command, the decision module 305 also determines one or more pen commands to invoke. At this point, the decision module 305 may check the context of pen activity applied to the touchscreen 220. For example, the decision module 305 may check previous strokes or the relationship of the activity with respect to existing digital ink within the canvas. For example, the decision module 305 may use a shape recognizer to identify the specific shape of a stroke (e.g., a vertical line, a horizontal line, a circle, a drag, etc.). As one example, the decision module 305 may find that the preceding stroke is a circle stroke drawn around existing words, an image, a table, etc., included in the canvas.
  • a shape recognizer e.g., a shape of a stroke
  • the decision module 305 may find that the preceding stroke is a circle stroke drawn around existing words, an image, a table, etc., included in the canvas.
  • the command invoking engine 307 may be configured to invoke a pen command according to the decision made by the decision module 305.
  • certain digital content may become interactive with the pen 115, and, thus, allow the digital pen to manipulate the relevant content without requiring that the user navigate to particular tools or menus at the designated location of the screen or canvas. That is, once a pen command is invoked, the digital content related to the pen command may become active for ink editing by the digital pen 115.
  • the digital inking application 110 includes multiple command invoking engines, wherein each engine is associated with a particular pen command.
  • one or more command invoking engines can register with the decision module 305, which allows functionality of the digital inking application 110 to be expanded, updated, repaired, or the like.
  • the decision module 305 determines the intent of the detected pen activity and invokes one or more command invoking engines to respond to the detected intent.
  • Each command invoking engine is configured to receive input from the decision module 305 and invoke a particular pen command.
  • a command invoking engine may access or communicate with one or more services to invoke a pen command.
  • the command execution engine 309 may be configured to execute certain ink editing related to an invoked pen command. In some implementations, after a pen command is invoked, different ink editing processes may be then executed through the invoked pen command.
  • the command execution engine 309 may be configured to monitor stroke input via the pen 115 and applying highlighting accordingly.
  • the command execution engine 309 may be combined with the command invoking engine 307.
  • a command execution engine 309 may not be used for particular pen commands. For example, in some embodiments, when a particular pen command is invoked, no further monitoring of input is needed as the pen command is invoked and subsequent input is processed as new input as described above.
  • the digital inking application 110 (and the components described above with respect to FIG. 3) is configured to provide digital ink editing functionality triggered based on received digital ink or other pen input or activity as compared to requiring that the user navigate to a particular menu or tool, switch the digital pen 115 to a different mode, or the like. Examples of this digital ink editing functionality are provided below. It should be understood that the digital inking application 110 may provide any one or more of the below examples in various combinations.
  • the default reading order for notes or other content in a journal is often defined by the physical location of content within a page and the order of the pages. However, in many cases, this is not the optimal way to traverse notes. For example, users may take notes over time in a sequential manner. In this situation, if a user takes notes on one day regarding a particular topic and a few days later takes additional notes on the same topic, the notes in the digital journal may be separated by unrelated notes and may even be separated by one or more pages.
  • a digital journal may more easily allow a user to reorder notes, such as by selecting and repositioning individual notes. This manual repositioning, however, can be time-consuming and may result in loss of data (e.g., as the digital journal is no longer in date order).
  • the digital inking application 110 can be configured to establish a link between digital ink instances (i.e., dry ink instances) in response to receiving digital ink drawing a connecting line between the digital ink instances.
  • the digital inking application 110 in response to receiving digital ink matching a predetermined signifier (or a sequence of signifiers), is configured to invoke an “insert link” command that updates metadata of one or both of the digital ink instances with an identifier of the other instance (e.g., a hyperlink), which makes the digital ink instances active or otherwise selectable to navigate through linked content within a canvas, such as a digital journal.
  • a user can use natural gestures provided through digital ink added to the canvas to link content within a journal without having to navigate or select an insertion tool from a menu and then define or create the desired link within a dedicated user interface or prompt, which creates inefficiencies from both a user perspective and a computing resources perspective.
  • FIG. 4 illustrates a user interface 800 provided via the digital inking application 110.
  • the user interface 800 includes two pages 815 and 820 of an electronic document (e.g., a digital journal) within a canvas 810.
  • the page 820 includes a first digital ink instance 825 including the letter “w,” and the page 815 includes a second digital ink instance 830 including the words “Elaborating on w.”
  • a user can mark and connect the first digital ink instance 825 and the second digital ink instance 830 using digital ink including or matching a predetermined signifier.
  • the predetermined signifier includes a mark 835 having a predetermined shape or format (e.g., circling all or a predetermined portion of the first digital ink instance 825, adding a bounding box around all or a predetermined portion of the first digital ink instance 825, or the like acting as a first signifier) and a line 840 (acting as a second signifier) extending from the mark 835 that connects the marked first digital ink instance 825 and the second digital ink instance 830.
  • the user marks the second digital ink instance 830 using an end of the connecting line 840.
  • the user marks the second digital ink instance 830 by marking the instance 830 similar to how the user marked the first digital ink instance 825 (e.g., by circling all or a portion of the second digital ink instance 830).
  • the predetermined signifier includes the predetermined mark 835 and the connecting line 840 within a continuous digital stroke.
  • the predetermined signifier may be received as a set of one or more discrete strokes.
  • the digital inking application 110 invokes the “insert link” command, which causes the digital inking application 110 to determine the first and second digital ink instances 825 and 830 based on the position of the analyzed digital strokes (e.g., the mark 835 and the connecting line 840) within the canvas 810 and update metadata of the first digital ink instance 825, the second digital ink instance 830, or both with an identifier of the other instance.
  • the “insert link” command causes the digital inking application 110 to determine the first and second digital ink instances 825 and 830 based on the position of the analyzed digital strokes (e.g., the mark 835 and the connecting line 840) within the canvas 810 and update metadata of the first digital ink instance 825, the second digital ink instance 830, or both with an identifier of the other instance.
  • the digital inking application 110 also removes the digital ink (i.e., the mark 835 and the connecting line 840) that triggered the “insert link” command from the canvas 810 (as this digital ink was determined to be pen command rather than dry ink).
  • the first digital ink instance 825 can be active or selectable within the canvas 810 wherein, in response to receiving a selection or tap of the first digital ink instance 825, the digital inking application 110 navigates to or otherwise scrolls the canvas such that the second digital ink instance 830 is included within the user interface 800.
  • the link is bi-directional such that the second digital ink instance 830 can also be active or selectable to navigate to the first digital ink instance 825.
  • the first and second digital ink instances 825 and 830 can include any type of content, e.g., words, paragraphs, drawings, images, symbols, etc.
  • first and second digital ink instances 825 and 830 may be included on the same page of the digital journal or may be separated by multiple pages. Also, in some embodiments, the first and second digital ink instances may be included in separate journals (included within the same canvases or separate canvases). Also, in some embodiments, active or selectable digital ink may be provided within the canvas in a particular format or associated with an icon or other mark that alerts the user that the ink is active and can be selected.
  • FIG. 5 is a flow chart illustrating a method 900 of linking digital ink instances.
  • the method 900 is described as being performed by the digital inking application 110 (and the components described above with respect to FIG. 3) as executed by the electronic processor 204 of the user device 105.
  • the method 900 or portions thereof may be performed by the digital inking service 130 provided via the server 140.
  • the method 900 includes receiving detected interactions between the touchscreen 220 and the digital pen 115 representing one or more digital strokes within a canvas displayed via the touchscreen 220 (at block 905).
  • the digital inking application 110 analyzes the digital strokes to detect whether the digital strokes match a predetermined signifier associated with an “insert link” command (at block 910).
  • digital strokes match a particular predetermined signifier when the digital strokes include one or more strokes matching a shape, size, pattern, or timing of the predetermined signifier.
  • the predetermined signifier associated with an “insert link” command includes a predetermined mark, such as a circle, a rectangle, a triangle, or another predetermined shape, and a line extending from the predetermined mark, wherein, optionally, the predetermined mark and the line are included in a continuous stroke.
  • the digital inking application 110 invokes an “insert link” command and determines a first digital ink instance and a second digital ink instance within the canvas based on a position of the digital strokes (at block 915).
  • the digital inking application 110 can use the state of the electronic document provided within the canvas (as defined by the DOM) to determine existing digital ink positioned adjacent to the predetermined mark.
  • the predetermined mark is a circle or other shape drawn around existing digital ink
  • digital ink positioned adjacent to the predetermined mark can include digital ink including a threshold amount of digital strokes or ink points positioned in an area of the canvas within the predetermined mark or otherwise overlapping with the predetermined mark.
  • digital ink positioned adjacent to the predetermined mark includes digital ink including one or more digital strokes or ink points positioned within a predetermined distance of digital strokes or ink points of the predetermined mark.
  • the digital inking application 110 may determine existing digital ink having ink points closest to ink points of the star.
  • digital ink “adjacent” to a predetermined signifier is determined by comparing positions of existing digital ink with positions of the predetermined signifier and identifying digital ink including (e.g., the most) digital strokes or ink points closest to, contained within, or overlapping with positions of the predetermined signifier.
  • identifying digital ink including (e.g., the most) digital strokes or ink points closest to, contained within, or overlapping with positions of the predetermined signifier.
  • the entire page and, optionally, all digital ink instances within the page can be set as the first digital ink instance.
  • the set of instances can be set as the first digital ink instance or the digital ink instance closest to or having the most overlapping position with the predetermined mark may be set as the first digital ink instance.
  • the digital inking application 110 can use the state of the electronic document provided within the canvas to determine existing digital ink positioned adjacent to an ending position of the line extending from the predetermined mark. Alternatively, if another predetermined mark is used at the end of the line, the digital inking application 110 can use a position of this additional predetermined mark to determine the second digital ink instance as described above for the first digital ink instance.
  • the entire page and, optionally, all digital ink instances within the page can be set as the second digital ink instance.
  • the set of instances can be set as the second digital ink instance or the digital ink instance being closest or having the largest overlapping position with the end of the connecting line can set as the second digital ink instance.
  • the digital inking application 110 After determining the first and second digital ink instances (at block 915), the digital inking application 110 updates metadata of the first digital ink instance to include an identifier of the second ink instance (at block 920).
  • the identifier includes a hyperlink or other type of pointer or link to the second ink instance.
  • a bi-directional link is established between the first and second digital ink instances by also updating the metadata of the second digital ink instance to include an identifier of the first ink instance.
  • digital ink instances including a link to other content such as another digital ink instance, is considered “active” and marked or formatted within the canvas to distinguish the digital ink instance from inactive content.
  • active digital ink instances may be provided with underlining, in a particular color, or with a particular icon or animation that designates the instance as being active. As illustrated in FIG.
  • the digital inking application 110 in response to receiving a selection of the first digital ink instance, such as, for example, a click or tap of the first digital ink instance, hovering the digital pen 115 above the first digital ink instance, or the like (at block 925), navigates the journal to the linked second digital ink instance within the canvas (i.e., updating the canvas to include the second digital ink instance).
  • the link (established through the predetermined mark and associated connecting line drawn via digital ink) enables a user to easily navigate digital ink content within an electronic document, such as a digital journal, which may include many pages and disjointed content.
  • These links can also be used to leverage organization or understanding of the content, such as by using the links as part of performing keyword searching within the document.
  • links usually point to navigation constructs (e.g., pages).
  • the underlying expectation is that the content is static and the link might be “broken” if the user rearranges or adds content.
  • the link described herein is between digital ink instances, which can be defined at various levels of granularity, the link points to user content rather than navigation constructs and the digital inking application 110 is responsible for choosing the right navigation action when a link is activated. Accordingly, as user-generated content evolves, previously generated links still work correct. For example, if a user creates a link to a paragraph on page 1 of a digital journal but subsequently extracts words from this paragraph to page 2 and eventually moves the entire paragraph to page 3, the link will still work as the user intended and, in particular, the journal will navigate to page 3 when the link is selected.
  • the link is established through digital ink and, in particular, natural gestures (i.e., drawing a connecting line)
  • the user is not required to learn or know any mark-up languages, retrieve any type of URL or identifier of content as a separate step (e.g., a page name, a page URL, or the like), remember any identifier of target or linked content, or invoke any type of user interface or process that prompts the user to choose target or linked content.
  • the user only needs to draw a visual representation of the link within the canvas, such as by circling content that will become a link and pointing to the target of the link via a connecting line.
  • the user s “flow” is not broken, and the user can naturally express the desired source and target for a link.
  • a user While working within an electronic document including a canvas for digital ink (e.g., an electronic journal), a user may have a question or need supporting information from one or more external sources. Traditionally, to answer this question, the user must navigate out of the electronic document and access a search engine or other on-line or local service or resource to input a question and receive an answer. The user then has to return to the electronic document and manually add the retrieved answer to the electronic document. This navigation outside of the electronic document clearly creates inefficiencies and also relies on the user to correctly remember or understand the answer and add the answer to the electronic document. Furthermore, even if a user can copy the answer and paste the answer into the electronic document, this pasting modifies the organization and structure of the electronic document that may otherwise primarily include digital ink.
  • the digital inking application 110 in response to detecting digital ink as being a pen command associated with a “question” command, can be configured to extract text from digital ink marked as a question within the canvas, send a query to an answer service, such as the Bing® Instant Answer Service provided by Microsoft Corporation, receive a result from the answer service, and add the result to the canvas as digital ink (not text), which allows the result to be moved and modified as other digital ink within he canvas.
  • an answer service such as the Bing® Instant Answer Service provided by Microsoft Corporation
  • FIG. 6 A illustrates a user interface 1000 provided via the digital inking application 110.
  • the user interface 1000 includes a portion of an electronic document (e.g., a digital journal) within a canvas 1002.
  • a user may ask a question in digital ink (e.g., a digitally inked question 1005) and receive an answer in digital ink within the canvas 1002.
  • a question in digital ink e.g., a digitally inked question 1005
  • the user can mark the digitally inked question 1005 via digital ink including a predetermined signifier 1015 associated with a “question” command.
  • the predetermined signifier 1015 for invoking a “question” command can include a predetermined shape, such as, for example, a circle, a triangle, a rectangle, a diamond, or another predetermined shape drawn adjacent to the digitally inked question 1005 (e.g., within a predetermined distance from the digitally inked question 1005 or adjoining or overlapping the digitally inked question 1005).
  • the predetermined signifier 1015 for invoking a “question’ command can include a predetermined character or predetermined string of characters, such as for, example, a stylized question mark, the word “ask,” the word “question,” the abbreviation “ques,” or the like.
  • the predetermined signifier 1015 includes a combination of at least two of a predetermined shape, a predetermined character, and a predetermined string of characters.
  • the digitally inked question 1005 is dry ink within the canvas 1002.
  • an answer 1020 (e.g., in digital ink form) can be added to the canvas 1002, such as in place of the predetermined signifier 1015.
  • FIG. 7 is a flow chart illustrating a method 1100 of submitting a question via digital ink according to some embodiments.
  • the method 1100 is described as being performed by the digital inking application 110 (and the components described above with respect to FIG. 3) as executed by the electronic processor 204 of the user device 105.
  • the method 1100 or portions thereof may be performed by the digital inking service 130 provided via the server 140.
  • the method 1100 includes receiving detected interactions between the touchscreen 220 and the digital pen 115 representing one or more digital strokes within a canvas displayed via the touchscreen 220 (at block 1105).
  • the digital inking application 110 analyzes the digital strokes to detect whether the digital strokes match a predetermined signifier associated with a “question” command (at block 1110).
  • the predetermined signifier for invoking a “question” command includes a predetermined mark, such as a diamond, circle, a rectangle, a triangle, or another predetermined shape.
  • the digital inking application 110 invokes the “question” command and determines a digital ink instance within the canvas based on a position of the digital strokes (at block 1115). For example, to determine the digital ink instance, the digital inking application 110 can use the state of the electronic document included in the canvas (as defined by the DOM) to determine existing digital ink (i.e., dry ink) positioned adjacent to the predetermined mark.
  • digital ink positioned adjacent to the predetermined mark includes a digital ink instance including one or more digital strokes or ink points positioned within a predetermined distance of digital strokes or ink points of the predetermined mark.
  • the digital inking application 110 may be configured to identify digital ink adjacent to the predetermined mark in a particular direction (e.g., following or to the right of the digital ink).
  • the predetermined mark can be placed adjacent to the question in any direction within the canvas 1002.
  • the digital inking application 110 After determining the digital ink instance adjacent to the predetermined mark, the digital inking application 110 extracts text from the digital ink instance (i.e., converts the contents of the digital ink instance to text using handwriting recognition technology)
  • the answer service includes the Bing® Instant Answer service provided by Microsoft Corporation. However, other similar answer or search engine services may be used. Also, in some embodiments, the answer service may be included as part of the digital inking application 110 or stored locally on the user device 105.
  • the digital inking application 110 In response to receiving a text-based result to the submitted text query from the answer service, the digital inking application 110 converts the text-based result to digital ink (e.g., as described below) and adds the generated digital ink to the canvas as a new digital ink instance (at block 1130). In some embodiments, the digital inking application 110 positions the new digital ink adjacent to the digital ink forming the question or otherwise aligns the new digital ink with the original digital ink forming the question in at least one direction (e.g., vertically, horizontally, or a combination thereof).
  • digital inking application 110 positions the new digital ink adjacent to the digital ink forming the question or otherwise aligns the new digital ink with the original digital ink forming the question in at least one direction (e.g., vertically, horizontally, or a combination thereof).
  • the digital inking application 110 positions the new digital ink in place of the predetermined mark that triggered the “question” command (e.g., as this digital ink is wet ink representing a pen command as described above that is removed from the canvas). It still other embodiments, the answer (in digital ink form) can be combined with the question to form a new or updated digital ink instance within the canvas.
  • the digital inking application 110 is also configured to initially convert the text to a particular font, a particular font size, or a combination thereof.
  • the digital inking application 110 can be configured to initially convert the text to a size matching a size (e.g., height of characters within the digitally inked question).
  • the digital inking application 110 can be configured to initially convert the text to a particular font, such as script-like font or a font that matches characteristics of the digitally inked question).
  • the digital inking application 110 may also apply formatting to the text (e.g., bolding, underlining, italics, etc.) to match the digitally inked question.
  • Performing one or more of these initial conversions causes the resulting generated text to better match the digitally inked question.
  • the digital inking application 110 can perform this conversion or formatting on the generated ink rather than as pre-processing on the text-based answer.
  • the new digital ink is added to the DOM of the electronic document and is represented like any other digital ink within the canvas (e.g., a collection of ink points or strokes at particular locations).
  • the new digital ink (representing the answer to the digitally inked question) can be manipulated in the same manner as digital ink originally inked using the digital pen 115.
  • This manipulation includes, for example, erasing part of a character of the digital ink, manipulating the location of individual characters, manipulating an orientation of the digital ink, etc.
  • a user can draw the predetermined mark associated with the “question” command adjacent to the new digital ink representing the answer to submit the answer (or a version thereof) as yet another question.
  • the digital inking application 110 can be configured to allow a user to ask a question with digital ink directly in the canvas by drawing a specific symbol to mark the digital ink as a question to be submitted to an answer service. Furthermore, rather than receiving the result in text, the digital inking application 110 can provide the result as ink, the user can position and otherwise modify the answer as needed within the electronic document.
  • This functionality gives the user flexibility when organizing content within the electronic document and preserves a visual style of the electronic document. For example, by preserving the original query or question in digital ink and providing the resulting answer in digital ink, an overall structure or visual style of the electronic document is maintained consistent.
  • the question illustrated in FIGS. 6A and 6B is provided as one example question and the digital inking application 110 may be configured to provide answers for other types of questions and, in particular, is not limited to answer numerical or math questions.
  • the digital inking application 110 (through the answer service) may be configured to answer general questions with discrete answers, such as, for example, “How tall is the Space Needle?” and “What time is it in Paris?”
  • the answer may be associated with a link (e.g., a hyperlink), wherein, in response to receiving a selection the answer, a source of the answer can be provided to the user (e.g., within the digital inking application 110 or a separate application).
  • the digital inking application 110 may output an error message, which may be provided as text or digital ink.
  • an inking application wants to programmatically provide text to the user, the application uses a script-type font for the text, which is overlaid on the canvas as a separate object. While the font may look similar to handwritten digital ink, the user is unable to interact with the text using the same controls available for digital ink because the text is not actually digital ink.
  • the digital inking application 110 can be configured to generate ink from a text font, wherein the generated ink can be added to the canvas as if the user had drawn the ink using the digital pen 115. Generating ink from the text allows the ink to be manipulated like other ink (e.g., erasing part of a character, changing the color, moving the ink to an arbitrary position within the canvas, or the like). As described herein, the digital inking application 110 can convert text to digital ink in response to activation of a “convert” command, which a user may select or invoke in various ways, including, for example, using digital ink provided within the canvas as a pen command.
  • a “convert” command which a user may select or invoke in various ways, including, for example, using digital ink provided within the canvas as a pen command.
  • the “convert” command can also be automatically invoked by the digital inking application 110, such as, for example, as part of executing other commands like the “question” command described above or as part of providing programmatically-generated feedback to a user, such as by converting programmatically- generated text associated with an error message, a notification, or the like to digital ink for presentation to a user.
  • FIG. 8 illustrates a user interface 1200 provided via the digital inking application 110.
  • the user interface 1200 includes a portion of an electronic document (e.g., a digital journal) within a canvas 1202.
  • the canvas 1002 includes text 1205 which is provided as a separate object overlaid on the top of the canvas 1002.
  • the text 1205 may be copied into the canvas 1202 from another application or data source. Alternatively or in addition, the text 1205 may be typed or otherwise entered into the canvas, such as via a keyboard.
  • a user can use the digital pen 115 to draw a predetermined mark 1215 adjacent to the text 1205 (e.g., within a predetermined distance from the text 1205, around the text 1205, or the like).
  • the predetermined mark 1215 may be a circle, a square, a triangle, or other predetermined shape.
  • the digital inking application 110 identifies the predetermined mark 1215 as invoking a “convert” command and generates digital ink 1220 based on the text 1205 associated with the predetermined mark 1215.
  • the generated digital ink 1220 is added to the canvas 1202 including the original text 1205.
  • the original text 1205 may be retained within the canvas 1202 or, in some embodiments, may be removed.
  • FIG. 9 is a flowchart illustrating a method 1300 of converting text to digital ink within a canvas according to some embodiments.
  • the method 1300 is described as being performed by the digital inking application 110 (and the components described above with respect to FIG. 3) as executed by the electronic processor 204 of the user device 105.
  • the method 1300 or portions thereof may be performed by the digital inking service 130 provided via the server 140.
  • the method 1300 includes receiving detected interactions between the touchscreen 220 and the digital pen 115 representing one or more digital strokes within a canvas included in an electronic document displayed via the touchscreen 220 (at block 1305).
  • the digital inking application 110 analyzes the digital strokes to detect whether the digital strokes matches a predetermined signifier for invoking a “convert” command (at block 1310).
  • the predetermined signifier for invoking a “convert” command includes a predetermined mark, such as a diamond, circle, a rectangle, a triangle, or another predetermined shape, which may be added adjacent to (e.g., around) text within the canvas.
  • the digital inking application 110 invokes the “convert” command and determines a text object within the electronic document based on a position of the digital strokes (at block 1315).
  • the digital inking application may identify this text object by identifying a text object within a predetermined distance from the predetermined mark or at least partially within or overlapping with the predetermined mark.
  • the digital inking application 110 converts the text to digital ink (at block 1320).
  • the digital inking application 110 can generate the digital ink by generating or rendering an image of the text and then tracing characters within the image to produce ink points that are orientated in the same manner as the characters of the text.
  • the characters within the image can be traced by tracing an outline of each character included in the text, a line established down the center of each stroke of a character, filling in an area represented by the characters (e.g., creating ink points that fill or otherwise overlap with the strokes establish each character within the image), or the like.
  • the ink points can be joined to form digital strokes representing the characters of the text and the ink points and digital strokes can be stored in the DOM as another digital ink instance.
  • the digital inking application 110 can call a text Tenderer service or function (e.g., via an application programming interface).
  • the text Tenderer is configured to convert characters in a particular font into a set of calls to draw lines from one point to another. This type of text Tenderer decouples the understanding of what a "font" is from knowledge about how to draw the lines representing text in the font, as the lines may need to be rendered on a screen, saved to a bitmap in memory, or printed onto physical paper.
  • input to the text Tenderer can include the one or more characters to be rendered (e.g., a string of one or more characters), an identifier of a font, and an identifier of a font size.
  • the text Tenderer returns a set of lines, which can be represented as a series of Bezier control points.
  • the digital inking application 110 converts the output from the text Tenderer to a series of coordinates (e.g., x and y coordinates), which the digital ink application 110 saves to memory (rather than rendering the lines on a screen).
  • the digital ink application 110 converts the coordinates into ink points and ink strokes (e.g., using one or more application programming interfaces).
  • the "rendered" ink points and associated ink strokes may be defined relative to a default coordinate (e.g., (0,0)) and, thus, the ink points and ink strokes may not have positional information.
  • the digital inking application 110 after generating the ink strokes, can be configured to apply an additional transform to move the ink points and ink strokes to the location within the canvas where the digital ink should be provided.
  • This transformation can include adding fixed values for X and Y to each ink point to move the ink point to the desired position, which, as noted above, may be determined based on original position of the text, other digital ink within the canvas (e.g., a digitally-inked question), or the like.
  • the strokes are added to the DOM just like any other digital ink directly input via a user with the digital pen 115. Accordingly, from the viewpoint of the digital inking application 110, there's no difference between ink that gets generated from text as compared to ink that was directly input via the digital pen 115. In particular, after the ink points and ink strokes are added to the DOM, the user can do anything with the digital ink (and the strokes included therein) that the user could do with directly-input digital ink.
  • the digital inking application 110 is configured to hook into the rendering process for text before the text gets rendered to a bitmap and then "render" the text as digital ink.
  • the digital inking application 110 After generating the digital ink, the digital inking application 110 adds the digital ink to the canvas (e.g., adds the digital ink instance to the DOM, including position information based on where the new ink is added within a page of the electronic document) (at block 1325). In some embodiments, the digital inking application 110 replaces the text with the generated digital ink (i.e., removes the text from the canvas and adds the generated digital ink within a page in place of the removed text at the same position as the original text). However, in other embodiments, the digital inking application 110 retains the text within the canvas, which may allow the user to decide whether to remove or retain the text or check an accuracy of the generated digital ink.
  • the digital inking application 110 retains the text within the canvas, which may allow the user to decide whether to remove or retain the text or check an accuracy of the generated digital ink.
  • the digital inking application 110 may position the generated digital ink to match the original position of text within the canvas.
  • the digital inking application 110 may be configured to perform various pre-processing of the text, such as by converting the text to a particular font or font size (e.g., based on characteristics of existing digital ink within the electronic document).
  • generating ink from a text font allows users to interact with the “text” using the same toolset as when they are inking with the digital pen 115.
  • Existing solutions either require a different toolset to be used or disallow any interaction altogether.
  • the user has the capability to personalize the way the digital inking application 110 generates ink, such as by installing new fonts onto the user device 105 without the digital inking application needing any corresponding updates.
  • the text conversion described above can be used in various contexts and even by other commands invoked within the digital inking application 110, such as with the question and answer functionality described above with respect to FIGS. 6A, 6B, and 7.
  • the “convert” command can be invoked by the digital inking application 110 as a way to display various text generated by the computer.
  • the digital inking application 110 can automatically invoke the “convert” command as part of signaling an error to the user or providing another type of notification to a user, when synchronizing a list of items (e.g., to-do items) in an electronic journal, wherein the items were authored in a different program (using, for example, a keyboard), and stored as text, as part of managing electronic messages where one user is writing in digital ink but another user involved in the conversation is writing in text (e.g., using a keyboard), and the like.
  • the “convert” command described above can be used any time the digital inking application needs to provide feedback to the user. Accordingly, the “convert” command can be used to convert existing text included in an electronic document to digital ink or convert text to digital ink before the text is included in the electronic document. Thus, the detection of a pen command as described above with respect to FIGS. 8 and 9 (or other forms of user input or selections) is not required to invoke the “convert” command.
  • the digital inking application 110 can be configured to update metadata of a digital ink instance associated with the pen command as a particular type of ink object, duplicate the digital ink instance, and add the duplicated digital ink instance to a summary or aggregate page (or separate document) associated with other ink objects of the same type.
  • FIGS. 10A through 10D illustrate user interfaces 1400 provided via the digital inking application 110.
  • the user interfaces 1400 includes a portion of an electronic document (e.g., a digital journal) within a canvas 1402. As illustrated in FIG.
  • the canvas 1402 includes a digital ink instance 1405 within a page 1407 of an electronic document.
  • the digital ink 1405 is marked by a user as a to-do type object (e.g., using a predetermined signifier 1420 associated with a “aggregate” or “to-do” command, such as a circle positioned adjacent to, for example to the left or before, the digital ink instance 1405), and, in response to this marking, the digital inking application 110 creates a duplicated digital ink instance 1410 and adds the duplicated digital ink instance 1410 to an aggregate page 1415 (which may be in the same electronic document or a separate electronic document) (see FIG. 10B).
  • a predetermined signifier 1420 associated with a “aggregate” or “to-do” command, such as a circle positioned adjacent to, for example to the left or before, the digital ink instance 1405
  • the digital inking application 110 creates a duplicated digital ink instance 1410 and adds the duplicated digital ink instance
  • a link is maintained between the original digital ink instance 1405 and the duplicated digital ink instance 1410, such that changes made to the digital ink instance 1405 (such as marking the instance 1405 as complete, such as via the digital pen 115) are automatically applied to the duplicated digital ink instance 1410.
  • the link may be established by having both the digital ink instance 1405 (the source object) and the duplicated digital ink instance 1410 (as stored within the DOM) refer to the same set of ink points or digital strokes.
  • FIG. 11 is a flowchart illustrating a method 1500 of aggregating digital ink instances of a particular type on a particular page (or other portion) of an electronic document according to some embodiments.
  • the method 1500 is described as being performed by the digital inking application 110 (and the components described above with respect to FIG. 3) as executed by the electronic processor 204 of the user device 105.
  • the method 1500 or portions thereof may be performed by the digital inking service 130 provided via the server 140.
  • the method 1500 includes receiving detected interactions between the touchscreen 220 and the digital pen 115 representing one or more digital strokes within a canvas included in an electronic document displayed via the touchscreen 220 (at block 1505).
  • the digital inking application 110 analyzes the digital strokes to detect whether the digital strokes match a predetermined signifier for invoking an “aggregate” command (at block 1510).
  • the predetermined signifier for invoking a “aggregate” command includes a predetermined mark, such as a diamond, circle, a rectangle, a triangle, or another predetermined shape, which may be added adjacent to (e.g., to the left of) digital ink within a canvas.
  • the digital inking application 110 invokes the “aggregate” command and determines a digital ink instance within the canvas based on a position of the digital strokes (at block 1515).
  • the digital inking application 110 may identify an ink instance as being adjacent to the predetermined mark associated with the “aggregate” command when the ink instance is within a predetermined distance from the predetermined mark and, in some embodiments, in a particular direction, such as to the left.
  • the digital inking application 110 may update metadata of the source digital ink instance to indicate a particular type associated with the predetermined mark (e.g., a to-do, a list item, a reminder, or the like).
  • a particular type associated with the predetermined mark e.g., a to-do, a list item, a reminder, or the like.
  • the particular signifier included in the detected digital strokes establish the type. For example, a circle symbol may designate “to-do” type objects while a square or rectangle symbol may designate “list” type objects.
  • the digital inking application 110 also creates a duplicated digital ink instance (representing a target digital ink instance) that includes the same content as the source digital ink instance (at block 1520) and adds the duplicated digital ink instance to an aggregation page or portion of the electronic document (or a separate electronic document) associated with the type of the source digital ink instance that aggregates digital ink instances of the specified type (at block 1525).
  • the digital inking application 110 establishes a link between the source digital ink instance and the target digital ink instance. As noted above, this link may be established by having both the source digital ink instance and the duplicated target digital ink instance reference the same set of ink points, the same set of digital strokes, or both within the DOM. Thus, in response to receiving an update to the target digital ink instance (at block 1530), the digital inking application 110 can automatically apply the same update to the source digital ink instance (and vice versa).
  • ink objects e.g., of a particular type or otherwise marked by a user or automatically recognized
  • a user can more easily navigate and track related ink objects, such as to-dos, lists, or reminders. Since the objects on these aggregation pages maintain their relationship with the source objects, when a user makes changes on an aggregation page, the same changes are reflected on the source page (and vice versa).
  • the to-do when a user creates a to-do on a page in a digital journal, the to-do also appears on an aggregation page for “to-do” objects, which accumulates ink objects marked or identified as to-dos (e.g., via a predetermined signifier as described above).
  • the user updates the to-do on the aggregation page e.g., marks the to-do as being complete
  • the original or source to-do is also marked as being complete (e.g., to avoid any confusion due to conflicting information within the journal).
  • the digital inking application 110 can be configured to organize objects and create aggregated lists or pages without requiring that the objects (such as digital ink) be initially converted to text.
  • objects such as digital ink
  • converting digital ink to text removes the personalization of how the user originally created the content and prevents the user rom being able to edit the content with the same tools (inking tool) originally used to create the content.
  • creating a mere copy of content without establishing a link between the copies can cause the user to lose track of content as edits made to the original content is not reflected in the copy and vice versa.
  • the digital inking application 110 can also be configured to not only create exact duplicate copies of digital ink in different locations but also modify how the duplicate copy is presented within the canvas.
  • a digital ink instance may represent a task to be the performed and the user may write the task over multiple lines due to constraints on the dimensions of the page and available space.
  • lines of characters within a digital ink instance are a single unit containing multiple words across multiple lines, the digital inking application 110 can not only provide a duplicate ink object at a different location but can also lay out the duplicate ink object in a way that expands or contracts to better fit the space at the new location.
  • the digital inking application can be configured to create semantic objects (e.g., semantic to-do objects) that reference an ink instance.
  • semantic objects e.g., semantic to-do objects
  • the ink instances referenced by the semantic objects are extracted and rendered.
  • semantic objects work as virtual groups and have references to the ink objects and do not need to have a one-to-one mapping to an ink object.
  • one or more words in a line or a plurality of continuous lines can be recognized as entities (phone number, address, people name, etc.) or two ink lines in a paragraph including four ink lines can be recognized as one to-do item.
  • a page in an electronic document is analyzed to extract a document structure.
  • this structure is extracted as a tree with a hierarchy (e.g., page->writing regions (or drawings) -> paragraphs -> lines -> words).
  • the hierarchy can include a page, which is broken down into one or more writing regions or drawings, and the writing regions are further broken down into one or more paragraphs, wherein each paragraph is broken down into one or more lines where each line has one or more words.
  • Ink objects at the same level within the tree are mutually exclusive in terms of the strokes they own or contain.
  • This hierarchy can be referred to as a layout tree.
  • semantic objects are recognized, such as recognizing a two lines of ink as a to-do item (e.g., where the first word in the first line includes a predetermined signifier or mark as described above (e.g., a bullet represented as a circle).
  • the recognized semantic objects are defined via a semantic tree that is created and saved with the layout tree in the DOM.
  • the semantic tree for an aggregation page for to-do items can include the following: semanticPage (which references the source page) -> to-dos -> bullet (reference to the ink word referencing the bullet or circle) and lines (reference to the ink lines).
  • semanticPage entries in the semantic tree are iterated and all the to-dos are extracted.
  • the referenced ink lines can be found in the layout tree and used in the rendering.
  • semantic trees are also updated as needed. For example, removing a leading word that was previously recognized as a bullet marking a to-do item will cause the to-do item to be removed from the semantic tree.
  • a stroke overlapping with the bullet can be rendered on the aggregation page (e.g., as a checkmark or filling in the bullet).
  • predetermined marks may be established to identify particular types of digital ink instances (e.g., a circle, a square or rectangle, a triangle, or the like) and, in some embodiments, the predetermined marks may be required to be adjacent to associated digital ink in a predetermined direction. Also, in some embodiments, rather than having a user mark or otherwise identify digital objects of a particular type, the digital inking application 110 may be configured to automatically identify (or learn how to identify) such objects based on context, such as the contents of the ink, how the ink is written or structured, headings or subheadings included on the page with the ink, or the like.
  • aggregation could be performed passively wherein the digital ink application 110 is configured to generate and populate a summary or aggregation page in response to detecting, for example, using heuristics or a machine learning module, one or more predetermined interest items (e.g., a list, a to-do item, or the like).
  • predetermined interest items e.g., a list, a to-do item, or the like.
  • the digital inking application 110 may provide a highlighting tool that enables a user to highlight content, including digital ink, within the canvas using the digital pen 115.
  • the user may be required to access a menu within the application 110 and select a highlighting tool.
  • the user similarly has to access a menu to switch back to the pen tool. This context switching wastes time and processing resources and disrupts the user’s flow.
  • the digital inking application 110 can be configured to automatically switch to a highlighter mode in response to detecting a particular tilt of the digital pen 115.
  • FIGS. 12A and 12B illustrate user interfaces 1600 provided via the digital inking application 110.
  • the user interfaces 1600 includes a portion of an electronic document (e.g., a digital journal) within a canvas 1602. As illustrated in FIG.
  • the canvas 1602 includes a digital ink instance 1605 within an electronic document.
  • the digital in instance 1605 was created via the digital pen 115.
  • a user can change the tilt of the digital pen 115.
  • the user can use the digital pen 115 to highlight existing digital ink (i.e., dry ink) within the canvas 1602.
  • FIG. 13 is a flow chart illustrating a method 1700 of activating a highlighting mode of a digital pen 115 according to some embodiments.
  • the method 1700 is described as being performed by the digital inking application 110 (and the components described above with respect to FIG. 3) as executed by the electronic processor 204 of the user device 105. However, as noted above, in some embodiments, the method 1700 or portions thereof may be performed by the digital inking service 130 provided via the server 140.
  • the method 1700 includes receiving, while the digital pen 115 is operating in a first input mode (e.g., an inking mode), data representing a current tilt angle of the digital pen 115 (at block 1705).
  • a first input mode e.g., an inking mode
  • the digital inking application 110 is configured to receive positional information, including tilt angle, from the digital pen 115 when the pen 115 includes an active pen.
  • the digital pen 115 can include electronic components or circuitry, such as an inertial measurement (IMU), such one or more accelerometers, gyroscopes, magnetometers, and the like, that allow the digital pen 115 to detects an tilt angle (e.g. with respect to a horizontal position, such as a surface of the touchscreen 220) and this information can be transmitted to the user device 105 (e.g., via a digitizer included in the user device 105).
  • IMU inertial measurement
  • the digital pen 115 can include electronic components or circuitry, such as an inertial measurement (IMU), such one or more accelerometers, gyroscopes, magnetometers, and the like, that allow the digital pen 115 to detects an tilt angle (e.g. with respect to a horizontal position, such as a surface of the touchscreen 220) and this information can be transmitted to
  • the digital inking application 110 can be configured to detect the tilt angle of the digital pen 115 based on data received from other data sources, such as based on detected interactions between the digital pen 115 and the touchscreen 220, one or more cameras included in the user device 105 configured to detect a position of the digital pen 115, or the like.
  • the digital inking application 110 is configured to detect the tilt angle of the digital pen 115 when the digital pen 115 is contacting the touchscreen 220.
  • the digital inking application 110 can be configured to determine a title angle of the digital pen 115 for each digital stroke made via the digital pen 115.
  • a tilt angle of the digital pen 115 may be detected using a pointer segment tracker that tracks interactions between the digital pen 115 and the touchscreen, such as the displacement of the tip of the digital pen 115 from the touchscreen 220, which varies as the digital pen 115 is tilted by different amounts.
  • the digital inking application 110 may be configured to detect the tilt angle of the digital pen 115 regardless of whether the digital pen 115 is contacting the touchscreen.
  • the digital inking application determines whether the tilt angle satisfies a predetermined threshold associated with a second input mode (e.g., by comparing the detected tilt angle to the predetermined threshold) (at block 1710).
  • a predetermined threshold associated with a second input mode e.g., by comparing the detected tilt angle to the predetermined threshold
  • the digital inking application 110 sets the input mode of the digital pen 115 to the second input mode, which, as described above, may be a highlighting mode (at block 1715).
  • the digital pen 115 in its current mode (e.g., the first input mode) (at block 1720).
  • the predetermined threshold associated with second input mode is specified in terms of degrees (e.g., from a horizontal position, wherein a tilt angle of approximately 90° represents a vertical position of the digital pen 115) and, in some embodiments, is 60°.
  • a current tilt angle of the digital pen 115 satisfies the predetermined threshold when the current tilt angle is equal to or less than 60°.
  • a user when using the digital pen 115 as a pen or other writing device to digitally ink content, a user typically holds the digital pen 115 in a generally upright position (e.g., greater than approximately 60° from a horizontal position).
  • the digital inking application 110 can be configured to determine whether the current tilt position of the digital pen 115 is less than equal to the predetermined threshold associated with the second input mode and, if so, automatically switch the digital pen 115 to the second input mode.
  • the predetermined threshold for the second input mode can include various values, such as, for example, values between approximately 30° to 70°. Also, in some embodiments, ranges of values may be used as compared to single thresholds.
  • the digital inking application 110 is also configured to automatically switch the digital pen 115 back to the first input mode (or another input mode, such as any mode specified as a default input mode for the pen 115) by continuing to detect a current tilt angle of the digital pen (e.g., for each detected ink stroke) and switching the digital pen 115 to the first input mode in response to detecting a current tilt angle that no longer satisfies the predetermined threshold associated with the second input mode.
  • the digital inking application 110 may require a particular change in tilt before switching the digital pen 115 to a different input mode (e.g., to prevent inadvertent or rapid switching between modes).
  • the digital inking application 110 may require a detected tilt angle of the pen 115 to satisfy (or not satisfy depending on the switch) the predetermined threshold for a particular time period (e.g., again to prevent inadvertent or rapid switching between modes).
  • the digital inking application 110 can apply different thresholds (or ranges) based on the user (e.g., whether the user is right-handed or left-handed), the digital pen 115, the user device 105, or a combination thereof.
  • the digital inking application 110 is configured to learn a particular tilt that a user operates the digital pen 115 when using the pen in a particular input mode and automatically set a learned tilt (or range thereof) as the predetermined threshold for the user (and optionally other users).
  • the digital inking application 110 can learn these tilt angles based on reference tilt angles, which may obtained during normal use of the digital pen 115 after the user manually selects a particular input mode, based on prompts by the digital inking application 110 for the user operating the digital pen 115 to tilt the pen 115 in desired way for one or more different uses of the digital pen 115, or a combination thereof. Accordingly, the predetermined threshold described above can be customizable.
  • interactions detected between the digital pen 115 and the touchscreen 220 can be translated to ink points and digital strokes as described above, wherein the digital strokes are provided in a highlighter color (e.g., yellow, orange, pink, green, etc.) with a level of transparency that allows objects (such as digital ink) under the highlighting strokes to be seen through the highlighting strokes.
  • the highlighting strokes are stored as a digital ink instance as described above, which can be positioned on top of or otherwise overlaid over existing digital ink (e.g., dry ink) or other content within the canvas.
  • the digital inking application 110 can be configured to automatically format the highlighting strokes to position the strokes over adjacent digital ink or other content, to match a height of adjacent content, or the like. Similarly, in some embodiments, the digital inking application 110 can be configured to automatically smooth or straighten highlighting strokes to improve the visual appearance of the strokes. In other embodiments, when the digital pen 115 is in the highlighting mode, digital strokes can be used to trigger a pen command to update existing content (e.g., existing digital ink) such as to change the color of digital strokes, a color of a background provided by the canvas, or both.
  • existing content e.g., existing digital ink
  • pen input representing highlighting strokes can be used to trigger command that edit existing digital ink to add formatting representing the highlighting.
  • the digital pen 115 may include other types of input modes and, in some embodiments, these modes can be associated with particular tilt or positions of the digital pen 115, which allows the user to change the input mode of the digital pen 115 through changes in position, orientation, or grip on the pen 115, which results in more efficient switching and less distraction to the user.
  • a digital pen 115 may have an underling mode, an italics mode, an underling mode, a selection mode, an eraser mode, a grouping mode, or a combination thereof.
  • the digital inking application 110 may be configured to “beautify” digital ink while retaining its personal style.
  • FIG. 14 illustrates a user interface 1800 provided by the digital inking application 110.
  • the user interface 1800 includes a portion of an electronic document (e.g., a digital journal) within a canvas 1802.
  • the canvas 1802 includes a digital ink instance 1805 within an electronic document.
  • a user uses the digital pen 115 to draw a predetermined signifier associated with a “beautify” command.
  • this predetermined signify includes a predetermined mark 1810 adjacent to the digital ink instance 1805 (e.g., within a predetermined distance from the instance 1805, around the instance 1805, or the like).
  • the predetermined mark 1810 may be a circle, a square, a triangle, or other predetermined shape.
  • the predetermined signifier also includes a second predetermined mark 1815, which may be a symbol drawn adjacent to the digital ink instance 1805, adjacent to the first predetermined mark 1810, or both. It should be understood that in some embodiments, only one of the first predetermined mark 1810 and the second predetermined mark 1815 is used to invoke the “beautify” command.
  • the digital inking application 110 identifies the predetermined marks 1810 and 1815 as invoking the “beautify” command and generates a new digital ink instance 1820 or updates the existing digital ink instance 1805 to improve the formatting of the contents without, in some embodiments, changing the ink contents.
  • invoking the “beautify” command can modify the contents of the marked digital ink 1850 align lines to the left of the canvas 1802, align words to a baseline in each line, unify line spacing between lines in the same paragraph, unify word spacing between words in the same line, or a combination thereof.
  • the digital inking application 110 uses various services, such as various application programming interfaces, to identify characters, words, and word groups (e.g., lines or paragraphs) within digital ink (e.g., using various handwriting analysis tool) and to compute font metrics (e.g., descender, baseline, meanline, ascender, etc.).
  • the digital inking application 110 may be configured to determine a minimum, an average, a maximum, or other default character height and, by identifying individual characters within the digital ink, such as by using handwriting analysis tools, and modify the digital strokes making up a particular character to have a particular height.
  • the position of individual characters i.e., the associated digital strokes defining a character
  • the digital inking application 110 can also beautify digital ink by reflowing the contents of the digital ink based on the available space within the canvas 1802.
  • this new digital ink instance can be added to the canvas as a separate instance 1820.
  • the original digital ink instance 1805 can be updated to perform the beautification or replaced with a newly generated digital ink instance.
  • the beautified digital ink instance can retain (e.g., as part of the DOM) the original ink points and digital strokes and merely apply formatting to the data when rendering the digital ink within the canvas. Retaining the original ink points and digital strokes enables the user to return to the original digital ink (i.e., undo a requested beautification).
  • the digital inking application 110 can be configured to re-format digital ink without merely changing the digital to text and without requiring that the user ink in certain areas of the canvas or on guided lines or other templates.
  • the digital inking application 110 enables users to freely ink on a canvas while understanding the ink’s layout hierarchy, which allows the application 110 to align digital ink and adjust spacing at various levels (e.g., characters, lines, paragraphs, lists, etc.).
  • the digital inking application 110 can also compute a rotation angle of lines, which allows the digital inking application 110 to rotate lines (e.g., horizontally), such as to a particular slant level. For example, as handwritten content often includes a particular slant, the digital inking application 110 can remove this slant or can unify the slant to preserve the unique handwritten nature of content.
  • the “beautify” command can be invoked in additional or alternative ways than the pen command described above.
  • a user can select digital ink (e.g., using a select input mode of the digital pen 115, a lasso tool, or the like) and then select a “beautify” option from a menu or toolbar included in the digital inking application 110 or using various pen gestures, such as a press and hold of the digital pen 115 on the touchscreen to access a menu or mode or automatically invoke the “beautify” command.
  • the user can turn on automatic beautification, which automatically invokes the “beautify” command for digital ink periodically or in response to a particular trigger, such as when the user pauses while inking (e.g., no new digital ink or other input from the digital pen 115 is received for a predetermined amount of time).
  • the digital inking application 110 can be configured to apply only particular types of “beautification” when the “beautify” command is invoked to allow a user to apply only one or more types of beautification or beautification at particular levels (e.g., at a line level but not a character level).
  • the digital inking application 110 can apply one or more types of “beautifications” as established by one or more configurations, which may be set by a user or automatically selected based context (e.g., the language used in the digital ink).
  • “beautification” illustrated in FIG. 14 aligns lines to the left
  • the digital inking application 110 can be configured to align lines in other formats, such as center-aligned or right-aligned.
  • the digital inking application 110 may be configured to left-align digital ink written in English or other Latin-based languages since these languages are read left to right.
  • the digital inking application 110 may be configured to right-align digital ink written in Arabic as this language is read right to left.
  • the “beautify” command can be invoked as part of other commands or functionality provided by the digital inking application 110.
  • the duplicated digital ink instance as added to the appropriate aggregation page, can be beautified to create an aesthetically pleasing and organized list or page.
  • the digital inking application 110 is configured to reflow digital ink.
  • the digital inking application 110 can be configured to reflow of digital ink content as a user inserts or removes content without requiring that the content of the digital ink be initially converted to text.
  • the digital inking application 110 enables a user to invoke a “reflow” command using a pen command, which eliminates the need for a user to switch tools (e.g., from an inking tool or mode to an editing tool or mode).
  • FIG. 15A illustrates a user interface 1900 provided by the digital inking application 110.
  • the user interface 1900 includes a portion of an electronic document (e.g., a digital journal) within a canvas 1902.
  • the canvas 1902 includes a digital ink instance 1905 within an electronic document.
  • a user can draw a line (e.g., a vertical line) within (e.g. overlaid on) the digital ink instance 1905 at a position where the user wants to reflow the content of the digital ink (e.g., make remove for the insertion of additional digital ink content or remove excess whitespace within the digital ink content).
  • a line e.g., a vertical line
  • the user can press and hold the pen 115 to the touchscreen 220 to convert the line into a command line.
  • the user can drag, with the digital pen 115, the command line (e.g., left and right) to reflow the digital ink within the digital ink instance 1905 to create room within the digital ink instance to insert additional ink or to remove excess whitespace.
  • Reflowing content of a digital ink instance can be performed by modifying the position (e.g., vertical, horizontal, or both) of the ink points, digital strokes, or both of the digital ink instance as stored in the DOM.
  • FIG. 15B illustrates the digital ink instance 1905 where a user has drawn a line 1910 the word “the” included in the digital ink instance 1905.
  • the digital inking application 110 converts the line 1910 into a command line, which the user can drag to the left or the right.
  • FIG. 15C in response to the user dragging the command line 1910 to the right, the contents of the digital ink instance 1905 (positioned to the right of the command line) is reflowed to add whitespace 1915 within the digital ink instance 1905 (i.e., before or to the left of the word “the” where the line 1910 was originally drawn).
  • the space between the word “the” and the word “Fijipedia” can be reduced or eliminated.
  • the digital inking application 110 can be configured to perform reflow on raw digital ink without requiring that the digital ink be converted to text.
  • user interface interaction is minimized as a user can draw a line as a pen command and can then drag the line without need to initially lift the pen 115 or perform other gestures or actions.
  • the “reflow” command can be automatically invoked in response to a user removes contents of a paragraph or line or drags and drops a word into an existing line.
  • the “reflow” command can be automatically invoked in response to a user adding (e.g., dragging and dropping) a new ink instance into the list or removing an ink instance from the list.
  • the digital inking application 110 can be configured to automatically reflow an existing list of digital ink instances as a user marks (i.e., with a predetermined signifier in digital ink) additional digital ink instances to be duplicated and aggregated on the page.
  • reflowing a digital ink instance maintains the original ink points, digital strokes, or both (as stored in the DOM) but updates the positions of the ink points, digital strokes, or both to reflect the new formatting of the contents. Accordingly, even when reflowing digital ink contents, the handwriting style can be maintained.
  • a user may be required to select a digital ink instance to be edited (e.g., by marking the digital ink instance with a predetermined signifier in digital ink, using a lasso tool, select the instance with the digital pen in a selection input mode, or the like).
  • a portion of an electronic document includes multiple digital ink instances or even multiple words, phrases, lines, or paragraphs, the user may not know what digital ink content is included in a particular digital ink instance.
  • a user may be able to see boundaries of a particular digital ink instance after selecting particular content within a canvas, when the revealed boundaries are not what the user thought or expected, they may have to cancel the selection and potentially undo editing that may be performed in response to selecting the instance, which is inefficient and wastes computing resources.
  • the digital inking application 110 can be configured to provide a “reveal” or “inspect” command that allows a user to preview boundaries of a digital ink stance (before actually selecting the instance), which allows the user to verify the boundaries and edit the boundaries of an ink instance as needed before selecting the instance, editing the instance, or both.
  • FIG. 16 illustrates a user interface 2000 provided by the digital inking application 110.
  • the user interface 2000 includes a portion of an electronic document (e.g., a digital journal) within a canvas 2002.
  • the canvas 2002 includes a digital ink instance 2005 within an electronic document.
  • a user can hover the digital pen 115 (while being operated in an inking mode where the pen 115 can be used to add digital ink to the canvas 2002) adjacent to the digital ink instance 2005, such as to the left of the digital ink instance 2005.
  • the digital inking application 110 updates the canvas 2002 to disclose the boundaries of the digital ink instance 2005, such as using a boundary graphic 2010 (e.g., a box) as illustrated in FIG. 16.
  • a boundary graphic 2010 e.g., a box
  • other icons or graphics can be used to designate which digital ink content is included within a digital ink instance positioned adjacent to the hovering location of the digital pen 115. For example, content included within a digital ink instance can be highlighted, pulsed, underlined. Similarly, in some embodiments, comer edges, partial lines, or other graphics can be used to indicate the boundaries.
  • the boundary graphic 2010 surrounds or otherwise marks what digital ink content is included in the digital ink instance 2005 and what digital ink content is not included.
  • the boundary graphic 2010 enables the user to see what content will be edited if the user invokes a particular command on the digital ink instance 2005. If the contents are not what the user expected or desires, the user can edit the digital ink instance 2005. For example, a user can add additional content to the digital ink instance, such as by dragging and dropping other digital ink content 2015 on the digital ink instance 2005 as illustrated in FIG. 16.
  • FIG. 17 illustrates a user interface 2100 illustrating the digital pen 115 hovering left of the digital ink instance 2105 including the content reflow text” within a canvas 2102.
  • This hovering triggers the “inspect” command as described above, which adds a boundary graphic 2110 to the canvas 2102 to indicate the boundaries of the digital ink instance 2105.
  • the digital inking application 110 can add an icon 2115 to the canvas 2102 that indicates where a user can tap (while the boundary graphic 2110 is being displayed) to select the digital ink instance 2105 associated with the boundary graphic 2110.
  • the digital inking application 110 in response to receiving a tap or selection of the icon 2115 with the digital pen 115, is configured to select the digital ink instance 2105, which allows the user to move the instance 2105 (e.g., by dragging and dropping the instance 2105, which can reflow digital ink as described above), invoke another editing command on the instance 2105 (e.g., copy the instance 2105, delete the instance 2105, change a color or other display property of the instance 2105), connect the instance 2105 to another object, or the like.
  • the digital inking application 110 may be configured to remove the boundary graphic 2110 and the icon 2115 from the canvas 2102.
  • the digital inking application 110 may similarly remove the boundary graphic 2110 and the icon 2115 from the canvas 2102 in response to the hover position of the digital pen 115 changing by a predetermined amount (e.g., indicating that the digital pen 115 is no longer positioned adjacent to the digital ink instance 2105).
  • a predetermined amount e.g., indicating that the digital pen 115 is no longer positioned adjacent to the digital ink instance 2105.
  • FIG. 18 is a flow chart illustrating a method 2200 for inspecting or revealing boundaries of a digital ink instance according to some embodiments.
  • the method 2200 is described as being performed by the digital inking application 110 (and the components described above with respect to FIG. 3) as executed by the electronic processor 204 of the user device 105.
  • the method 2200 or portions thereof may be performed by the digital inking service 130 provided via the server 140.
  • the method 2200 includes receiving data representing a hover position of the digital pen 115 (at block 2205).
  • the digital inking application 110 is configured to detect the hover position of digital pen 115 based on data received from the digital pen 115 when the pen 115 includes an active pen.
  • the digital inking application 110 can be configured to detect the hover position of the digital pen 115 based on other data sources, such as one or more cameras included in the user device 105 configured to detect a position of the digital pen 115, components configured to detect the presence of the digital pen 115 through the touchscreen 220 even the digital pen 115 is not contacting the touchscreen 220, or the like.
  • the digital inking application 110 determines if the hover position is adjacent to an existing digital ink instance (e.g., within a predetermined distance to the left of an existing digital ink instance) (at block 2210). In some embodiments, if multiple existing digital ink instances are positioned adjacent to the hover position, the digital inking application 110 may use the digital ink instance that is positioned closest to the hover position. In some embodiments, the digital inking application 110 also determines whether the hover position has been maintained for a predetermined amount of time (e.g., to ensure that the user is intended to invoke the “inspect” command and not just moving the digital pen 115 for other purposes).
  • a predetermined amount of time e.g., to ensure that the user is intended to invoke the “inspect” command and not just moving the digital pen 115 for other purposes.
  • the digital inking application 110 updates the canvas to display a boundary graphic for the digital ink instance, wherein the boundary ground defines the bounds or what content within the canvas is included in the digital ink instance (at block 2215).
  • the boundary graphic can take various forms and, in some embodiments, includes a bounding box.
  • the digital inking application 110 is also configured to, in response to the hover position being adjacent to an existing digital ink instance, display an icon within the canvas corresponding to the hover position (at block 2220).
  • the digital inking application 110 is configured to select the digital ink instance (currently displayed with the boundary graphic) as an active or selected item similar to when user selects a digital ink instance using a lasso tool, a finger tap, or another selection mechanism (at block 2225). After the digital ink instance is selected, a user can move or otherwise manipulate the instance.
  • the digital inking application 110 can cancel the “inspect” command, which removes both the boundary graphic and the icon from the canvas. Also, in some embodiments, as illustrated in FIG. 18, in response to the hover position of the digital pen 115 changing so that it is not adjacent to a digital ink instance within the canvas (at block 2210), the digital inking application 110 is configured to remove any existing boundary graphic from the canvas (at block 2227).
  • the “inspect” command allows a user to perform more precise manipulation of digital ink in an efficient manner. For example, not only can a user confirm what content is included in a particular digital ink instance before going through the process of selecting the instance (which often requires an input mode switch and may automatically apply editing to the selected instance) but the user can subsequently select the instance without having to go through the normal context switch (from inking mode to selection mode). Also, as compared to selecting content using a lasso tool, a finger touch, or another mechanism, using the boundary reveal allows a user to make a more accurate selection. For example, when different ink objects are close together within a canvas, it can be difficult to accurate make a selection using a lasso or a finger touch.
  • embodiments described herein provide, among other things, systems and methods of digital inking and digital ink editing.
  • embodiments, described herein provide a digital inking application configured to invoke various commands in response to detected pen gestures, which eliminate or reduce the amount of input and associated computing resources needed to perform desired functionality.
  • embodiments described herein enable a user to use natural gestures with a digital pen within a canvas to invoke various functionality.
  • Some embodiments described herein also ease the difficulty of modifying digital ink while preserving the aesthetic structure and organization of an electronic document (i.e., without requiring that the user convert handwritten content to text).
  • the pen gestures i.e., positions, predetermined signifiers in digital ink, and the like
  • the pen gestures can take different form to invoke particular commands.
  • a circle illustrated as a predetermined signifier for a particular pen command can, in some embodiments, be a different shape or symbol and may have fewer or different requirements in terms of size, position, timing (e.g., continuous stroke or multiple strokes, etc.), or the like.
  • a user can customize or otherwise configure the digital inking application 110 to define user-specific pen gestures and associated signifiers.
  • various machine learning techniques can be applied by the digital inking application 110 to automatically learn signifiers written by a user via a digital pen that may be associated with particular commands or functionality to create a customized inking environment for a user.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

Systèmes et procédés pour révéler des limites d'une instance d'encre numérique dans un document électronique. Un système comprend une mémoire configurée pour stocker des instructions et un processeur électronique couplé à la mémoire. Le processeur électronique, par l'exécution des instructions stockées dans la mémoire, est configuré pour recevoir une position de survol d'un stylo numérique sur une toile affichée par l'intermédiaire d'un écran tactile, déterminer si la position de survol du stylo numérique est adjacente à une position de l'instance d'encre numérique à l'intérieur de la toile, et, en réponse à la position de survol du stylo numérique adjacente à la position de l'instance d'encre numérique à l'intérieur de la toile, révéler les limites de l'instance d'encre numérique à l'intérieur de la toile par l'intermédiaire d'un graphique de limite ajouté à la toile.
PCT/US2022/018399 2021-03-16 2022-03-01 Révélation et sélection de groupement d'encres WO2022197436A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202117202543A 2021-03-16 2021-03-16
US17/202,543 2021-03-16

Publications (1)

Publication Number Publication Date
WO2022197436A1 true WO2022197436A1 (fr) 2022-09-22

Family

ID=80819618

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2022/018399 WO2022197436A1 (fr) 2021-03-16 2022-03-01 Révélation et sélection de groupement d'encres

Country Status (1)

Country Link
WO (1) WO2022197436A1 (fr)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150015504A1 (en) * 2013-07-12 2015-01-15 Microsoft Corporation Interactive digital displays
US20180329589A1 (en) * 2017-05-15 2018-11-15 Microsoft Technology Licensing, Llc Contextual Object Manipulation

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150015504A1 (en) * 2013-07-12 2015-01-15 Microsoft Corporation Interactive digital displays
US20180329589A1 (en) * 2017-05-15 2018-11-15 Microsoft Technology Licensing, Llc Contextual Object Manipulation

Similar Documents

Publication Publication Date Title
US11526659B2 (en) Converting text to digital ink
KR102413461B1 (ko) 제스쳐들에 의한 노트 필기를 위한 장치 및 방법
US7966352B2 (en) Context harvesting from selected content
US9665259B2 (en) Interactive digital displays
US7996589B2 (en) Auto-suggest lists and handwritten input
US20040257346A1 (en) Content selection and handling
EP2871563A1 (fr) Dispositif électronique, procédé et support de stockage
CN108700994A (zh) 用于数字墨水交互性的系统和方法
WO2018080949A1 (fr) Manipulation de table à main libre
EP4309071A1 (fr) Duplication et agrégation d'instances d'encre numérique
EP4309148A1 (fr) Soumission de questions à l'aide d'encre numérique
US20150154443A1 (en) Electronic device and method for processing handwritten document
US11361153B1 (en) Linking digital ink instances using connecting lines
US11372486B1 (en) Setting digital pen input mode using tilt angle
WO2022197436A1 (fr) Révélation et sélection de groupement d'encres
CA3075627A1 (fr) Editeur de document integre

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22711746

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22711746

Country of ref document: EP

Kind code of ref document: A1