EP3268849A1 - Traitement d'entrée d'encre numérique soumise à la surveillance et à l'intervention d'un programme d'application - Google Patents

Traitement d'entrée d'encre numérique soumise à la surveillance et à l'intervention d'un programme d'application

Info

Publication number
EP3268849A1
EP3268849A1 EP16707343.6A EP16707343A EP3268849A1 EP 3268849 A1 EP3268849 A1 EP 3268849A1 EP 16707343 A EP16707343 A EP 16707343A EP 3268849 A1 EP3268849 A1 EP 3268849A1
Authority
EP
European Patent Office
Prior art keywords
input
ink
event
application
events
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP16707343.6A
Other languages
German (de)
English (en)
Inventor
Yibo SUN
Krishnan Menon
Connor WEINS
Xiao TU
Olumuyiwa Durojaiye
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
Publication of EP3268849A1 publication Critical patent/EP3268849A1/fr
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/0416—Control or interface arrangements specially adapted for digitisers
    • G06F3/04162—Control or interface arrangements specially adapted for digitisers for exchanging data with external devices, e.g. smart pens, via the digitiser sensing hardware
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B41—PRINTING; LINING MACHINES; TYPEWRITERS; STAMPS
    • B41J—TYPEWRITERS; SELECTIVE PRINTING MECHANISMS, i.e. MECHANISMS PRINTING OTHERWISE THAN FROM A FORME; CORRECTION OF TYPOGRAPHICAL ERRORS
    • B41J2/00—Typewriters or selective printing mechanisms characterised by the printing or marking process for which they are designed
    • B41J2/485—Typewriters or selective printing mechanisms characterised by the printing or marking process for which they are designed characterised by the process of building-up characters or image elements applicable to two or more kinds of printing or marking processes
    • B41J2/49—Typewriters or selective printing mechanisms characterised by the printing or marking process for which they are designed characterised by the process of building-up characters or image elements applicable to two or more kinds of printing or marking processes by writing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/0416—Control or interface arrangements specially adapted for digitisers
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/542—Event management; Broadcasting; Multicasting; Notifications

Definitions

  • Electronic ink is content added to an electronic document based upon a user input technique where the user handwrites, draws, or performs other arbitrary spatial movements.
  • visual marks are added to the document that correspond to the path of the movement. For example, in the case of handwriting, handwritten words appear in the document in the same handwriting style that they were executed by the user.
  • the user performs this input by moving a stylus or his or her finger across the surface of the display device on which the document is displayed.
  • the electronic ink generally appears in the document in the positions where the interactions occurred.
  • the user performs the inking input by moving a displayed position pointer on the screen, using an input device such as a mouse or touchpad.
  • a facility for processing ink input is described.
  • the facility receives ink input from an input device.
  • the facility generates and renders ink stroke data structures in response to received ink input in accordance with an ink input processing pipeline.
  • the facility provides to an executing application access to information traversing the ink input processing pipeline at a selected point in the ink input processing pipeline.
  • Figure 2 is a flow diagram showing example acts that may be performed by the facility in some examples to process broad device input.
  • Figure 3 is a table diagram showing sample contents of an ink event registration table used by the facility in some examples to store information about applications' registration of ink event handlers.
  • Figure 4 is a block diagram showing some of the components that may be incorporated in at least some of the computer systems and other devices on which the facility operates.
  • the application's event handlers can use their access to events of these types to affect the behavior of the ink processing pipeline.
  • an event handler for raw device input can, based upon analysis of this raw device input, instruct the pipeline to abort processing and rendering of an in- progress stroke, such as where a stroke would be improper in the current location of the application display area, or where the application recognizes the raw device input making up the stroke as matching a user intent other than inking a stroke, such as issuing a command to the application via a gesture.
  • the facility monitors execution of application's event handlers in high-priority threads to ensure that the application's event handlers do not abuse the high priority of these threads by doing extensive processing, thus preventing such event handlers from interfering with the pipeline's prompt generation and rendering of ink strokes, which is an important part of providing a good inking experience to users.
  • the computer systems and devices may include any number of the following: a central processing unit ("CPU") 101 for executing computer programs; a computer memory 102 for storing programs and data while they are being used, including the facility and associated data, an operating system including a kernel and device drivers, and one or more applications; a persistent storage device 103, such as a hard drive or flash drive for persistently storing programs and data; a computer-readable media drive 104, such as a floppy, CD-ROM, or DVD drive, for reading programs and data stored on a computer- readable medium; and/or a communications subsystem 105 for connecting the computer system to other computer systems and/or other devices to send and/or receive data, such as via the Internet or another wired or wireless network and its networking hardware, such as switches, routers, repeaters, electrical cables and optical fibers, light emitters and receivers, radio transmitters and receivers, and the like.
  • CPU central processing unit
  • a computer memory 102 for storing programs and data while they are being used, including the facility and associated data
  • these computer systems and other devices 100 may further include any number of the following: a display 106 for presenting visual information, such as text, images, icons, documents, menus, etc.; and a touchscreen digitizer 107 for sensing interactions with the display, such as touching the display with one or more fingers, styluses, or other objects.
  • the touchscreen digitizer uses one or more available techniques for sensing interactions with the display, such as resistive sensing, surface acoustic wave sensing, surface capacitance sensing, projected capacitance sensing, infrared grid sensing, infrared acrylic projection sensing, optical imaging sensing, dispersive signal sensing, and acoustic pulse recognition sensing.
  • the computer systems and other devices 100 include input devices of various other types, such as keyboards, mice, styluses, etc. (not shown).
  • Figure 2 is a flow diagram showing example acts that may be performed by the facility in some examples to process broad device input.
  • the facility performs the acts of Figure 2 in a thread having high execution priority, such as a background thread for generating and rendering ink strokes.
  • the facility receives device input. If the application has registered an input event handler for receiving device input, then the facility continues at 211, else the facility continues at 203.
  • the facility provides various mechanisms for an application to register an event handler, such as calling an operating system API, calling an ink system API, storing one or more key/value pairs in the registry, etc.
  • the facility revises and renders the in-progress stroke as wet ink. In some examples, such revision involves extending the stroke to a new point that is the destination of the move input, smoothing the shape of the stroke in light of this extension, etc. After 207, the facility continues at 205.
  • the facility continues at 210, else the facility returns to 201.
  • this other input can be of various types, such as hovering, erasing, pressing a stylus barrel button, clicking a mouse's right button, etc.
  • the facility fires a stroke event, serviced by the application on the host thread.
  • the non-stroke event handler performs various processing, such as performing custom forms of erasing, selection of ink and other content, hovering, etc. After 210, the facility returns to 201.
  • the facility fires an input event corresponding to the input received at 201.
  • the input event is serviced by the application on the high-priority background thread.
  • the input event handler performs various processing, such as detecting a gesture and invoking a corresponding command in response; preventing inking in a forbidden region; etc.
  • the application's input event handler can determine whether the input event should be marked as "handled,” reflecting that the application is claiming the device input that makes up any current in- progress stroke, and that this device input should not be processed by the pipeline.
  • the facility continues at 213, else the facility continues at 203.
  • the facility cancels and de-renders the in-process stroke.
  • the facility receives additional device input.
  • the facility fires an input event in a manner similar to act 211, that corresponds to the input received at 214.
  • the facility returns to 201, else the facility returns to 214.
  • Figure 3 is a table diagram showing sample contents of an ink event registration table used by the facility in some examples to store information about applications' registration of ink event handlers.
  • the table 300 is made up of rows, such as rows 301-303, each corresponding to a different event handler registered by an application.
  • Each row is made up of the following columns: a process column 311 containing identifier for the process in which the application that registered the event handler is executing; a type column 312 indicating the category of events - input, stroke, or nonstroke - for which the event handler is registered; and a code pointer column 313 containing an entry point for the event handler, or other information that can be used to invoke the event handler as part of firing an event of the registered category.
  • Figure 3 shows a table whose contents and organization are designed to make it more comprehensible by a human reader
  • actual data structures used by the facility to store this information may differ from the table shown, in that they, for example, may be organized in a different manner; may contain more or less information than shown; may be compressed and/or encrypted; may contain a much larger number of rows than shown; etc.
  • FIG. 4 is a block diagram showing some of the components that may be incorporated in at least some of the computer systems and other devices on which the facility operates.
  • the computing system 400 includes a processor 410 configured to execute an application.
  • the computing system further includes an input device interface 420 configured to receive ink input from an input device.
  • the computing system also includes an ink input processing subsystem 430 configured to generate and render ink stroke data structures. It does so in response to ink input received from an input device by the input device interface, in accordance with an ink input processing pipeline.
  • the computing system also includes an ink input processing pipeline access subsystem 440 configured to provide to the application executed by the processor access to information traversing the ink input processing pipeline at a selected point in the ink input processing pipeline.
  • the facility provides a computing system for processing ink input, comprising: a processor configured to execute an application; an input device interface configured to receive ink input; an ink input processing subsystem configured to generate and render ink stroke data structures in response to ink input received by the input device interface in accordance with an ink input processing pipeline; and an ink input processing pipeline access subsystem configured to provide to the application executed by the processor access to information traversing the ink input processing pipeline at a selected point in the ink input processing pipeline.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • User Interface Of Digital Computer (AREA)
  • Character Discrimination (AREA)

Abstract

La présente invention concerne une installation de traitement d'entrée d'encre. Dans un exemple d'installation, l'installation reçoit une entrée d'encre de la part d'un dispositif d'entrée. L'installation génère et procède au rendu des structures de données de traits d'encre en réponse à une entrée d'encre reçue en fonction d'un pipeline de traitement d'entrée d'encre. L'installation fournit à une application en cours d'exécution un accès à des informations traversant le pipeline de traitement d'entrée d'encre au niveau d'un point sélectionné dans le pipeline de traitement d'entrée d'encre.
EP16707343.6A 2015-03-12 2016-02-19 Traitement d'entrée d'encre numérique soumise à la surveillance et à l'intervention d'un programme d'application Withdrawn EP3268849A1 (fr)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201562132428P 2015-03-12 2015-03-12
US14/830,607 US9950542B2 (en) 2015-03-12 2015-08-19 Processing digital ink input subject to monitoring and intervention by an application program
PCT/US2016/018565 WO2016144502A1 (fr) 2015-03-12 2016-02-19 Traitement d'entrée d'encre numérique soumise à la surveillance et à l'intervention d'un programme d'application

Publications (1)

Publication Number Publication Date
EP3268849A1 true EP3268849A1 (fr) 2018-01-17

Family

ID=55447164

Family Applications (1)

Application Number Title Priority Date Filing Date
EP16707343.6A Withdrawn EP3268849A1 (fr) 2015-03-12 2016-02-19 Traitement d'entrée d'encre numérique soumise à la surveillance et à l'intervention d'un programme d'application

Country Status (4)

Country Link
US (1) US9950542B2 (fr)
EP (1) EP3268849A1 (fr)
CN (1) CN107438818B (fr)
WO (1) WO2016144502A1 (fr)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10228775B2 (en) * 2016-01-22 2019-03-12 Microsoft Technology Licensing, Llc Cross application digital ink repository
US10147159B2 (en) 2017-04-07 2018-12-04 Microsoft Technology Licensing, Llc Ink render using high priority queues

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5347295A (en) * 1990-10-31 1994-09-13 Go Corporation Control of a computer through a position-sensed stylus
US5710831A (en) * 1993-07-30 1998-01-20 Apple Computer, Inc. Method for correcting handwriting on a pen-based computer
US5561446A (en) * 1994-01-28 1996-10-01 Montlick; Terry F. Method and apparatus for wireless remote information retrieval and pen-based data entry
US8166388B2 (en) 2002-05-14 2012-04-24 Microsoft Corporation Overlaying electronic ink
US20030214531A1 (en) 2002-05-14 2003-11-20 Microsoft Corporation Ink input mechanisms
US7174042B1 (en) 2002-06-28 2007-02-06 Microsoft Corporation System and method for automatically recognizing electronic handwriting in an electronic document and converting to text
US7436535B2 (en) * 2003-10-24 2008-10-14 Microsoft Corporation Real-time inking
JP2007520374A (ja) * 2004-01-15 2007-07-26 コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ スキャナおよび施与器をもつ電子ペイントブラシ
US7284192B2 (en) 2004-06-24 2007-10-16 Avaya Technology Corp. Architecture for ink annotations on web documents
US8160363B2 (en) * 2004-09-25 2012-04-17 Samsung Electronics Co., Ltd Device and method for inputting characters or drawings in a mobile terminal using a virtual screen
US7499058B2 (en) 2005-04-22 2009-03-03 Microsoft Corporation Programmatical access to handwritten electronic ink in a tree-based rendering environment
US8315482B2 (en) 2007-06-26 2012-11-20 Microsoft Corporation Integrated platform for user input of digital ink
US8116569B2 (en) 2007-12-21 2012-02-14 Microsoft Corporation Inline handwriting recognition and correction
US8189922B2 (en) 2008-09-25 2012-05-29 Lenovo (Singapore) Pte. Ltd. Techniques for adjusting a drying time of digital ink
US8477103B2 (en) 2008-10-26 2013-07-02 Microsoft Corporation Multi-touch object inertia simulation
US9589241B2 (en) * 2011-03-31 2017-03-07 The United States Of America As Represented By The Secretary Of The Navy Electrical resource controller
US20140210798A1 (en) 2013-01-31 2014-07-31 Hewlett-Packard Development Company, L.P. Digital Drawing Using A Touch-Sensitive Device To Detect A Position And Force For An Input Event
US9286703B2 (en) 2013-02-28 2016-03-15 Microsoft Technology Licensing, Llc Redrawing recent curve sections for real-time smoothing

Also Published As

Publication number Publication date
US9950542B2 (en) 2018-04-24
CN107438818B (zh) 2020-09-04
WO2016144502A1 (fr) 2016-09-15
US20160263911A1 (en) 2016-09-15
CN107438818A (zh) 2017-12-05

Similar Documents

Publication Publication Date Title
US9996176B2 (en) Multi-touch uses, gestures, and implementation
US7907125B2 (en) Recognizing multiple input point gestures
TWI553541B (zh) 用於詞意圖像對比的方法及計算裝置
US20090183098A1 (en) Configurable Keyboard
KR102323892B1 (ko) 멀티 터치 가상 마우스
JP2014241139A (ja) 仮想タッチパッド
JP2005196740A (ja) 競合を解決するグラフィックマルチユーザインタフェース及びグラフィックマルチユーザインタフェースの競合を解決する方法
US9268476B2 (en) Drag and drop interaction paradigm with image swap
JP2013222263A (ja) 情報処理装置およびその制御方法、コンピュータプログラム
US20140298275A1 (en) Method for recognizing input gestures
US20120096349A1 (en) Scrubbing Touch Infotip
US9950542B2 (en) Processing digital ink input subject to monitoring and intervention by an application program
CN108885556B (zh) 控制数字输入
US20150143289A1 (en) Automatic check box interaction
CN111142775A (zh) 一种手势交互方法和装置
EP3433713B1 (fr) Sélection d'un premier comportement d'entrée numérique sur la base de la présence d'une seconde entrée simultanée
WO2022062806A1 (fr) Procédé de rejet de toucher erroné, dispositif terminal et support de stockage
EP4177702A1 (fr) Appareil et procédé de prise en charge d'événements d'entrée tactiles
Procházka et al. Mainstreaming gesture based interfaces
US20160210038A1 (en) Electronic inking
JP2014089592A (ja) 情報処理装置
KR20170002165A (ko) 사용자 정의 제스처 설정 시스템 및 이를 이용한 제스처 설정 방법

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20170825

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
17Q First examination report despatched

Effective date: 20181010

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN WITHDRAWN

18W Application withdrawn

Effective date: 20190125