WO2014200736A1 - Gestion de reconnaissance d'écriture manuscrite en temps réel - Google Patents

Gestion de reconnaissance d'écriture manuscrite en temps réel Download PDF

Info

Publication number
WO2014200736A1
WO2014200736A1 PCT/US2014/040417 US2014040417W WO2014200736A1 WO 2014200736 A1 WO2014200736 A1 WO 2014200736A1 US 2014040417 W US2014040417 W US 2014040417W WO 2014200736 A1 WO2014200736 A1 WO 2014200736A1
Authority
WO
WIPO (PCT)
Prior art keywords
handwriting
recognition
handwriting input
input
user
Prior art date
Application number
PCT/US2014/040417
Other languages
English (en)
Inventor
Mei-qun XIA
Jannes G. DOLFING
Ryan S. DIXON
Karl M. GROETHE
Karan Misra
Jerome R. Bellegarda
Ueli Meier
Original Assignee
Apple Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US14/290,945 external-priority patent/US9465985B2/en
Priority to CN201811217768.9A priority Critical patent/CN109614845B/zh
Priority to KR1020157033627A priority patent/KR101892723B1/ko
Priority to KR1020217043310A priority patent/KR20220003662A/ko
Priority to CN201480030897.0A priority patent/CN105247540B/zh
Priority to KR1020207016098A priority patent/KR102221079B1/ko
Priority to JP2016518366A priority patent/JP6154550B2/ja
Priority to KR1020217005264A priority patent/KR102347064B1/ko
Application filed by Apple Inc. filed Critical Apple Inc.
Priority to CN201811217822.XA priority patent/CN109614847B/zh
Priority to KR1020187024261A priority patent/KR102005878B1/ko
Priority claimed from US14/292,138 external-priority patent/US20140361983A1/en
Priority claimed from US14/291,722 external-priority patent/US20140363082A1/en
Priority to KR1020197021958A priority patent/KR102121487B1/ko
Priority claimed from US14/291,865 external-priority patent/US9495620B2/en
Publication of WO2014200736A1 publication Critical patent/WO2014200736A1/fr
Priority to HK16108185.0A priority patent/HK1220276A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/22—Character recognition characterised by the type of writing
    • G06V30/226—Character recognition characterised by the type of writing of cursive writing
    • G06V30/2264—Character recognition characterised by the type of writing of cursive writing using word shape
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/32—Digital ink
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/28—Character recognition specially adapted to the type of the alphabet, e.g. Latin alphabet
    • G06V30/287—Character recognition specially adapted to the type of the alphabet, e.g. Latin alphabet of Kanji, Hiragana or Katakana characters
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/28—Character recognition specially adapted to the type of the alphabet, e.g. Latin alphabet
    • G06V30/293—Character recognition specially adapted to the type of the alphabet, e.g. Latin alphabet of characters other than Kanji, Hiragana or Katakana

Definitions

  • This specification relates to providing handwriting input functionality on a computing device, and more specifically, to providing real-time, multi-script, stroke-order independent handwriting recognition and input functionality on a computing device.
  • a handwriting input method is an important alternative input method for computing devices that are equipped with touch-sensitive surfaces (e.g., touch-sensitive display screens, or touch-pads).
  • touch-sensitive surfaces e.g., touch-sensitive display screens, or touch-pads.
  • Many users particularly users in some Asian or Arab countries, are accustomed to writing in a cursive style, and may feel comfortable writing in longhand as opposed to typing on a keyboard.
  • This specification describes a technique for providing multi-script handwriting recognition using a universal recognizer.
  • the universal recognizer is trained using a large multi-script corpus of writing samples for characters in different languages and scripts.
  • the training of the universal recognizer is language-independent, script-independent, stroke- order independent, and stroke-direction independent.
  • the same recognizer is capable of recognizing mixed-language, mixed-script handwriting input without requiring manual switching between input languages during use.
  • the universal recognizer is lightweight enough to be deployed as a standalone module on mobile devices to enable handwriting input in different languages and scripts used in different regions worldwide.
  • the universal recognizer is trained on spatially-derived features which are stroke-order independent and stroke-direction independent, and require no temporal or sequence information at the stroke-level, the universal recognizer provides numerous additional features and advantages over conventional temporally-based recognition methods (e.g., recognition methods based on the Hidden Markov Method (HMM)). For example, the user is permitted to enter the strokes of one or more characters, phrases, and sentences in any order, and still obtain the same recognition results. Thus, out- of-order multi-character input, and out-of-order corrections (e.g., additions or rewrites) of earlier-entered characters are now possible.
  • HMM Hidden Markov Method
  • the universal recognizer is used for real-time handwriting recognition, where temporal information for each stroke is available and is optionally used to disambiguate or segment the handwriting input before character recognition is performed by the universal recognizer.
  • the real-time, stroke-order independent recognition described herein differs from conventional offline recognition methods (e.g., Optical Character Recognition (OCR)) and can offer better performance than conventional offline recognition methods.
  • OCR Optical Character Recognition
  • the universal recognizer described herein is able to handle high variability in individual writing habits (e.g., variability in speed, tempo, stroke-order, stroke-direction, stroke-continuity, etc.) without explicitly embedding distinguishing features of the different variations (e.g., variations in speed, tempo, stroke-order, stroke- direction, stroke-continuity, etc.) in the recognition system, thereby reducing the overall complexity of the recognition system.
  • high variability in individual writing habits e.g., variability in speed, tempo, stroke-order, stroke-direction, stroke-continuity, etc.
  • distinguishing features of the different variations e.g., variations in speed, tempo, stroke-order, stroke- direction, stroke-continuity, etc.
  • temporally-derived stroke distribution information is optionally reintroduced into the universal recognizer to enhance recognition accuracy and disambiguate between similar-looking recognition outputs for the same input image.
  • the re-introduction of the temporally-derived stroke distribution information does not destroy the stroke-order and stroke-direction independence of the universal recognizer, because the temporally-derived features and the spatially-derived features are obtained through a separate training process and are only combined in the handwriting recognition model after the separate training has been completed.
  • the temporally-derived stroke distribution information is carefully designed such that it captures distinguishing temporal characteristics of similar-looking characters, without relying on explicit knowledge on the differences in the stroke-orders of the similar-looking characters.
  • a method of providing multi-script handwriting recognition includes: training a multi-script handwriting recognition model based on spatially-derived features of a multi-script training corpus, the multi-script training corpus including respective handwriting samples corresponding to characters of at least three non-overlapping scripts; and providing real-time handwriting recognition for a user's handwriting input using the multi- script handwriting recognition model that has been trained on the spatially-derived features of the multi-script training corpus.
  • a method of providing multi-script handwriting recognition includes: receiving a multi-script handwriting recognition model, the multi- script recognition model having been trained on spatially-derived features of a multi-script training corpus, the multi-script training corpus including respective handwriting samples corresponding to characters of at least three non- overlapping scripts; receiving a handwriting input from a user, the handwriting input comprising one or more handwritten strokes provided on a touch- sensitive surface coupled to the user device; and in response to receiving the handwriting input, providing in real-time one or more handwriting recognition results to the user based on the multi-script handwriting recognition model that has been trained on the spatially-derived features of the multi-script training corpus.
  • a method of providing real-time handwriting recognition includes: receiving a plurality of handwritten strokes from a user, the plurality of handwritten strokes corresponding to a handwritten character; generating an input image based on the plurality of handwritten strokes; providing the input image to a handwriting recognition model to perform real-time recognition of the handwritten character, wherein the handwriting recognition model provides stroke-order independent handwriting recognition; and displaying in real-time of receiving the plurality of handwritten strokes, an identical first output character irrespective of a respective order by which the plurality of handwritten strokes have been received from the user.
  • the method further includes: receiving a second plurality of handwritten strokes from the user, the second plurality of handwritten strokes corresponding to a second handwritten character; generating a second input image based on the second plurality of handwritten strokes; providing the second input image to the handwriting recognition model to perform real-time recognition of the second handwritten character; and displaying in real-time of receiving the second plurality of handwritten strokes, a second output character corresponding to the second plurality of handwritten strokes, wherein the first output character and the second output character are concurrently displayed in a spatial sequence independent of a respective order by which the first plurality of handwriting inputs and the second plurality of handwriting inputs have been provided by the user.
  • the second plurality of handwritten strokes spatially follow the first plurality of handwritten strokes along a default writing direction of a handwriting input interface of the user device, and the second output character follows the first output character in a spatial sequence along the default writing direction
  • the method further includes: receiving a third handwritten stroke from the user to revise the handwritten character, the third handwritten stroke being received temporally after the first and the second plurality of handwritten strokes; in response to receiving the third handwritten stroke, assigning the handwritten stroke to a same recognition unit as the first plurality of handwritten strokes based on relative proximity of the third handwritten stroke to the first plurality of handwritten strokes; generating a revised input image based on the first plurality of handwritten stroke and the third handwritten stroke; providing the revised input image to the handwriting recognition model to perform real-time recognition of the revised handwritten character; and displaying in response to receiving the third handwriting input, a third output character corresponding to the revised input image, wherein the third output character replaces
  • the method further includes: while the third output character and the second output character are concurrently displayed as a recognition result in a candidate display area of the handwriting input interface, receiving a deletion input from the user; and in response to the deletion input, deleting the second output character from the recognition result, while maintaining the third output character in the recognition result.
  • rendering in real-time the first plurality of handwritten strokes, the second plurality of handwritten strokes, and the third handwritten stroke in the handwriting input area of the handwriting input interface as each of said handwritten stroke is provided by the user; and in response to receiving the deletion input, deleting a respective rendering of the second plurality of handwritten strokes from the handwriting input area, while maintaining respective renderings of the first plurality of handwritten strokes and the third handwritten stroke in the handwriting input area.
  • a method of providing real-time handwriting recognition includes: receiving a handwriting input from a user, the handwriting input comprising one or more handwritten strokes provided in a handwriting input area of a handwriting input interface; based on a handwriting recognition model, identifying a plurality of output characters for the handwriting input; dividing the plurality of output characters into two or more categories based on a predetermined categorization criterion; displaying, in an initial view of a candidate display area of the handwriting input interface, respective output characters in a first category of the two or more categories, wherein the initial view of the candidate display area is concurrently provided with an affordance for invoking an extended view of the candidate display area; receiving a user input selecting the affordance for invoking the extended view; and in response to the user input, displaying, in the extended view of the candidate display area, the respective output characters in the first category and respective output characters in at least a second category of the two or more categories which were not previously displayed in the initial view of
  • a method of providing real-time handwriting recognition includes: receiving a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided in an handwriting input area of a handwriting input interface; recognizing, based on a handwriting recognition model, a plurality of output characters from the handwriting input, the output characters including at least a first emoji character and at least a first character from a script of a natural human language; and displaying a recognition result comprising the first emoji character and the first character from the script of the natural human language in a candidate display area of the handwriting input interface.
  • a method of providing handwriting recognition includes: receiving a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided in a touch- sensitive surface coupled to the device; rendering, in real-time, the plurality of handwritten strokes in a handwriting input area of a
  • handwriting input interface receiving one of a pinch gesture input and a expand gesture input over the plurality of handwritten strokes; upon receiving a pinch gesture input, generating a first recognition result based on the plurality of handwritten strokes by treating the plurality of handwritten strokes as a single recognition unit; upon receiving a expand gesture input, generating a second recognition result based on the plurality of handwritten strokes by treating the plurality of handwritten strokes as two separate recognition units pulled apart by the expand gesture input; and upon generating a respective one of the first and second recognition results, displaying the generated recognition result in a candidate display area of the handwriting input interface.
  • a method of providing handwriting recognition includes: receiving a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided in an handwriting input area of a handwriting input interface; identifying a plurality of recognition units from the plurality of handwritten strokes, each recognition unit including a respective subset of the plurality of handwriting strokes;
  • generating a multi-character recognition result comprising respective characters recognized from the plurality of recognition units; displaying the multi-character recognition result in a candidate display area of the handwriting input interface; while the multi-character recognition result is displayed in the candidate display area, receiving a deletion input from the user; and in response to receiving the deletion input, removing an end character from the multi-character recognition result displayed in the candidate display area.
  • a method of providing real-time handwriting recognition includes: determining an orientation of the device; providing a handwriting input interface on the device in a horizontal input mode in accordance with the device being in a first orientation, wherein a respective line of handwriting input entered in the horizontal input mode is segmented into one or more respective recognition units along a horizontal writing direction; and providing the handwriting input interface on the device in a vertical input mode in accordance with the device in a second orientation, wherein a respective line of handwriting input entered in the vertical input mode is segmented into one or more respective recognition units along a vertical writing direction.
  • a method of providing real-time handwriting recognition includes: receiving a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided on a touch-sensitive surface coupled to the device; rendering the plurality of handwritten strokes in a handwriting input area of a handwriting input interface; segmenting the plurality of handwritten strokes into two or more
  • recognition units each recognition unit comprising a respective subset of the plurality of handwritten strokes; receiving an edit request from the user; in response to the edit request, visually distinguishing the two or more recognition units in the handwriting input area; and providing a means for individually deleting each of the two or more recognition units from the handwriting input area.
  • a method of providing real-time handwriting recognition includes: receiving a first handwriting input from a user, the first handwriting input comprising a plurality of handwritten strokes, and the plurality of handwritten strokes forming multiple recognition units distributed along a respective writing direction associated with a handwriting input area of a handwriting input interface; rendering each of the plurality of handwritten strokes in the handwriting input area as the handwritten stroke is provided by the user; starting a respective fading process for each of the multiple recognition units after the recognition unit is completely rendered, wherein during the respective fading process, the rendering of the recognition unit in the first handwriting input are becomes increasingly faded; receiving a second handwriting input from the user over a region of the handwriting input area occupied by a faded recognition unit of the multiple recognition unit; and in response to receiving the second handwriting input: rendering the second handwriting input in the handwriting input area; and clearing all faded recognition units from the handwriting input area.
  • a method of providing hand- writing recognition includes: separately training a set of spatially-derived features and a set of temporally-derived features of a handwriting recognition model, wherein: the set of spatially-derived features are trained on a corpus of training images each being an image of a handwriting sample for a respective character of an output character set, and the set of temporally-derived features are trained on a corpus of stroke-distribution profiles, each stroke-distribution profile numerically characterizing a spatial distribution of a plurality of strokes in a handwriting sample for a respective character of the output character set; and combining the set of spatially-derived features and the set of temporally-derived features in the handwriting recognition model; and providing real-time handwriting recognition for a user's
  • Figure 1 is a block diagram illustrating a portable multifunction device with a touch-sensitive display in accordance with some embodiments.
  • Figure 2 illustrates a portable multifunction device having a touch-sensitive display in accordance with some embodiments.
  • Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface in accordance with some embodiments.
  • Figure 4 illustrates an exemplary user interface for a multifunction device with a touch-sensitive surface that is separate from the display in accordance with some embodiments.
  • Figure 5 is a block diagram of an operating environment of a handwriting input system in accordance with some embodiments.
  • Figure 6 is a block diagram of a multi- script handwriting recognition model in accordance with some embodiments.
  • Figure 7 is a flow chart of an exemplary process for training a multi-script handwriting recognition model in accordance with some embodiments.
  • Figures 8A-8B show exemplary user interfaces showing real-time, multi- script handwriting recognition and input on a portable multifunction device in accordance with some embodiments.
  • Figures 9A-9B are flow charts of an exemplary process for providing real-time multi-script handwriting recognition and input on a portable multifunction device.
  • Figures 1 OA- IOC are flow charts of an exemplary process for providing realtime stroke-order independent handwriting recognition and input on a portable
  • Figures 1 lA-1 IK show exemplary user interfaces for selectively displaying one category of recognition results in a normal view of a candidate display area, and other categories of recognition results in an extended view of the candidate display area, in accordance with some embodiments.
  • Figures 12A-12B is are flow charts of an exemplary process for selectively displaying one category of recognition results in a normal view of a candidate display area, and other categories of recognition results in an extended view of the candidate display area, in accordance with some embodiments.
  • Figures 13A-13E show exemplary user interfaces for entering emoji characters through handwriting input in accordance with some embodiments.
  • Figure 14 is a flow chart of an exemplary process for entering emoji characters through handwriting input in accordance with some embodiments.
  • Figures 15A-15K show exemplary user interfaces for using a pinch or expand gesture to inform the handwriting input module about how to divide a currently
  • Figures 16A-16B are flow charts of an exemplary process for using a pinch or expand gesture to inform the handwriting input module about how to divide a currently accumulated handwriting input into one or more recognition units, in accordance with some embodiments.
  • Figures 17A-17H show exemplary user interfaces for providing character-by- character deletion of a user's handwriting input in accordance with some embodiments.
  • Figures 18A-18B are flow charts of an exemplary process for providing character-by-character deletion of a user' s handwriting input in accordance with some embodiments.
  • Figures 19A-19F show exemplary user interfaces for switching between a vertical writing mode and a horizontal writing mode in accordance with some
  • Figures 20A-20C are flow charts of an exemplary process for switching between a vertical writing mode and a horizontal writing mode in accordance with some
  • Figures 21A-21H show user interfaces for providing a means to display and selectively delete individual recognition units identified in a user' s handwriting input, in accordance with some embodiments.
  • Figures 22A-22B are flow charts of an exemplary process for providing a means to display and selectively delete individual recognition units identified in a user's handwriting input, in accordance with some embodiments.
  • Figures 23A-23L show exemplary user interfaces for utilizing a new
  • handwriting input provided over an existing handwriting input in the handwriting input area as an implicit confirmation input for entering a recognition result displayed for the existing handwriting input, in accordance with some embodiments.
  • Figures 24A-24B are flow charts of an exemplary process for utilizing a new handwriting input provided over an existing handwriting input in the handwriting input area as an implicit confirmation input for entering a recognition result displayed for the existing handwriting input, in accordance with some embodiments.
  • Figures 25A-25B are flow charts of an exemplary process for integrating temporally-derived stroked distribution information into a handwriting recognition model based on spatially-derived features, without destroying the stroke-order and stroke direction independence of the handwriting recognition model, in accordance with some embodiments.
  • Figure 26 is a block diagram illustrating separate training and subsequent integration of spatially-derived features and temporally-derived features of an exemplary handwriting recognition system in accordance with some embodiments.
  • Figure 27 is a block diagram illustrating an exemplary method for computing the stroke distribution profile of a character.
  • Many electronic devices have graphical user interfaces with soft keyboards for character entry.
  • a user may also be able to install or enable a handwriting input interface that allows the user to input characters via handwriting on a touch-sensitive display screen or a touch-sensitive surface coupled to the devices.
  • Figures 1-4 below provide a description of exemplary devices.
  • Figures 5, 6, and 26-27 illustrate exemplary handwriting recognition and input systems.
  • Figures 8A-8B, 11A- 11K, 13A-13E, 15A-15K, 17A-17H, 19A-19F, 21A-21H, 23A-12L illustrate exemplary user interfaces for handwriting recognition and input.
  • Figures 7, 9A-9B, lOA-lOC, 12A- 12B, 14, 16A-16B, 18A-18B, 20A-20C, 22A-22B, 24A-24B, and 25 are flow charts illustrating methods of enabling handwriting recognition and input on user devices, including training handwriting recognition models, providing real-time handwriting recognition results, providing means for inputting and revising a handwriting input, and providing means for entering a recognition result as a text input.
  • FIGS 8A-8B, 11A-11K, 13A-13E, 15A-15K, 17A-17H, 19A-19F, 21A-21H, 23A-12L are used to illustrate the processes in Figures 7, 9A-9B, lOA-lOC, 12A-12B, 14, 16A-16B, 18A-18B, 20A-20C, 22A-22B, 24A-24B, and 25.
  • first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.
  • a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the present invention.
  • the first contact and the second contact are both contacts, but they are not the same contact.
  • the term “if may be construed to mean “when” or “upon” or “in response to determining” or “in response to detecting,” depending on the context.
  • the phrase “if it is determined” or “if [a stated condition or event] is detected” may be construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event],” depending on the context.
  • the device is a portable communications device, such as a mobile telephone, that also contains other functions, such as PDA and/or music player functions.
  • portable multifunction devices include, without limitation, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California.
  • Other portable electronic devices such as laptops or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and/or touch pads), may also be used.
  • the device is not a portable communications device, but is a desktop computer with a touch-sensitive surface (e.g., a touch screen display and/or a touch pad).
  • an electronic device that includes a display and a touch-sensitive surface is described. It should be understood, however, that the electronic device may include one or more other physical user-interface devices, such as a physical keyboard, a mouse and/or a joystick.
  • the device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a gaming application, a telephone application, a video conferencing
  • an e-mail application an instant messaging application, a workout support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application.
  • the various applications that may be executed on the device may use at least one common physical user-interface device, such as the touch-sensitive surface.
  • One or more functions of the touch- sensitive surface as well as corresponding information displayed on the device may be adjusted and/or varied from one application to the next and/or within a respective application.
  • a common physical architecture (such as the touch- sensitive surface) of the device may support the variety of applications with user interfaces that are intuitive and transparent to the user.
  • Figure 1 is a block diagram illustrating portable multifunction device 100 with touch- sensitive displays 112 in accordance with some embodiments.
  • Touch- sensitive display 112 is sometimes called a "touch screen" for convenience, and may also be known as or called a touch-sensitive display system.
  • Device 100 may include memory 102 (which may include one or more computer readable storage mediums), memory controller 122, one or more processing units (CPU's) 120, peripherals interface 118, RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, input/output (I/O) subsystem 106, other input or control devices 116, and external port 124.
  • Device 100 may include one or more optical sensors 164. These components may communicate over one or more communication buses or signal lines 103.
  • device 100 is only one example of a portable multifunction device, and that device 100 may have more or fewer components than shown, may combine two or more components, or may have a different configuration or arrangement of the components.
  • the various components shown in Figure 1 may be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and/or application specific integrated circuits.
  • Memory 102 may include high-speed random access memory and may also include non- volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by other components of device 100, such as CPU 120 and the peripherals interface 118, may be controlled by memory controller 122.
  • Peripherals interface 118 can be used to couple input and output peripherals of the device to CPU 120 and memory 102.
  • the one or more processors 120 run or execute various software programs and/or sets of instructions stored in memory 102 to perform various functions for device 100 and to process data.
  • peripherals interface 118, CPU 120, and memory controller 122 may be implemented on a single chip, such as chip 104. In some other embodiments, they may be implemented on separate chips.
  • RF (radio frequency) circuitry 108 receives and sends RF signals, also called electromagnetic signals. RF circuitry 108 converts electrical signals to/from
  • electromagnetic signals and communicates with communications networks and other communications devices via the electromagnetic signals.
  • Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between a user and device 100.
  • Audio circuitry 110 receives audio data from peripherals interface 118, converts the audio data to an electrical signal, and transmits the electrical signal to speaker 111.
  • Speaker 111 converts the electrical signal to human-audible sound waves.
  • Audio circuitry 110 also receives electrical signals converted by microphone 113 from sound waves.
  • Audio circuitry 110 converts the electrical signal to audio data and transmits the audio data to peripherals interface 118 for processing. Audio data may be retrieved from and/or transmitted to memory 102 and/or RF circuitry 108 by peripherals interface 118.
  • audio circuitry 110 also includes a headset jack (e.g., 212, Figure 2).
  • I/O subsystem 106 couples input/output peripherals on device 100, such as touch screen 112 and other input control devices 116, to peripherals interface 118.
  • I/O subsystem 106 may include display controller 156 and one or more input controllers 160 for other input or control devices.
  • the one or more input controllers 160 receive/send electrical signals from/to other input or control devices 116.
  • the other input control devices 116 may include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and so forth.
  • input controller(s) 160 may be coupled to any (or none) of the following: a keyboard, infrared port, USB port, and a pointer device such as a mouse.
  • the one or more buttons (e.g., 208, Figure 2) may include an up/down button for volume control of speaker 111 and/or microphone 113.
  • the one or more buttons may include a push button (e.g., 206, Figure 2).
  • Touch- sensitive display 112 provides an input interface and an output interface between the device and a user.
  • Display controller 156 receives and/or sends electrical signals from/to touch screen 112.
  • Touch screen 112 displays visual output to the user.
  • the visual output may include graphics, text, icons, video, and any combination thereof (collectively termed "graphics"). In some embodiments, some or all of the visual output may correspond to user-interface objects.
  • Touch screen 112 has a touch- sensitive surface, sensor or set of sensors that accepts input from the user based on haptic and/or tactile contact.
  • Touch screen 112 and display controller 156 (along with any associated modules and/or sets of instructions in memory 102) detect contact (and any movement or breaking of the contact) on touch screen 112 and converts the detected contact into interaction with user-interface objects (e.g., one or more soft keys, icons, web pages or images) that are displayed on touch screen 112.
  • user-interface objects e.g., one or more soft keys, icons, web pages or images
  • a point of contact between touch screen 112 and the user corresponds to a finger of the user.
  • Touch screen 112 may use LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies may be used in other embodiments.
  • Touch screen 112 and display controller 156 may detect contact and any movement or breaking thereof using any of a plurality of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch screen 112.
  • touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch screen 112.
  • projected mutual capacitance sensing technology is used, such as that found in the iPhone®, iPod Touch®, and iPad® from Apple Inc. of Cupertino, California.
  • Touch screen 112 may have a video resolution in excess of 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi.
  • the user may make contact with touch screen 112 using any suitable object or appendage, such as a stylus, a finger, and so forth.
  • the user interface is designed to work primarily with finger-based contacts and gestures, which can be less precise than stylus- based input due to the larger area of contact of a finger on the touch screen.
  • the device translates the rough finger-based input into a precise
  • Handwriting input may be provided on the touch screen 112 via locations and movements of finger-based or stylus-based contacts.
  • the touch screen 112 renders the finger-based or stylus-based input as instantaneous visual feedback to the current handwriting input, and provides the visual effect of actual writing on a writing surface (e.g., a sheet of paper) with a writing instrument (e.g., a pen).
  • a writing surface e.g., a sheet of paper
  • a writing instrument e.g., a pen
  • device 100 may include a touchpad (not shown) for activating or deactivating particular functions.
  • the touchpad is a touch-sensitive area of the device that, unlike the touch screen, does not display visual output.
  • the touchpad may be a touch- sensitive surface that is separate from touch screen 112 or an extension of the touch- sensitive surface formed by the touch screen.
  • Device 100 also includes power system 162 for powering the various components.
  • Power system 162 may include a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light- emitting diode (LED)) and any other components associated with the generation, management and distribution of power in portable devices.
  • power sources e.g., battery, alternating current (AC)
  • AC alternating current
  • a power failure detection circuit e.g., a power failure detection circuit
  • a power converter or inverter e.g., a power converter or inverter
  • a power status indicator e.g., a light- emitting diode (LED)
  • Device 100 may also include one or more optical sensors 164.
  • Figure 1 shows an optical sensor coupled to optical sensor controller 158 in I/O subsystem 106.
  • Optical sensor 164 may include charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) photo transistors.
  • CCD charge-coupled device
  • CMOS complementary metal-oxide semiconductor
  • Optical sensor 164 receives light from the environment, projected through one or more lens, and converts the light to data representing an image.
  • imaging module 143 also called a camera module
  • optical sensor 164 may capture still images or video.
  • Device 100 may also include one or more proximity sensors 166.
  • Figure 1 shows proximity sensor 166 coupled to peripherals interface 118.
  • proximity sensor 166 may be coupled to input controller 160 in I/O subsystem 106.
  • the proximity sensor turns off and disables touch screen 112 when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).
  • Device 100 may also include one or more accelerometers 168.
  • Figure 1 shows accelerometer 168 coupled to peripherals interface 118.
  • accelerometer 168 may be coupled to an input controller 160 in I/O subsystem 106.
  • information is displayed on the touch screen display in a portrait view or a landscape view based on an analysis of data received from the one or more accelerometers.
  • Device 100 optionally includes, in addition to accelerometer(s) 168, a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information concerning the location and orientation (e.g., portrait or landscape) of device 100.
  • GPS or GLONASS or other global navigation system
  • the software components stored in memory 102 include operating system 126, communication module (or set of instructions) 128, contact/motion module (or set of instructions) 130, graphics module (or set of instructions) 132, text input module (or set of instructions) 134, Global Positioning System (GPS) module (or set of instructions) 135, and applications (or sets of instructions) 136.
  • memory 102 stores handwriting input module 157, as shown in Figures 1 and 3.
  • the handwriting input module 157 includes a handwriting recognition model, and provides handwriting recognition and input function to a user of the device 100 (or device 300). More details of the handwriting input module 157 are provided with respect to Figures 5-27 and accompanying descriptions thereof.
  • Operating system 126 e.g., Darwin, RTXC, LINUX, UNIX, OS X,
  • WINDOWS or an embedded operating system such as VxWorks
  • VxWorks includes various software components and/or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.
  • general system tasks e.g., memory management, storage device control, power management, etc.
  • Communication module 128 facilitates communication with other devices over one or more external ports 124 and also includes various software components for handling data received by RF circuitry 108 and/or external port 124.
  • External port 124 e.g., Universal Serial Bus (USB), FIREWIRE, etc.
  • USB Universal Serial Bus
  • FIREWIRE FireWire
  • Communication module 128 facilitates communication with other devices over one or more external ports 124 and also includes various software components for handling data received by RF circuitry 108 and/or external port 124.
  • External port 124 e.g., Universal Serial Bus (USB), FIREWIRE, etc.
  • USB Universal Serial Bus
  • FIREWIRE FireWire
  • Communication module 128 facilitates communication with other devices over one or more external ports 124 and also includes various software components for handling data received by RF circuitry 108 and/or external port 124.
  • External port 124 e.g., Universal Serial Bus (USB), FIREWIRE, etc.
  • USB Universal Serial Bus
  • FIREWIRE Fire
  • Contact/motion module 130 may detect contact with touch screen 112 (in conjunction with display controller 156) and other touch sensitive devices (e.g., a touchpad or physical click wheel).
  • Contact/motion module 130 includes various software components for performing various operations related to detection of contact, such as determining if contact has occurred (e.g., detecting a finger-down event), determining if there is movement of the contact and tracking the movement across the touch- sensitive surface (e.g., detecting one or more finger-dragging events), and determining if the contact has ceased (e.g., detecting a finger-up event or a break in contact).
  • Contact/motion module 130 receives contact data from the touch-sensitive surface.
  • Determining movement of the point of contact may include determining speed (magnitude), velocity (magnitude and direction), and/or an acceleration (a change in magnitude and/or direction) of the point of contact. These operations may be applied to single contacts (e.g., one finger contacts) or to multiple simultaneous contacts (e.g., "multitouch'Vmultiple finger contacts).
  • contact/motion module 130 and display controller 156 detect contact on a touchpad.
  • Contact/motion module 130 may detect a gesture input by a user. Different gestures on the touch- sensitive surface have different contact patterns. Thus, a gesture may be detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger-down event followed by detecting a finger-up (lift off) event at the same position (or substantially the same position) as the finger-down event (e.g., at the position of an icon). As another example, detecting a finger swipe gesture on the touch- sensitive surface includes detecting a finger-down event followed by detecting one or more finger-dragging events, and subsequently followed by detecting a finger-up (lift off) event.
  • Contact/motion module 130 is optionally utilized by the handwriting input module 157 to register input of handwritten strokes within a handwriting input area of a handwriting input interface displayed on the touch-sensitive display screen 112 (or within an area of the touch-pad 355 corresponding to the handwriting input area displayed on display 340 in Figure 3).
  • locations, motion path, and intensities associated with the contact during the initial finger-down event, the final finger-up event, any time there between is recorded as a handwritten stroke. Based on such information, the handwritten strokes can be rendered on the display, as feedback for the user input.
  • one or more input images can be generated based on handwritten strokes registered by the contact/motion module 130.
  • Graphics module 132 includes various known software components for rendering and displaying graphics on touch screen 112 or other display, including components for changing the intensity of graphics that are displayed.
  • graphics includes any object that can be displayed to a user, including without limitation text, web pages, icons (such as user-interface objects including soft keys), digital images, videos, animations and the like.
  • graphics module 132 stores data representing graphics to be used. Each graphic may be assigned a corresponding code. Graphics module 132 receives, from applications etc., one or more codes specifying graphics to be displayed along with, if necessary, coordinate data and other graphic property data, and then generates screen image data to output to display controller 156.
  • Text input module 134 which may be a component of graphics module 132, provides soft keyboards for entering text in various applications (e.g., contacts 137, e-mail 140, IM 141, browser 147, and any other application that needs text input).
  • the handwriting input module 157 is optionally invocated through a user interface of the text input module 134, e.g., though a keyboard selection affordance.
  • the same or similar keyboard selection affordance is also provided in the handwriting input interface to invoke the text input module 134.
  • GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to telephone 138 for use in location-based dialing, to camera 143 as picture/video metadata, and to applications that provide location-based services such as weather widgets, local yellow page widgets, and
  • Applications 136 may include the following modules (or sets of instructions), or a subset or superset thereof: contacts module 137 (sometimes called an address book or contact list); telephone module 138; video conferencing module 139; e-mail client module 140; instant messaging (IM) module 141; workout support module 142; camera module 143 for still and/or video images; image management module 144; browser module 147;
  • calendar module 148 widget modules 149, which may include one or more of: weather widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user, as well as user-created widgets 149-6; widget creator module 150 for making user-created widgets 149-6; search module 151; video and music player module 152, which may be made up of a video player module and a music player module; notes module 153; map module 154; and/or online video module 155.
  • widget modules 149 which may include one or more of: weather widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user, as well as user-created widgets 149-6; widget creator module 150 for making user-created widgets 149-6; search module 151; video and music player module 152, which may be made up of a video player module and a music player module; notes module 153
  • Examples of other applications 136 that may be stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights
  • contacts module 137 may be used to manage an address book or contact list (e.g., stored in application internal state 192 of contacts module 137 in memory 102 or memory 370), including: adding name(s) to the address book; deleting name(s) from the address book; associating telephone number(s), e-mail address(es), physical address(es) or other information with a name; associating an image with a name; categorizing and sorting names; providing telephone numbers or e-mail addresses to initiate and/or facilitate communications by telephone 138, video conference 139, e-mail 140, or IM 141; and so forth.
  • an address book or contact list e.g., stored in application internal state 192 of contacts module 137 in memory 102 or memory 370
  • telephone module 138 may be used to enter a sequence of characters corresponding to a telephone number, access one or more telephone numbers in address book 137, modify a telephone number that has been entered, dial a respective telephone number, conduct a conversation and disconnect or hang up when the conversation is completed.
  • the wireless communication may use any of a plurality of communications standards, protocols and technologies.
  • videoconferencing module 139 includes executable instructions to initiate, conduct, and terminate a video conference between a user and one or more other participants in accordance with user instructions.
  • e-mail client module 140 includes executable instructions to create, send, receive, and manage e-mail in response to user instructions.
  • e-mail client module 140 makes it very easy to create and send e-mails with still or video images taken with camera module 143.
  • the instant messaging module 141 includes executable instructions to enter a sequence of characters corresponding to an instant message, to modify previously entered characters, to transmit a respective instant message (for example, using a Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for telephony-based instant messages or using XMPP, SIMPLE, or IMPS for Internet-based instant messages), to receive instant messages and to view received instant messages.
  • SMS Short Message Service
  • MMS Multimedia Message Service
  • XMPP extensible Markup Language
  • SIMPLE Session Initiation Protocol
  • IMPS Internet Messaging Protocol
  • transmitted and/or received instant messages may include graphics, photos, audio files, video files and/or other attachments as are supported in a MMS and/or an Enhanced Messaging Service (EMS).
  • EMS Enhanced Messaging Service
  • instant messaging refers to both telephony- based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).
  • workout support module 142 includes executable instructions to create workouts (e.g., with time, distance, and/or calorie burning goals); communicate with workout sensors (sports devices); receive workout sensor data; calibrate sensors used to monitor a workout; select and play music for a workout; and display, store and transmit workout data.
  • create workouts e.g., with time, distance, and/or calorie burning goals
  • communicate with workout sensors sports devices
  • receive workout sensor data calibrate sensors used to monitor a workout
  • select and play music for a workout and display, store and transmit workout data.
  • camera module 143 includes executable instructions to capture still images or video (including a video stream) and store them into memory 102, modify characteristics of a still image or video, or delete a still image or video from memory 102.
  • image management module 144 includes executable instructions to arrange, modify (e.g., edit), or otherwise manipulate, label, delete, present (e.g., in a digital slide show or album), and store still and/or video images.
  • modify e.g., edit
  • present e.g., in a digital slide show or album
  • browser module 147 includes executable instructions to browse the Internet in accordance with user instructions, including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.
  • calendar module 148 includes executable instructions to create, display, modify, and store calendars and data associated with calendars (e.g., calendar entries, to do lists, etc.) in accordance with user instructions.
  • widget modules 149 are mini-applications that may be downloaded and used by a user (e.g., weather widget 149-1, stocks widget 149- 2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or created by the user (e.g., user-created widget 149-6).
  • a widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file.
  • a widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widgets).
  • the widget creator module 150 may be used by a user to create widgets (e.g., turning a user- specified portion of a web page into a widget).
  • search module 151 includes executable instructions to search for text, music, sound, image, video, and/or other files in memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.
  • search criteria e.g., one or more user-specified search terms
  • video and music player module 152 includes executable instructions that allow the user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and executable instructions to display, present or otherwise play back videos (e.g., on touch screen 112 or on an external, connected display via external port 124).
  • device 100 may include the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).
  • notes module 153 includes executable instructions to create and manage notes, to do lists, and the like in accordance with user instructions.
  • map module 154 may be used to receive, display, modify, and store maps and data associated with maps (e.g., driving directions; data on stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.
  • maps e.g., driving directions; data on stores and other points of interest at or near a particular location; and other location-based data
  • online video module 155 includes instructions that allow the user to access, browse, receive (e.g., by streaming and/or download), play back (e.g., on the touch screen or on an external, connected display via external port 124), send an e-mail with a link to a particular online video, and otherwise manage online videos in one or more file formats, such as H.264.
  • instant messaging module 141 rather than e-mail client module 140, is used to send a link to a particular online video.
  • modules and applications correspond to a set of executable instructions for performing one or more functions described above and the methods described in this application (e.g., the computer-implemented methods and other information processing methods described herein).
  • modules i.e., sets of instructions
  • memory 102 may store a subset of the modules and data structures identified above.
  • memory 102 may store additional modules and data structures not described above.
  • device 100 is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen and/or a touchpad.
  • a touch screen and/or a touchpad as the primary input control device for operation of device 100, the number of physical input control devices (such as push buttons, dials, and the like) on device 100 may be reduced.
  • FIG. 2 illustrates a portable multifunction device 100 having a touch screen 112 in accordance with some embodiments.
  • the touch screen may display one or more graphics within user interface (UI) 200.
  • UI user interface
  • a user may select one or more of the graphics by making a gesture on the graphics, for example, with one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure).
  • selection of one or more graphics occurs when the user breaks contact with the one or more graphics.
  • the gesture may include one or more taps, one or more swipes (from left to right, right to left, upward and/or downward) and/or a rolling of a finger (from right to left, left to right, upward and/or downward) that has made contact with device 100.
  • inadvertent contact with a graphic may not select the graphic. For example, a swipe gesture that sweeps over an application icon may not select the corresponding application when the gesture corresponding to selection is a tap.
  • Device 100 may also include one or more physical buttons, such as "home" or menu button 204.
  • menu button 204 may be used to navigate to any application 136 in a set of applications that may be executed on device 100.
  • the menu button is implemented as a soft key in a GUI displayed on touch screen 112.
  • device 100 includes touch screen 112, menu button 204, push button 206 for powering the device on/off and locking the device, volume adjustment button(s) 208, Subscriber Identity Module (SIM) card slot 210, head set jack 212, and docking/charging external port 124.
  • Push button 206 may be used to turn the power on/off on the device by depressing the button and holding the button in the depressed state for a predefined time interval; to lock the device by depressing the button and releasing the button before the predefined time interval has elapsed; and/or to unlock the device or initiate an unlock process.
  • device 100 also may accept verbal input for activation or deactivation of some functions through microphone 113.
  • FIG. 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface in accordance with some embodiments.
  • Device 300 need not be portable.
  • device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a child's learning toy), a gaming system, a telephony device, or a control device (e.g., a home or industrial controller).
  • Device 300 typically includes one or more processing units (CPU's) 310, one or more network or other communications interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components.
  • CPU's processing units
  • Communication buses 320 may include circuitry (sometimes called a chipset) that interconnects and controls communications between system components.
  • Device 300 includes input/output (I/O) interface 330 comprising display 340, which is typically a touch screen display. I/O interface 330 also may include a keyboard and/or mouse (or other pointing device) 350 and touchpad 355.
  • Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and may include non- volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 370 may optionally include one or more storage devices remotely located from CPU(s) 310. In some embodiments, memory 370 stores programs, modules, and data structures analogous to the programs, modules, and data structures stored in memory 102 of portable multifunction device 100 ( Figure 1), or a subset thereof.
  • memory 370 may store additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100.
  • memory 370 of device 300 may store drawing module 380, presentation module 382, word processing module 384, website creation module 386, disk authoring module 388, and/or spreadsheet module 390, while memory 102 of portable multifunction device 100 ( Figure 1) may not store these modules.
  • Each of the above identified elements in Figure 3 may be stored in one or more of the previously mentioned memory devices.
  • Each of the above identified modules corresponds to a set of instructions for performing a function described above.
  • the above identified modules or programs i.e., sets of instructions
  • memory 370 may store a subset of the modules and data structures identified above.
  • memory 370 may store additional modules and data structures not described above.
  • Figure 4 illustrates an exemplary user interface on a device (e.g., device 300, Figure 3) with a touch-sensitive surface 451 (e.g., a tablet or touchpad 355, Figure 3) that is separate from the display 450 (e.g., touch screen display 112).
  • a touch-sensitive surface 451 e.g., a tablet or touchpad 355, Figure 3
  • the device detects inputs on a touch- sensitive surface that is separate from the display, as shown in Figure 4.
  • the touch sensitive surface (e.g., 451 in Figure 4) has a primary axis (e.g., 452 in Figure 4) that corresponds to a primary axis (e.g., 453 in Figure 4) on the display (e.g., 450).
  • the device detects contacts (e.g., 460 and 462 in Figure 4) with the touch-sensitive surface 451 at locations that correspond to respective locations on the display (e.g., in Figure 4, 460 corresponds to 468 and 462 corresponds to 470).
  • user inputs e.g., contacts 460 and 462, and movements thereof
  • the device on the touch- sensitive surface e.g., 451 in Figure 4
  • the device on the touch- sensitive surface e.g., 451 in Figure 4
  • FIG. 5 is a block diagram illustrating an exemplary handwriting input module 157 that interacts with the I/O interface module 500 (e.g., I/O interface 330 in Figure 3 or I/O subsystem 106 in Figure 1) to provide handwriting input capabilities on the device in accordance with some embodiments.
  • the handwriting input module 157 includes an input processing module 502, a handwriting recognition module 504, and a result generation module 506.
  • the input processing module 502 includes a segmentation module 508, and a normalization module 510.
  • the result generation module 506 includes a radical clustering module 512 and one or more language models 514.
  • the input processing module 502 communicates with the I/O interface module 500 (e.g., I/O interface 330 in Figure 3 or I/O subsystem 106 in Figure 1) to receive handwriting inputs from a user.
  • Handwriting is input via any suitable means, such as a touch-sensitive display system 112 in Figure 1 and/or a touchpad 355 in Figure 3.
  • the handwriting inputs include data representing each stroke provided by the user within a predetermined handwriting input area within the handwriting input UI.
  • the data representing each stroke of the handwriting input includes data such as the start and end locations, the intensity profile, and the motion path of a sustained contact (e.g., a contact between the user's finger or a stylus and the touch-sensitive surface of the device) within the handwriting input area.
  • the I O Interface module 500 passes the sequences of handwritten strokes 516 with associated temporal and spatial information to the input processing module 502 in real-time.
  • the I/O Interface module also provides real-time rendering 518 of the handwritten strokes within the handwriting input area of the handwriting input user interface as visual feedback to the user's input.
  • the input processing module 502 provides instructions back to the I/O interface modules 500 to render the received strokes on a display 518 (e.g., display 340 in Figure 3 or touch- sensitive display 112 in Figure 1) of the device.
  • a display 518 e.g., display 340 in Figure 3 or touch- sensitive display 112 in Figure 1
  • the rendering of the received strokes is animated to provide a visual effect mimicking actual progress of writing on a writing surface (e.g., a sheet of paper) with a writing instrument (e.g., a pen).
  • a writing instrument e.g., a pen
  • the user is optionally allowed to specify the pen-tip style, color, texture, etc. of the rendered strokes.
  • the input processing module 502 processes the strokes currently accumulated in the handwriting input area to assign the strokes into one or more recognition units.
  • each recognition unit corresponds to a character that is to be recognized by the handwriting recognition model 504.
  • each recognition unit corresponds to an output character or a radical that is to be recognized by the handwriting recognition model 504.
  • a radical is a recurring component that is found in multiple composite logographic characters.
  • a composite logographic character may include two or more radicals arranged in accordance with a common layout (e.g., a left- right layout, a top-bottom layout, etc.). In one example, a single Chinese character " is constructed using two radicals, i.e., a left radical " ⁇ " and a right radical "i ' f".
  • the input processing module 502 relies on the
  • segmentation module to assign or divide the currently accumulated handwritten strokes into one or more recognition units. For example, when segmenting the strokes for the handwritten character " ? ", the segmentation module 508 optionally assigns the strokes clustered on the left side of handwriting input to one recognition unit (i.e., for the left radical " ⁇ "), and the strokes clustered on the right side of the handwriting input to another recognition unit (i.e., for the right radical "/ "). Alternatively, the segmentation module 508 may also assign all of the strokes into a single recognition unit (i.e., for the character " r ") ⁇
  • the segmentation module 508 segments the currently accumulated handwriting input (e.g., one or more handwritten strokes) into a group of recognition units in several different ways to create a segmentation lattice 520. For example, suppose a total of nine strokes have been accumulated in the handwriting input area so far. According to a first segmentation chain of the segmentation lattice 520, strokes 1, 2, 3 are grouped into a first recognition unit 522, and strokes 4, 5, 6 are grouped into a second recognition unit 526. According to a second segmentation chain of the segmentation lattice 520, all of strokes 1-9 are grouped into one recognition unit 526.
  • each segmentation chain is given a segmentation score to measure the likelihood that the particular segmentation chain is a correct segmentation of the current handwriting input.
  • factors that are optionally used to calculate the segmentation score of each segmentation chain include: absolute and/or relative size of the stroke, relative and/or absolute span of the stroke in various directions (e.g., x, y, z directions), average of and/or variations in the saturation level of the stroke, absolute and/or relative distances to adjacent strokes, absolute and/or relative locations of the strokes, the order or sequence by which the strokes are entered, the duration of each stroke, average of and/or variations in the speed (or tempo) by which each stroke has been entered, the intensity profile of each stroke along the length of the stroke, etc.
  • one or more functions or transformations are optionally applied to one or more of these factors to generate the segmentation scores of the different segmentation chains in the segmentation lattice 520.
  • the segmentation module 508 passes the segmentation lattice 520 to the normalization module 510.
  • the normalization module 510 generates an input image (e.g., input images 528) for each recognition unit (e.g., recognition units 522, 524, and 526) specified in the segmentation lattice 520.
  • the normalization module performs the necessary or desired normalization (e.g., stretching, cropping, down-sample or up-sampling) to the input image, such that the input image can be provided to the handwriting recognition model 504 as input.
  • each input image 528 includes the strokes assigned to one respective recognition unit, and corresponds to one character or radical that is to be recognized by the handwriting recognition module 504.
  • the input images generated by the input processing module 502 does not include any temporal information associated with the individual strokes, and only spatial information (e.g. information represented by the location and density of pixels in the input image) are preserved in the input image.
  • a handwriting recognition model trained purely on spatial information of the training writing samples is capable of handwriting recognition based on spatial information alone.
  • the handwriting recognition model is stroke-order and stroke-direction independent, without exhaustively enumerating all possible permutations of stroke-orders and stroke-directions for all characters in its vocabulary (i.e., all output classes) during training.
  • the handwriting recognition module 502 does not differentiate the pixels belonging to one stroke versus another within the input image.
  • some temporally-derived stroke distribution information is reintroduced into a purely- spatial handwriting recognition model to improve recognition accuracy without compromising the stroke-order, and stroke-direction independence of the recognition model.
  • the input image generated by the input processing module 502 for one recognition unit does not overlap with the input image of any other recognition unit in the same segmentation chain.
  • input images generated for different recognition units may have some overlap.
  • some overlap between input images is permitted for recognizing handwriting input written in a cursive writing style and/or including run-on characters (e.g., one stroke connecting two adjacent characters).
  • some normalization is performed before segmentation.
  • the functions of the segmentation module 508 and the normalization module 510 may be performed by the same module or two or more other modules.
  • the handwriting recognition model 504 produces an output consisting of different likelihood of the recognition unit being a respective output character in the repertoire or vocabulary (i.e., the list of all characters and radicals recognizable by the handwriting recognition module 504) of handwriting recognition model 504.
  • the handwriting recognition model 504 has been trained to recognize a large number of characters in multiple scripts (e.g., at least three non-overlapping scripts that have been encoded by the Unicode standard). Examples of non-overlapping scripts include the Latin script, Chinese characters, Arabic letters, Farsi, Cyrillic, and artificial scripts such as emoji characters.
  • the handwriting recognition model 504 produces one or more output characters for each input image (i.e., for each recognition unit), and assigns a respective recognition score for each output character based on the confidence level associated with the character recognition.
  • the handwriting recognition model 504 generates a candidate lattice 530 in accordance with the segmentation lattice 520, where each arc in a segmentation chain (e.g., corresponding to a respective recognition unit 522, 524, 526) in the segmentation lattice 520 is expanded into one or more candidate arcs (e.g., arcs 532, 534, 536, 538, 540 each corresponding to a respective output character) within the candidate lattice 530.
  • Each candidate chain within the candidate lattice 530 is scored according to the respective segmentation score of the segmentation chain underlying the candidate chain, and the recognition scores associated with the output characters in the character chain.
  • the candidate lattice 530 is passed to the result generation module 506 to generate one or more recognition results for the currently accumulated handwriting input 516.
  • the result generation module 506 utilizes the radical clustering module 512 to combine one or more radicals in a candidate chain into a composite character.
  • the result generation module 506 uses one or more language models 514 to determine whether a character chain in the candidate lattice 530 is a likely sequence in a particular language represented by the language models.
  • the result generation module 506 generates a revised candidate lattice 542 by eliminating particular arcs or combining two or more arcs in the candidate lattice 530.
  • the result generation module 506 generates an integrated recognition score for each character sequence still remaining in the revised candidate lattice 542 (e.g., character sequences 544 and 546), based on the recognition scores of the output characters in the character sequence, as modified (e.g., augmented or diminished) by the radical clustering module 512 and language models 514. In some embodiments, the result generation module 506 ranks the different character sequences remaining in the revised candidate lattice 542 based on their integrated recognition scores.
  • the result generation module 506 sends the top-ranked character sequences as ranked recognition results 548 to the I/O interface module 500 to display to the user.
  • the I/O interface module 500 displays the received recognition results 548 (e.g., " and "3 ⁇ 4") in a candidate display area of the handwriting input interface.
  • the I/O interface module displays multiple recognition results (e.g., '"W and "III") for the user, and allows the user to select a recognition result to enter as a text input for a relevant application.
  • the I/O interface module automatically enters a top-ranked recognition result (e.g., "I3 ⁇ 4") in response to other inputs or indications of user confirmation of the recognition result.
  • Effective automatic entry of a top-ranked result can improve the efficiency of the input interface and provide a better user experience.
  • the result generation module 506 uses other factors to alter the integrated recognition scores of the candidate chains. For example, in some embodiments, the result generation module 506 optionally maintains a log of most frequently used characters for a particular user, or a multitude of users. The result generation module 506 optionally boosts the integrated recognition scores of particular candidate characters or character sequences, if the particular candidate characters or character sequences are found among the list of most frequently used characters or character sequences.
  • the handwriting input module 157 provides real-time updates for the recognition results displayed to the user. For example, in some embodiments, in some
  • the input processing module 502 for each additional stroke entered by the user, the input processing module 502 optionally re-segments the currently accumulated handwriting input, and revises the segmentation lattice and input images provided to the handwriting recognition model 504.
  • the handwriting recognition model 504 optionally revises the candidate lattice provided to the result generation module 506.
  • the result generation module 506 optionally updates the recognition results presented to the user.
  • real-time handwriting recognition refers to handwriting recognition in which handwriting recognition results are presented to the user instantaneously or within a short time period (e.g., within tens of milliseconds to seconds).
  • Real-time handwriting recognition differs from offline recognition (e.g., as in offline optical character-recognition (OCR) applications) in that recognition is initiated immediately and performed substantially contemporaneously with receipt of the handwriting input, rather than at a time after the current user session from a recorded image that is saved for later retrieval.
  • offline character recognition is performed without any temporal information regarding individual strokes and stroke sequences, and thus segmentation is performed without the benefit of such information. Further disambiguation between similar-looking candidate characters are also without the benefit of such temporal information.
  • the handwriting recognition model 504 is implemented as a convolutional neural network (CNN).
  • CNN convolutional neural network
  • Figure 6 illustrates an exemplary convolutional neural network 602 trained on a multi- script training corpus 604 containing writing samples for characters in multiple non- overlapping scripts.
  • the convolutional neural network 602 includes an input plane 606, and output plane 608. Between the input plane 606 and the output plane 608 reside a plurality of convolutional layers 610 (e.g., including a first convolutional layer 610a, zero or more intermediate convolutional layers (not shown), and a last convolutional layer 61 On). Each convolutional layer 610 is followed by a respective sub- sampling layer 612 (e.g., a first sub-sampling layer 612a, zero or more intermediate sub-sampling layers (not shown), and a last sub-sampling layer 612n). After the convolutional layers and the sub-sampling layers and right before the output plane 608 resides a hidden layer 614.
  • a hidden layer 614 After the convolutional layers and the sub-sampling layers and right before the output plane 608 resides a hidden layer 614.
  • the hidden layer 614 is the last layer before the output plane 608.
  • a kernel layer 616 e.g., including a first kernel layer 616a, zero or more intermediate kernel layers (not shown), and a last kernel layer 612n
  • each convolutional layer 610 is inserted before each convolutional layer 610 to improve computation efficiency.
  • the input plane 606 receives an input image 614 of a handwritten recognition unit (e.g., a handwritten character or radical), and the output plane 608 outputs a set of probabilities indicating the likelihood that the recognition unit belongs to respective output class (e.g., a particular character among an output character set that the neural network is configured to recognize).
  • a handwritten recognition unit e.g., a handwritten character or radical
  • the output plane 608 outputs a set of probabilities indicating the likelihood that the recognition unit belongs to respective output class (e.g., a particular character among an output character set that the neural network is configured to recognize).
  • the output classes of the neural network as a whole are also referred to as the repertoire or vocabulary of the handwriting recognition model.
  • the convolutional neural network described herein can be trained to have a repertoire of tens of thousands of characters.
  • each convolutional layer 610 is also referred to as a set of feature maps and act as filters for picking out particular features in the input image 614 for differentiating between the images corresponding to different characters.
  • the sub-sampling layers 612 ensure that features on an increasingly larger scale are captured from the input image 614.
  • the sub-sampling layers 612 are implemented using a max-pooling technique.
  • the max-pooling layers create position invariance over larger local regions and down samples the output image of the preceding convolutional layer by a factor of Kx and Ky along each direction, Kx and Ky being the size of the max-pooling rectangle.
  • Max-pooling leads to a faster convergence rate by selecting superior invariant features which improves generalization performances.
  • sub-sampling is achieved using other methods.
  • the fully-connected hidden layer 614 is a multi-layer perceptron that fully connects the nodes in the last sub- sampling layer 612n and the nodes in the output plane 608.
  • the hidden layer 614 takes the output images received from the layer before and through logistic regression reaches one of the output characters in the output layer 608.
  • the convolutional neural network 602 During training of the convolutional neural network 602, features and respective weights associated with the features in the convolutional layers 610, as well as weights associated with the parameters in the hidden layer 614 are tuned such that classification errors are minimized for the writing samples with known output classes in the training corpus 604. Once the convolutional neural network 602 has been trained, and the optimal set of parameters and associated weights have been established for the different layers in the network, the convolutional neural network 602 can be used to recognize new writing samples 618 that are not part of the training corpus 604, such as input images generated based on real-time handwriting input received from the user.
  • the convolutional neural network of a handwriting input interface is trained using a multi-script training corpus to enable multi-script or
  • the convolutional neural network is trained to recognize a large repertoire of 30 thousand to over 60 thousand characters (e.g., all characters encoded by the Unicode standard).
  • Most state-of-the-art handwriting recognition systems are based on stroke-order dependent Hidden Markov Methods (HMMs).
  • HMMs Hidden Markov Methods
  • most existing handwriting recognition models are language- specific, and include a small repertoire of tens of characters (e.g., characters of the English alphabet, the Greek alphabet, all ten digits, etc.), up to a few thousand of characters (e.g., a set of most commonly used Chinese character).
  • the universal recognizer described herein can handle orders of magnitudes more characters than most existing systems.
  • Some conventional handwriting systems may include several individually trained handwriting recognition models, each tailored for a particular language or a small set of characters.
  • a writing sample is propagated through the different recognition models until a classification can be made.
  • the handwriting sample may be provided to a series of concatenated language- specific or script-specific character recognition models, if the handwriting sample cannot be conclusively classified by a first recognition model, it is provided to a next recognition model, which attempts to classify the handwriting sample within its own repertoire.
  • the approach for classification is time consuming, and the memory requirement increases quickly with each additional recognition model that needs to be employed.
  • Other state-of-the art models require the user to specify a preferred language, and use the selected handwriting recognition model to classify the current input. Such implementations not only are cumbersome to use and consume significant memory, but also cannot be used to recognize mixed language input. Requiring the user to switch language preferences in the middle of inputting a mixed-language or mixed-script input is impractical.
  • FIG. 7 is a flow chart of an exemplary process 700 for training a handwriting recognition module (e.g., a convolutional neural network) using a large multi-script training corpus, such that the handwriting recognition module can be subsequently used to provide real-time multi-language and multi-script handwriting recognition for a user's handwriting input.
  • a handwriting recognition module e.g., a convolutional neural network
  • the training of the handwriting recognition model is performed on a server device, and the trained handwriting recognition model is then provided to a user device.
  • the handwriting recognition model optionally performs real-time handwriting recognition locally on the user device without requiring further assistance from the server.
  • both the training and the recognition is provided on the same device.
  • a server device can receive the user's handwriting input from a user device, performs the handwriting recognition, and sends the recognition results to the user device in real-time.
  • the device trains (702) a multi-script handwriting recognition model based on spatially-derived features (e.g., stroke-order independent features) of a multi-script training corpus.
  • the spatially-derived features of the multi-script training corpus are (704) stroke-order independent and stroke-direction independent.
  • the training of the multi-script handwriting recognition model is (706) independent of temporal information associated with respective strokes in the handwriting samples. Specifically, images of the handwriting samples are normalized to a
  • the images do not include any information on the order by which individual strokes are entered to form the image. Furthermore, the images also do not include any information on the direction by which individual strokes are entered to form the image.
  • features are extracted from the handwriting images without regard to how the images are temporally formed by the individual strokes. Therefore, during recognition, no temporal information related to the individual strokes is needed. As a result, the recognition robustly provides consistent recognition results despite of delayed, out-of-order strokes, and arbitrary stroke directions in the handwriting input.
  • the multi-script training corpus includes handwriting samples corresponding to characters of at least three non- overlapping scripts. As shown in Figure 6, the multi-script training corpus includes handwriting samples collected from many users. Each handwriting sample corresponds to one character of a respective script that is represented the handwriting recognition model. To adequately train the handwriting recognition model, the training corpus includes a large number of writing samples for each character of the scripts represented in the handwriting recognition model.
  • the at least three non-overlapping scripts include (708) Chinese characters, emoji characters, and Latin script.
  • the multi- script handwriting recognition model has (710) at least thirty thousand output classes, representing thirty thousand characters spanning at least three non- overlapping scripts.
  • multi- script training corpus includes respective writing samples for each character of all Chinese characters encoded in the Unicode standard (e.g., all or a substantial portion of all CJK (Chinese-Japanese-Korean) unified ideographs).
  • the Unicode standard defines a total of about seventy-four thousand CJK unified ideographs.
  • the basic block (4E00-9FFF) of the CJK unified ideographs includes 20,941 basic Chinese characters, which are used in the Chinese language, as well as in Japanese, Korean, and Vietnamese languages.
  • the multi-script training corpus includes writing samples for all characters in the basic block of the CJK unified ideographs.
  • the multi-script training corpus further includes writing samples for CJK radicals that can be used to structurally compose one or more composite Chinese characters. In some embodiments, the multi-script training corpus further includes writing samples for less frequently used Chinese characters, such as the Chinese characters encoded in one or more of the CJK unified ideograph extensions.
  • the multi-script training corpus further includes respective writing samples for each character of all characters in the Latin script encoded by the Unicode standard.
  • the characters in the basic Latin script include capital and small Latin letters, as well as various basic symbols and digits commonly used on a standard Latin keyboard.
  • the multi-script training corpus further includes characters in the extended Latin script (e.g., various accented forms of the basic Latin letters).
  • the multi-script training corpus includes writing samples that correspond to each character of an artificial script that is not associated with any natural human language.
  • a set of emoji characters is optionally defined in an emoji script, and writing samples corresponding to each of the emoji characters are included in the multi-script training corpus.
  • a hand- drawn heart-shaped symbol is a handwriting sample for the emoji character " V" in the training corpus.
  • a hand-drawn smiley face is a handwriting sample for the emoji character " ⁇ " in the training corpus.
  • emoji characters includes categories of icons showing different emotions (e.g., happy, sad, angry, embarrassed, shocked, laughing, crying, frustrated, etc.), different objects and characters (e.g., cat, dog, bunny, heart, fruit, eye, lips, gift, flowers, candle, moon, star, etc.), and different actions (e.g., handshake, kiss, run, dance, jump, sleep, eat, meet, love, like, vote, etc.), etc.
  • the strokes in the handwriting sample corresponding to an emoji character are simplified and/or stylized lines of the actual lines forming the corresponding emoji character.
  • each device or application may use a different design for the same emoji character. For example, a smiley emoji character presented to a female user may be different from a smiley emoji character presented to a male user, even if the handwriting inputs received from the two users are substantially the same.
  • the multi-script training corpus also includes writing samples for characters in other scripts, such as the Greek script (e.g., including Greek letters and symbols), the Cyrillic script, the Hebrew script, and one or more other scripts encoded according to the Unicode standard.
  • the at least three non- overlapping scripts included in the multi-script training corpus include Chinese characters, emoji characters, and characters in the Latin script. Chinese characters, emoji characters, and Characters in the Latin script are naturally non- overlapping scripts. Many other scripts may overlap with one another for at least some characters. For example, some characters (e.g., A, Z) in the Latin script may be found in many other scripts (e.g., Greek, and
  • the multi-script training corpus includes Chinese characters, Arabic script, and Latin script. In some embodiments, the multi-script training corpus includes other combinations of overlapping and/or non- overlapping scripts. In some embodiments, the multi-script training corpus includes writing samples for all characters encoded by the Unicode standard.
  • the device to train the multi-script handwriting recognition model, provides (712) the handwriting samples of the multi-script training corpus to a single convolutional neural network having a single input plane and a single output plane.
  • the device determines (714) using the convolutional neural network, the spatially-derived features (e.g., stroke-order independent features) of the handwriting samples and respective weights for the spatially-derived features for differentiating characters of the at least three non- overlapping scripts represented in the multi-script training corpus.
  • the spatially-derived features e.g., stroke-order independent features
  • the multi-script handwriting recognition model differs from conventional multi- script handwriting recognition models in that, a single handwriting recognition model having a single input plane and a single output plane is trained using all samples in the multi-script training corpus.
  • a single convolutional neural network is trained to distinguish all characters represented in the multi- script training corpus, without relying on individual sub-networks that each handles a small subset of the training corpus (e.g., sub-networks each trained for recognizing characters of a particular script or characters used in a particular language).
  • the single convolutional neural network is trained to distinguish a large number of characters spanning multiple non- overlapping scripts, rather than characters of a few overlapping scripts, such as the Latin script and the Greek script (e.g., with overlapping letters A, B, E, Z, etc.).
  • the device provides (716) real-time handwriting recognition for a user's handwriting input using the multi-script handwriting recognition model that has been trained on the spatially-derived features of the multi-script training corpus.
  • providing real-time handwriting recognition for a user's handwriting input includes continuously revising a recognition output for the user' s handwriting input as the user continues to provide additions and revisions of the
  • providing real-time handwriting recognition for a user's handwriting input further includes (718) providing the multi-script handwriting recognition model to a user device, where the user device receives a handwriting input from a user, and locally performs handwriting recognition on the handwriting input based on the multi-script handwriting recognition model.
  • the device provides the multi- script handwriting recognition model to a plurality of devices that have no existing overlap in their respective input languages, and the multi-script handwriting recognition model is used on each of the plurality of devices for handwriting recognition of a different language associated with said each user device.
  • the multi-script handwriting recognition model has been trained to recognize characters in many different scripts and languages, the same handwriting recognition model can be used worldwide to provide handwriting input for any of those input languages.
  • a first device for a user who only wishes to input in English and Hebrew can use the same handwriting recognition model to provide handwriting input functionality as a second device for another user who only wishes to input in Chinese and emoji characters.
  • a English handwriting input keyboard e.g., implemented with an English- specific
  • the same universal multi-script handwriting recognition model can be installed once on the first device, and used to provide handwriting input function for both English, Hebrew, as well as mixed input in both languages. Furthermore, instead of requiring the second user to install a Chinese handwriting input keyboard (e.g., implemented with a Chinese- specific
  • the same universal multi-script handwriting recognition model can be installed once on the second device, and used to provide handwriting input function for both Chinese, emoji, as well as mixed input in both scripts.
  • Using the same multi-script handwriting model to handle a large repertoire spanning multiple scripts e.g., a substantial portion or all of the characters encoded in nearly one hundred different scripts improves the utility of the recognizer without substantial burden on part of the device suppliers and the users.
  • the multi-script handwriting recognition model training using a large multi- script training corpus differ from conventional HMM-based handwriting recognition system, and does not rely on temporal information associated with individual strokes of the characters.
  • the resource and memory requirement for the multi-script recognition system does not increase linearly with the increase of symbols and languages covered by the multi- script recognition system.
  • increasing the number of languages means adding another independently trained model, and the memory requirement would be at least doubled to accommodate the increasing capability of the handwriting recognition system.
  • increasing language coverage requires retraining the handwriting recognition model with additional handwriting samples, and increasing the size of the output plane, but the amount of increase is very moderate.
  • the multi-script training corpus includes handwriting samples corresponding to n different languages
  • the multi-script handwriting recognition model occupies a memory of size m
  • the device re-trains the multi-script handwriting recognition model based on spatially-derived features of a second multi- script training corpus, the second multi- script training corpus including second handwriting samples corresponding to the N different languages.
  • the changes in M/m remain substantially constant within a range of 1-2, with the changes in N/n from 1 to 100.
  • Figures 8A-8B show exemplary user interfaces for providing real-time, multi-script handwriting recognition and input on a portable user device (e.g., device 100).
  • the handwriting input interface 802 is displayed on a touch- sensitive display screen (e.g., touch screen 112) of the user device.
  • the handwriting input interface 802 includes a handwriting input area 804, a candidate display area 806, and a text input area 808.
  • the handwriting input interface 802 further includes a plurality of control elements, where each can be invoked to cause the handwriting input interface to perform a predetermined function.
  • a delete button, a space button, a carriage return or Enter button, a keyboard switching button are included in the handwriting input interface.
  • Other control elements are possible, and can be optionally provided in the handwriting input interface to suit each different application utilizing the handwriting input interface 802.
  • the layout of the different components of the handwriting input interface 802 is merely illustrative, and can vary for different devices and different applications.
  • the handwriting input area 804 is a touch- sensitive area for receiving handwriting input from the user.
  • a sustained contact and its associated motion path on the touch screen within the handwriting input area 804 are registered as a handwritten stroke.
  • the handwritten stroke registered by the device is visually rendered within the handwriting input area 804 at the same locations traced by the sustained contact.
  • the user has provided a number of handwritten strokes in the handwriting input area 804, including some handwritten Chinese characters (e.g., " !.3 ⁇ 4fe"), some handwritten English letters (e.g., "Happy"), and a hand- drawn emoji character (e.g., a smiley).
  • the handwritten characters are distributed in multiple lines (e.g., two lines) in the handwriting input area 804.
  • the candidate display area 806 displays one or more recognition results (e.g., 810 and 812) for the handwriting input currently accumulated in the handwriting input area 804.
  • the top-ranked recognition result (e.g., 810) is displayed in the first position in the candidate display area.
  • the recognition result (e.g., 810) provided by the recognition model correctly includes the Chinese characters, English letters, and the emoji character represented by the handwriting input. The user is not required to stop in the middle of writing the input to choose or switch the recognition languages.
  • the text input area 808 is an area that displays text input provided to a respective application that is employing the handwriting input interface. As shown in Figure 8 A, the text input area 808 is used by a Notes application, and text (e.g., "America 3 ⁇ 43 ⁇ 43 ⁇ 4 ") currently shown within the text input area 808 is text input already provided to the Notes application. In some embodiments, a cursor 813 indicates a current text input position in the text input area 808.
  • a user can select a particular recognition result displayed in the candidate display area 806, e.g., by an explicit selection input (e.g., a tap gesture on one of the displayed recognition result), or an implicit confirmation input (e.g., a tap gesture on the "Enter” button or a double tap gesture in the handwriting input area).
  • an explicit selection input e.g., a tap gesture on one of the displayed recognition result
  • an implicit confirmation input e.g., a tap gesture on the "Enter” button or a double tap gesture in the handwriting input area.
  • the user has explicitly selected the top-ranked recognition result 810 using a tap gesture (as indicated by a contact 814 over the recognition result 810 in Figure 8A).
  • text of the recognition result 810 is inserted at the insertion point indicated by the cursor 813 in the text input area 808.
  • an implicit confirmation input causes the top- ranked recognition result to be entered into the text input area 808 without requiring the user to stop and select the top-ranked recognition result.
  • a well-designed implicit confirmation input improves text entry speed and reduces cognitive burden placed on the user during text composition.
  • the top-ranked recognition result of a current handwriting input is optionally tentatively displayed in the text input area 808.
  • the tentative text input shown in the text input area 808 is visually distinguished from other text input in the text input area, e.g., by a tentative input box surrounding the tentative text input.
  • the text shown in the tentative input box is not yet committed or provided to the associated application (e.g., the Notes application), and is automatically updated when the top-ranked recognition result is changed by the handwriting input module, e.g., in response to user revision of the current handwriting input.
  • Figures 9A-9B are flow charts of an exemplary process 900 for providing multi- script handwriting recognition on a user device.
  • the user device receives (902) a multi-script handwriting recognition model, the multi- script recognition model having been trained on spatially-derived features (e.g.,
  • the multi-script handwriting recognition model is (906) a single convolutional neural network having a single input plane and a single output plane, and includes spatially-derived features and respective weights for the spatially-derived features for differentiating characters of the at least three non-overlapping scripts represented in the multi-script training corpus.
  • the multi-script handwriting recognition model is (908) configured to recognize characters based on respective input images of one or more recognition units identified in the handwriting input, and respective spatially-derived features used for recognition are independent of respective stroke order, stroke direction, and continuity of strokes in the handwriting input.
  • the user device receives (908) a handwriting input from a user, the handwriting input including one or more handwritten strokes provided on a touch-sensitive surface coupled to the user device.
  • the handwriting input includes respective data on the location and movement of a contact between a finger or stylus and the touch-sensitive surface coupled to the user device.
  • the user device provides (910) in real-time one or more handwriting recognition results to the user based on the multi- script handwriting recognition model that has been trained on the spatially-derived features of the multi- script training corpus (912).
  • the user device segments (914) the user's handwriting input into one or more recognition units, each recognition unit including one or more of the handwritten strokes provided by the user.
  • the user device segments the user's handwriting input according to the shape, location, and size of the individual strokes made by the contact between the user's finger or stylus and the touch-sensitive surface of the user device.
  • the segmentation of the handwriting input further takes into account of the relative order, and relative position of the individual strokes made by the contact between the user's finger or stylus and the touch-sensitive surface of the user device.
  • the user's handwriting input is in a cursive writing style, and each continuous stroke in the handwriting input may correspond to multiple strokes in a recognized character in print form. In some embodiments, the user's handwriting input may include a continuous stroke spanning multiple recognized characters in printed form.
  • the segmentation of the handwriting input generates one or more input images each corresponding to a respective recognition unit. In some embodiments, some of the input images optionally include some overlapping pixels. In some embodiments, the input images do not include any overlapping pixels.
  • the user device generates a segmentation lattice, each segmentation chain of the segmentation lattice represents a respective way of segmenting the current handwriting input. In some embodiments, each arc in a segmentation chain corresponds to a respective group of strokes in the current handwriting input.
  • the user device provides (914) a respective image of each of the one or more recognition units as an input to the multi-script recognition model.
  • the user device obtains (916) from the multi-script handwriting recognition model at least a first output character from a first script and at least a second output from a second script different from the first script.
  • the same input image may cause the multi-script recognition model to output two or more similar looking output characters from different scripts as recognition results for the same input image.
  • the handwriting inputs for the letter "a" in the Latin script and the character "a" in the Greek script are often similar.
  • the handwriting inputs for the letter “J” in the Latin script and the Chinese character “ ⁇ " are often similar.
  • the handwriting input for the emoji character “ ⁇ ” may be similar to the handwriting input for the CJK radical '3 ⁇ 43 ⁇ 4".
  • the multi-script handwriting recognition model often produces multiple candidate recognition results that are likely correspond to the user' s handwriting input, because the visual appearance of the handwriting input would be difficult even for a human reader to decipher.
  • the first script is the CJK basic character block and the second script is the Latin script as encoded by the Unicode standard.
  • the first script is CJK basic character block
  • the second script is a set of emoji characters.
  • the first script is the Latin script
  • the second script is the emoji characters.
  • the user device displays (918) both the first output character and the second output character in a candidate display area of the handwriting input interface of the user device.
  • the user device selectively displays (920) one of the first output character and the second output character based on which one of the first and second scripts is a respective script used in a soft keyboard currently installed on the user device. For example, suppose the handwriting recognition model has identified the Chinese character "A” an d the Greek letter " ⁇ " as the output characters for the current handwriting input, the user device determines whether the user has installed a Chinese soft keyboard (e.g., a keyboard using the Pinyin input method) or the Greek input keyboard on the user device. If the user device determines that only the Chinese soft keyboard has been installed, the user device optionally displays only the Chinese character ' ⁇ " and not the Greek letter " ⁇ ” as the recognition result to the user.
  • a Chinese soft keyboard e.g., a keyboard using the Pinyin input method
  • the user device provides real-time handwriting recognition and input.
  • the user device continuously revises (922) one or more recognition results for the user's handwriting input in response to continued additions to or revisions of the handwriting input by the user, before the user makes an explicit or implicit selection of a recognition result displayed to the user.
  • the user in response to each revision of the one or more recognition results, displays (924) the respective revised one or more recognition results to the user in a candidate display area of the handwriting input user interface.
  • the multi-script handwriting recognition model is (926) trained to recognize all characters of at least three non-overlapping scripts including Chinese characters, emoji characters, and the Latin script encoded according to the Unicode standard.
  • the at least three non- overlapping scripts include Chinese characters, the Arabic script, and the Latin script.
  • the multi-script handwriting recognition model has (928) at least thirty thousand output classes, representing at least thirty characters spanning the at least three non-overlapping scripts.
  • the user device allows the user to enter a multi- script handwriting input, such as a phrase that includes characters in more than one script.
  • a multi-script handwriting input such as a phrase that includes characters in more than one script.
  • the user may write continuously and receives handwriting recognition results including characters in more than one script, without stopping in the middle of writing to manually switch the recognition language.
  • the user may write the multi-script sentence "Hello means 3 ⁇ 4 * 3 ⁇ 4' in Chinese.” in the handwriting input area of the user device, without having to switch the input language from English to Chinese before writing the
  • the multi- script handwriting recognition model is used to provide real-time handwriting recognition for a user's input.
  • the real-time handwriting recognition is used to provide real-time multi- script handwriting input functionality on a user's device.
  • Figures 1 OA- IOC are flow charts of an exemplary process 1000 for providing real-time handwriting recognition and input on a user device.
  • the real-time handwriting recognition is stroke-order independent on a character-level, a phrase level, and a sentence level.
  • stroke-order independent handwriting recognition on a character level requires that the handwriting recognition model provides the same recognition result for a particular handwritten character, regardless of the sequence by which the individual strokes of the particular character has been provided by the user.
  • Individual strokes of a Chinese character are typically written in a particular order.
  • native speakers of Chinese are often trained to write each character in a particular order in school, many users have later adopted personalized styles and stroke sequences that depart from the conventional stroke order.
  • cursive writing styles are highly individualized, and multiple strokes in a printed form of a Chinese character are often merged into a single stylized stroke that twists and turns, and sometimes even runs on to a next Character.
  • a stroke-order independent recognition model is trained based on images of writing samples that is free of temporal information associated with individual strokes. Therefore, the recognition is independent of stroke-order information. For example, for the Chinese character “ ⁇ the same recognition result " ⁇ "' " will be given by the handwriting recognition model regardless of whether the user wrote the horizontal stroke first or the vertical stroke first.
  • the user device receives (1002) a plurality of handwritten strokes from a user, the plurality of handwritten strokes
  • the handwriting input for the character " t " typically includes a substantially horizontal handwritten stroke intersecting a substantially vertical handwritten stroke.
  • the user device generates (1004) an input image based on the plurality of handwritten strokes.
  • the user device provides (1006) the input image to a handwriting recognition model to perform real-time handwriting recognition of the handwritten character, where the handwriting recognition model provides stroke-order independent handwriting recognition.
  • the user device displays (1008) in real-time of receiving the plurality of handwritten strokes, an identical first output character (e.g., the character " I " in printed form) irrespective of a respective order by which the plurality of handwritten strokes (e.g., the horizontal stroke and the vertical stroke) have been received from the user.
  • an identical first output character e.g., the character " I " in printed form
  • stroke-order independent handwriting recognition is performed independent of temporal information associated with individual strokes within each handwritten character.
  • stroke-order independent handwriting recognition is performed in conjunction with stroke-distribution information which takes into account of spatial distribution of individual strokes before they are merged into a flat input image. More details on how the temporally-derived stroke-distribution information is used to augment the stroke-order independent handwriting recognition described above are provided later in the specification (e.g., with respect to Figures 25A-27). The technique described with respect to Figures 25A-27 does not destroy the stroke-order independence of the handwriting recognition system.
  • the handwriting recognition model provides (1010) stroke-direction independent handwriting recognition. In some embodiments,
  • stroke-direction independent recognition requires that the user device displays in response to receiving the plurality of handwriting inputs, the identical first output character irrespective of a respective stroke direction by which each of the plurality of handwritten strokes have been provided by the user. For example, if the user has written the Chinese character " 1 " in the handwriting input area of the user device, the handwriting recognition model would output the same recognition result, regardless of whether the user has drawn the horizontal stroke from left to right or from right to left. Similarly, the handwriting recognition model would output the same recognition result, regardless of whether the user has drawn the vertical stroke in the downward direction or in the upward direction. In another example, many Chinese characters are structurally made of two or more radicals.
  • Some Chinese characters each include a left radical and a right radical, and people customarily write the left radical first, and the right radical second.
  • the handwriting recognition model would provide the same recognition result regardless of whether the user has written the right radical or the left radical first, as long as the resulting handwriting input shows the left radical to the left of the right radical when the user completes the handwritten character.
  • some Chinese characters each include a top radical and a bottom radical, and people customarily write the top radical first, and the bottom radical last.
  • the handwriting recognition model would provide the same recognition result regardless of whether the user has written the top radical or the bottom radical first, as long as the resulting handwriting input shows the top radical above the bottom radical. In other words, the handwriting recognition model does not rely on the directions by which the user provides the individual strokes of the handwritten character to determine the identity of the handwritten character.
  • the handwriting recognition model provides handwriting recognition based on the image of a recognition unit, regardless of the number of sub- strokes by which the recognition unit has been provided by the user.
  • the handwriting recognition model provides (1014) stroke-count independent handwriting recognition.
  • the user device displays in response to receiving the plurality of handwritten strokes, the identical first output character irrespective of how many handwritten strokes are used to form a continuous stroke in the input image.
  • the handwriting recognition model outputs the same recognition result, regardless of whether the user has provided four strokes (e.g., two short horizontal strokes and two short vertical strokes to make up the cross-shaped character), or two strokes (e.g., an L-shaped stroke and a 7-shaped stroke, or a horizontal stroke and a vertical stroke), or any other number of strokes (e.g., hundreds of extremely short strokes or dots) to make up the shape of the character" 1 ".
  • four strokes e.g., two short horizontal strokes and two short vertical strokes to make up the cross-shaped character
  • two strokes e.g., an L-shaped stroke and a 7-shaped stroke, or a horizontal stroke and a vertical stroke
  • any other number of strokes e.g., hundreds of extremely short strokes or dots
  • the handwriting recognition model is also capable of recognizing multiple characters regardless of the temporal order by which the strokes of the multiple characters have been provided by the user.
  • the user device has not only received the first plurality of handwritten strokes, but also received (1016) a second plurality of handwritten strokes from the user, where the second plurality of handwritten strokes correspond to a second handwritten character.
  • the user device generates (1018) a second input image based on the second plurality of handwritten strokes.
  • the user device provides (1020) the second input image to the handwriting recognition model to perform real-time recognition of the second handwritten character.
  • the user device displays (1022) in real-time of receiving the second plurality of handwritten strokes, a second output character corresponding to the second plurality of handwritten strokes.
  • the second output character and the first output character are concurrently displayed in a spatial sequence independent of a respective order by which the first plurality of handwritten strokes and the second plurality of handwritten strokes have been provided by the user. For example, if the user has written two Chinese characters (e.g., " ⁇ " " ' “ and” ' v ”) in the handwriting input area of the user device, the user device will display the recognition result " ⁇ ' A " regardless of whether the user has written strokes of the character " 1 " or the strokes of the character " first, as long as the handwriting input currently accumulated in the handwriting input area shows the strokes for the character " i " to the left of the strokes for the character " ⁇ ".
  • the spatial sequence of the first output character and the second output character corresponds (1024) to a spatial distribution of the first plurality of handwritten strokes and the second plurality of strokes along a default writing direction (e.g., from left to right) of a handwriting input interface of the user device.
  • the second plurality of handwritten strokes are received (1026) temporally after the first plurality of handwritten strokes, and the second output character precedes the first output character in a spatial sequence along a default writing direction (e.g., from left to right) of a handwriting input interface of the user device.
  • the handwriting recognition model provides stroke-order independent recognition on a sentence to sentence level. For example, even if the handwritten character " 1 " is in a first handwritten sentence and the handwritten character " A " is in a second handwritten sentence, and the two handwritten characters are separated by one or more other handwritten characters and/or words in the handwriting input area, the handwriting recognition model would still provide the recognition result showing the two characters in the spatial sequence" i ' ... The recognition result and the spatial sequence of the two recognized characters remain the same regardless of the temporal order by which the strokes of the two characters have been provided by the user, provided that the recognition units for the two characters are spatially arranged in the sequence " ...
  • the first handwritten character (e.g.,” 1 ") is provided by the user as part of a first handwritten sentence (e.g.,” t ' is a number.”
  • the second handwriting character (e.g., " A ) is provided by the user as part of a second handwritten sentence (e.g., " ⁇ is another number.") and the first and the second handwritten sentences are concurrently displayed in a handwriting input area of the user device.
  • the two sentences will be entered into a text input area of the user device, and the handwriting input area will be cleared for the user to enter another handwriting input.
  • the recognition result e.g., " i ' is a number
  • the handwriting recognition model is stroke-order independent not only on a character-level, but also on a phrase level and a sentence level, the user can make corrections to a previously uncompleted character after subsequent characters have been written. For example, if the user has forgotten to write a particular stroke for a character before moving on to write one or more subsequent characters in the handwriting input area, the user can still put down the missing stroke later at the right location in the particular character to receive the correct recognition result.
  • the user may also make a correction (e.g., add one or more strokes) to an earlier written character after having subsequently written one or more characters in the handwriting input interface.
  • a correction e.g., add one or more strokes
  • the user may also individually delete (e.g., using the methods described later with respect to Figures 21A- 22B) an earlier written character and rewrite it in the same location in the handwriting input interface.
  • the second plurality of handwritten strokes spatially follow (1028) the first plurality of handwritten strokes along a default writing direction of the handwriting input interface of the user device, and the second output character follows the first output character in a spatial sequence along the default writing direction in the candidate display area of the handwriting input interface.
  • the user device receives (1030) a third handwritten stroke from the user to revise the first handwritten character (i.e., the handwritten character formed by the first plurality of handwritten strokes), the third handwritten stroke being received temporally after the first and the second pluralities of handwritten strokes.
  • the user has written two characters in a spatial sequence from left to right (e.g., " A.i&") in the handwriting input area.
  • the first plurality of strokes form a handwritten character " ⁇ .”
  • the second plurality of strokes form the handwritten character
  • the user later realizes that he wished to write " ⁇ 4" instead of " A# : " the user can simply put one more vertical stroke below the strokes for the character " and the user device will assign the vertical stroke to the first recognition unit (e.g., the recognition unit for " ").
  • the user device will output a new output character (e.g., " A") for the first recognition unit, where the new output character will replace the previous output character (e.g., " A ") in the recognition result.
  • the user device assigns (1032) the third handwritten stroke to the same recognition unit as the first plurality of handwritten strokes based on relative proximity of the third handwritten stroke to the first plurality of handwritten strokes.
  • the user device generates (1034) a revised input image based on the first plurality of handwritten stroke and the third handwritten stroke.
  • the user device provides (1036) the revised input image to the handwriting recognition model to perform real-time recognition of the revised handwritten character.
  • the user device displays (1040) in response to receiving the third handwriting input, a third output character corresponding to the revised input image, where the third output character replaces the first output character and is concurrently displayed with the second output character in the spatial sequence along the default writing direction.
  • the handwriting recognition module recognizes the handwriting input written in a default writing direction from left to right. For example, the user can write characters from left to right, and in one or more rows.
  • the handwriting input module presents recognition results that include characters in a spatial sequence from left to right, and in one or more rows as necessary. If the user selects a recognition result, the selected recognition result is inputted into the text input area of the user device.
  • the default writing direction is from top to bottom. In some embodiments, the default writing direction is from right to left.
  • the user optionally changes the default writing direction to an alternative writing direction after a recognition result has been selected and the handwriting input area has been cleared.
  • the handwriting input module allows the user to enter multiple-character handwriting input in the handwriting input area and allows deletion of strokes from the handwriting input one recognition unit at a time, rather than all recognition units at once.
  • the handwriting input module allows deletion from the handwriting input one stroke at a time.
  • the deletion of recognition unit proceeds one by one in the direction opposite to the default writing direction, regardless of the order by which the recognition units or strokes have been entered to produce the current handwriting input.
  • the deletion of strokes proceeds one by one in the reverse order it has been entered within each recognition unit, and when the strokes in one recognition unit have all been deleted, the deletion of strokes proceed to the next recognition unit in the direction opposite to the default writing direction.
  • the user device receives a deletion input from the user.
  • the user device deletes the second output character from the recognition result, while maintaining the third output character in the recognition result displayed in the candidate display area.
  • the user device renders (1042) in real-time the first plurality of handwritten strokes, the second plurality of handwritten strokes, and the third handwritten stroke as each of said handwritten stroke is provided by the user.
  • the user device in response to receiving the deletion input from the user, deletes (1044) a respective rendering of the second plurality of handwriting inputs (e.g., corresponding to the second handwritten character) from the handwriting input area, while maintaining respective renderings of the first plurality of handwritten strokes and the third handwritten stroke (e.g., together corresponding to the revised first handwritten character) in the handwriting input area.
  • the strokes in the recognition unit for the character " is removed from the handwriting input area, and the character “l f >” is removed from the recognition result " ⁇ " in the candidate display area of the user device.
  • the strokes for the character " " remain in the handwriting input area, while the recognition result shows only the character " " r".
  • the handwritten character is a multi-stroke Chinese character.
  • the first plurality of handwriting input is provided in a cursive writing style.
  • the first plurality of handwriting input is provided in a cursive writing style and the handwritten character is a multi- stroke Chinese character.
  • the handwritten characters are written in Arabic in a cursive style. In some embodiments, the handwritten characters are written in other scripts in a cursive style.
  • the user device establishes respective predetermined constraints on a set of acceptable dimensions for a handwritten character input, and segments a currently accumulated plurality of handwritten strokes into a plurality of recognition units based on the respective predetermined constraints, where a respective input image is generated from each of the recognition units, provided to the handwriting recognition model, and recognized as a corresponding output character.
  • the user device receives an additional handwritten stroke from the user after having segmented the currently accumulated plurality of handwritten strokes.
  • the user device assigns the additional handwritten stroke to a respective one of the plurality of recognition units based on a spatial position of the additional handwritten stroke relative to the plurality of recognition units.
  • the exemplary user interfaces are provided on a user device based on a multi-script handwriting recognition model that provides real-time, stroke-order independent handwriting recognition of a user's handwriting input.
  • the exemplary user interfaces are user interfaces of an exemplary handwriting input interface 802 (e.g., shown in Figures 8A and 8B), including a handwriting input area 804, a candidate display area 804, and a text input area 808.
  • the exemplary handwriting input interface 802 also includes a plurality of control elements 1102, such as a deletion button, a space bar, an Enter button, a keyboard switching button, etc.
  • control elements 1102 such as a deletion button, a space bar, an Enter button, a keyboard switching button, etc.
  • One or more other areas and/or elements may be provided in the handwriting input interface 802 to enable additional functionalities described below.
  • the multi- script handwriting recognition model is capable of having a very large repertoire of tens of thousands of characters in many different scripts and languages. As a result, it is very likely that for a handwriting input, the recognition model will identify a large number of output characters that all have a reasonably good likelihood of being the character intended by the user. On a user device having a limited display area, it is advantageous to provide only a subset of the recognition results initially, while keeping the other results available upon user request.
  • Figures 1 lA-11G show exemplary user interfaces for displaying a subset of the recognition results in a normal view of the candidate display area, along with an affordance for invoking an extended view of the candidate display area for display the remainder of the recognition results.
  • the recognition results are divided into different categories, and displayed on different tabbed pages in the extended view.
  • Figure 11 A shows an exemplary handwriting input interface 802.
  • the handwriting input interface includes a handwriting input area 804, a candidate display area 806, and a text input area 808.
  • One or more control elements 1102 are also included in the handwriting input interface 1002.
  • the candidate display area 806 optionally includes an area for displaying one or more recognition results and an affordance 1104 (e.g., an expansion icon) to invoke an extended version of the candidate display area 806.
  • an affordance 1104 e.g., an expansion icon
  • Figures 11 A- 11C illustrate that, as the user provides one or more handwritten strokes in the handwriting input area 804 (e.g., strokes 1106, 1108, and 1110), the user device identifies and displays a respective set of recognition results corresponding to the currently accumulated strokes in the handwriting input area 804. As shown in Figure 1 IB, after the user has entered a first stroke 1106, the user device identifies and displays three recognition results 1112, 1114, and 1116 (e.g., characters "/", "1", and ",”). In some embodiments, the small number of candidate characters are displayed in the candidate display area 806 in an order according to the recognition confidence associated with each character.
  • recognition results 1112, 1114, and 1116 e.g., characters "/", "1", and ".”
  • the top-ranked candidate result (e.g., "/") is tentatively displayed in the text input area 808, e.g., within a box 1118.
  • the user can optionally confirm that the top-ranked candidate is the intended input with a simple confirmation input (e.g., by pressing the "Enter” key, or providing a double-tap gesture in the handwriting input area).
  • Figure 11C illustrates that, before the user has selected any candidate recognition result, as the user enters two more strokes 1108 and 1110 in the handwriting input area 804, the additional strokes are rendered in the handwriting input area 804 with the initial stroke 1106, and the candidate results are updated to reflect the changes to the recognition unit(s) identified from the currently accumulated handwriting inputs.
  • the user device based on the three strokes, the user device has identified a single recognition unit. Based on the single recognition unit, the user device has identified and displayed a number of recognition results 1118-1124.
  • one or more of the recognition results (e.g., 1118 and 1122) currently displayed in the candidate display area 806 are each a representative candidate character selected from among multiple similar- looking candidate characters for the current handwriting input.
  • the candidate display area changes from a normal view (e.g., shown in Figure 11C) to an extended view (e.g., shown in Figure 1 ID).
  • the extended view shows all of the recognition results (e.g., candidate characters) that have been identified for the current handwriting input.
  • the initially displayed normal view of the candidate display area 806 only shows the most commonly used characters used in a respective script or language, while the extended view shows all candidate characters, including characters that are rarely used in a script or language.
  • the extended view of the candidate display area may be designed in different ways.
  • Figures 1 lD-11G illustrate an exemplary design of the extended candidate display area in accordance with some embodiments.
  • the extended candidate display area 1128 includes one or more tabbed pages (e.g., pages 1130, 1132, 1134, and 1136) each presenting a respective category of candidate characters.
  • the tabbed design shown in Figure 1 ID allows the user to quickly find the desired category of characters, and then find the character that he or she intended to enter within the corresponding tabbed page.
  • the first tabbed page 1130 displays all candidate characters that have been identified for the currently accumulated handwriting input, including both commonly used characters as well as rare characters. As shown in Figure 1 ID, the tabbed page 1130 includes all of the characters shown in the initial candidate display area 806 in Figure 11C, and a number of additional characters (e.g., "'-' ⁇ ' ", " ⁇ ” ; " " ; e t c .) that were not included in the initial candidate display area 806.
  • the characters displayed in the initial candidate display area 806 include only characters from a set of commonly used characters associated with a script (e.g., all characters in the basic block of the CJK script, as encoded according to the Unicode standard).
  • the characters displayed in the extended candidate display area 1128 further include a set of rare characters associated with the script (e.g., all characters in the extended block of the CJK script, as encoded according to the Unicode standard).
  • the extended candidate display area 1128 further includes candidate characters from other scripts that are not commonly used by the user, e.g., the Greek script, the Arabic script, and/or the emoji script.
  • the extended candidate display area 1128 includes respective tabbed pages 1130, 1132, 1134, and 1138 each corresponding to a respective category of candidate characters (e.g., all characters, rare characters, characters from the Latin script, and characters from the emoji script, respectively).
  • a respective category of candidate characters e.g., all characters, rare characters, characters from the Latin script, and characters from the emoji script, respectively.
  • Figures 1 lE-11G illustrate that the user can select each of the different tabbed pages to reveal the candidate characters in a corresponding category.
  • Figure 1 IE shows only the rare characters (e.g., characters from the extended block of the CJK script) that correspond to the current handwriting input.
  • Figure 1 IF shows only Latin and Greek letters that correspond to the current handwriting input.
  • Figure 11G shows only emoji characters that correspond to the current handwriting input.
  • the extended candidate display area 1128 further includes one or more affordances to sort the candidate characters in a respective tabbed page based on a respective criterion (e.g., based on phonetic spelling, based on number of strokes, and based on radicals, etc.).
  • a respective criterion e.g., based on phonetic spelling, based on number of strokes, and based on radicals, etc.
  • Figures 1 lH-1 IK illustrate that, in some embodiments, similar-looking candidate characters may be grouped, and only a representative character from each group of similar-looking candidate characters are presented in the initial candidate display area 806. Since the multi-script recognition model described herein can produce many candidate characters that are almost equally good for a given handwriting input, the recognition model cannot always eliminate one candidate at the expense of another similar-looking candidate. On a device having a limited display area, displaying many similar-looking candidates all at once is not helpful for the user to pick out the correct character, because the fine distinction may not be easy to see, and even if the user can see the desired character, it may be difficult to select it from a very crammed display using a finger or stylus.
  • the user device identifies candidate characters that have a great resemblance to one another (e.g., according to a concordance or dictionary of similar-looking characters, or some image-based criteria), and group them into a respective group.
  • one or more groups of similar-looking characters may be identified from a set of candidate characters for a given handwriting input.
  • the user device identified a representative candidate character from among a plurality of similar-looking candidate characters in the same group, and only displays the representative candidate in the initial candidate display area 806. If a commonly used character does not look similar enough to any other candidate characters, it is displayed by itself.
  • the representative candidate character (e.g., candidate characters 1118 and 1122, " t°' and "T") of each group is displayed in a different manner (e.g., in a bolded box) from a candidate character (e.g., candidate characters 1120 and 1124, " ⁇ 3 ⁇ 4" and "J") that does not belong to any group.
  • the criterion for choosing the representative character of a group is based on a relative usage frequency of the candidate characters in the group. In some embodiments, other criteria may be used.
  • the user can optionally expand the candidate display area 806 to show the
  • the selection of a particular representative character can cause the expanded view of only those candidate characters in the same group as the selected representative character.
  • Figures 1 lH-1 IK illustrate one embodiment in which the expanded view of a representative candidate character is invoked by a predetermined gesture (e.g., an expand gesture) detected over a representative candidate character (e.g., representative character 1118).
  • the predetermined gesture e.g., an expand gesture
  • the predetermined gesture for invoking an expanded view is different from the predetermined gesture (e.g., a tap gesture) for selecting the representative character for text input.
  • the second character e.g., "T”
  • the selected character e.g., "7 "
  • Figure 1 IK once a character has been selected, the handwriting input in the handwriting input area 804 and the candidate characters in the candidate display area 806 (or the extended view of the candidate display area) are cleared for subsequent handwriting input.
  • the user can optionally use the same gesture to expand other representative characters displayed in the candidate display area 806.
  • expanding another representative character in the candidate display area 806 automatically restores the currently presented expanded view to a normal view.
  • the user optionally uses a contraction gesture to restore the currently expanded view to a normal view.
  • the user can scroll the candidate display area 806 (e.g., to the left or to the right) to reveal other candidate characters that are not visible in the candidate display area 806.
  • Figures 12A-12B are flow charts of an exemplary process 1200 in which a first subset of recognition results are presented in an initial candidate display area, while a second subset of recognition results are presented in an extended candidate display area that is hidden from view until specifically invoked by a user.
  • the device identifies from a plurality of handwriting recognition results for a handwriting input, a subset of recognition results that have a visual similarity level exceeding a predetermined threshold.
  • the user device selects a representative recognition result from the subset of recognition results, and displays the selected representative recognition result in a candidate display area of the display.
  • the process 1200 is illustrated in Figures 1 lA-1 IK.
  • the user device receives (1202) a handwriting input from a user.
  • the handwriting input includes one or more handwritten strokes (e.g., 1106, 1108, 1110 in Figure 11C) provided in a handwriting input area (e.g., 806 in Figure 11C) of a handwriting input interface (e.g., 802 in Figure 11C).
  • the user device based on a handwriting recognition model, identifies (1204) a plurality of output characters (e.g., characters shown in tabbed page 1130, Figure 11C) for the handwriting input.
  • the user device divides (1206) the plurality of output characters into two or more categories based on a predetermined categorization criterion. In some
  • the predetermined categorization criterion determines (1208) whether a respective character is a commonly used character or a rare character.
  • the user device displays (1210), in an initial view of a candidate display area (e.g., 806 as shown in Figure 11C) of the handwriting input interface, respective output characters in a first category (e.g., commonly used characters) of the two or more categories, wherein the initial view of the candidate display area is concurrently provided with an affordance (e.g., 1104 in Figure 11C) for invoking an extended view of the candidate display area (e.g., 1128 in Figure 1 ID).
  • a candidate display area e.g., 806 as shown in Figure 11C
  • an affordance e.g., 1104 in Figure 11C
  • an extended view of the candidate display area e.g., 1128 in Figure 1 ID
  • the user device receives (1212) a user input selecting the affordance for invoking the extended view, e.g., as shown in Figure 11C.
  • the user device displays (1214), in the extended view of the candidate display area, the respective output characters in the first category and respective output characters in at least a second category of the two or more categories which were not previously displayed in the initial view of the candidate display area, e.g., as shown in Figure 1 ID.
  • the respective characters in the first category are characters found in a dictionary of commonly used characters
  • respective characters in the second category are characters found in a dictionary of rare characters.
  • the dictionary of commonly used characters and the dictionary of rare characters are dynamically adjusted or updated based on a usage history associated with the user device.
  • the user device identifies (1216) from the plurality of output characters, a group of characters that are visually similar to one another in accordance with a predetermined similarity criterion (e.g., based on a dictionary of similar characters or based on some spatially-derived features).
  • the user device selects from the group of visually similar characters a representative character based on a predetermined selection criterion (e.g., based on historic usage frequency).
  • the predetermined selection criterion is based on a relative usage frequency of the characters in the group.
  • the predetermined selection criterion is based on a preferred input language associated with the device.
  • the representative candidate is selected based on other factors indicative of the likelihood that each candidate is the intended input by the user. These factors include, for example, whether the candidate character belongs to a script for a soft keyboard that is currently installed on the user's device, or whether the candidate character is among a set of most commonly used characters in a particular language associated with the user or user device, etc.
  • the user device displays (1220) the representative character (e.g., " 1 ") in the initial view of the candidate display area (e.g., 806 in Figure
  • the user device receives (1222), from the user, a predetermined expansion input (e.g., an expand gesture) directed to the representative character displayed in the initial view of the candidate display area, e.g., as shown in Figure 11H.
  • a predetermined expansion input e.g., an expand gesture
  • the user device in response to the receipt of the predetermined expansion input, concurrently displays (1224) a magnified view of the representative character and respective magnified view of one or more other characters in the group of visually similar characters, e.g., as shown in Figure 111.
  • the predetermined expansion input is an expand gesture detected over the representative character displayed in the candidate display area.
  • the predetermined expansion input is a contact that is detected over the representative character displayed in the candidate display area and that is sustained for longer than a predetermined threshold time.
  • the sustained contact for expanding the group has a longer threshold duration than a tap gesture of selecting the representative character for text input.
  • each representative character is displayed concurrently with a respective affordance (e.g., a respective expand button) to invoke the extended view of its group of similar-looking candidate characters.
  • a respective affordance e.g., a respective expand button
  • predetermined expansion input is a selection of the respective affordance associated with the representative character.
  • the repertoire of the multi-script handwriting recognition model includes an emoji script.
  • the handwriting input recognition module can recognizes an emoji character based on a user's handwriting input.
  • the handwriting recognition module presents both emoji characters identified directly from the handwriting, and a character or word in a natural human language representing the identified emoji character.
  • the handwriting input module recognizes a character or word in a natural human language based on a user's handwriting input, and presents both the recognized character or word, and an emoji character that corresponds to the recognized character or word.
  • the handwriting input module provides ways for entering an emoji character without switching from the handwriting input interface to an emoji keyboard.
  • the handwriting input module also provides a way to enter regular natural language characters and words, by drawing an emoji character by hand.
  • Figures 13A-13E provide exemplary user interfaces illustrating these different ways of entering emoji characters and regular natural language characters.
  • FIG. 13A shows an exemplary handwriting input interface 802 invoked under a chat application.
  • the handwriting input interface 802 includes a handwriting input area 804, a candidate display area 806, and a text input area 808.
  • the user can choose to send the text composition to another participant of the current chat session.
  • the dialog history of the chat session is shown in the dialogue panel 1302.
  • the user has received a chat message 1304 (e.g., "Happy Birthday 3 ⁇ 4 3 ") which is displayed in the dialogue panel 1302.
  • the user has provided a handwriting input 1306 for the English word "Thanks” in a handwriting input area 804.
  • the user device has identified a number of candidate recognition results (e.g., recognition results 1308, 1310, and 1312).
  • the top-ranked recognition result 1303 has been tentatively entered into the text input area 808 within a box 1314.
  • the user device Based on this recognized emoji character, the user device presents a first recognition result 1318 (e.g., "Thanks! with the stylized "!) in the candidate display area 806. In addition, the user device also identifies a number "8" which also visually resembles the newly entered recognition unit. Based on this recognized digit, the user device presents a second recognition result 1322 (e.g., "Thanks 8") in the candidate display area 806. In addition, based on the identified emoji character (e.g., the stylized "!), the user device also identifies a regular character (e.g., a regular character" !) corresponding to the emoji character.
  • a regular character e.g., a regular character" !
  • the user device Based on this indirectly identified regular character, the user device presents a third recognition result 1320 (e.g., "Thanks! with the regular”!) in the candidate display area 806. At this point, the user may select any one of the candidate recognition results 1318, 1320, and 1322 to enter it into the text input area 808.
  • a third recognition result 1320 e.g., "Thanks! with the regular”!”
  • the user continues to provide additional handwritten strokes 1324 in the handwriting input area 806. This time, the user has drawn a heart symbol following the stylized exclamation point.
  • the user device recognizes that the newly provided handwritten strokes 1324 form yet another new recognition unit. Based on the new recognition unit, the user device identifies the emoji character " ⁇ " and, alternatively, the digit "0" as the candidate characters for the new recognition unit. Based on these new candidate characters recognized from the new recognition unit, the user device presents two updated candidate recognition results 1326 and 1330 (e.g., "Thanks and "Thanks 80").
  • the user device further identifies the regular character(s) or word(s) (e.g., "Love") that correspond to the identified emoji character (e.g., "*"). Based on the identified regular character(s) or word(s) for the recognized emoji character, the user device presents a third recognition result 1328 in which the recognized emoji character(s) are replaced with the corresponding regular character(s) or word(s). As shown in Figure 13D, in the recognition result 1328, the emoji character "* " is replaced with a normal exclamation point "!, and the emoji character has been replaced with regular characters or word "Love".
  • the regular character(s) or word(s) e.g., "Love
  • the user has selected one of the candidate recognition results (e.g., candidate result 1326 showing the mixed-script text "Thanks- *"), and text of the selected recognition result is entered into the text input area 808, and subsequently sent to the other participant of the chat session.
  • the message bubble 1332 shows the text of the message in the dialogue panel 1302.
  • Figure 14 is a flow chart of an exemplary process 1400 in which the user enters an emoji character using a handwriting input.
  • Figures 13A- 13E illustrates the exemplary process 1400 in accordance with some embodiments.
  • the user device receives (1402) a handwriting input from a user.
  • the handwriting input includes a plurality of handwritten strokes provided in an handwriting input area of a handwriting input interface.
  • the user device recognizes (1404), based on a handwriting recognition model, a plurality of output characters from the handwriting input.
  • the output characters include at least a first emoji character (e.g., the stylized exclamation point " or the emoji character " "in Figure 13D) and at least a first character (e.g., a character from the word "Thanks” in Figure 13D) from a script of a natural human language.
  • the user device displays (1406) a recognition result (e.g., result 1326 in Figure 13D) comprising the first emoji character (e.g., the stylized exclamation point - " or the emoji character " ⁇ ” in Figure 13D) and the first character (e.g., a character from the word "Thanks” in Figure 13D) from the script of the natural human language in a candidate display area of the handwriting input interface, e.g., as shown in Figure 13D.
  • a recognition result e.g., result 1326 in Figure 13D
  • the first emoji character e.g., the stylized exclamation point - " or the emoji character " ⁇ ” in Figure 13D
  • the first character e.g., a character from the word "Thanks” in Figure 13D
  • the user device based on the handwriting recognition model, the user device optionally recognizes (1408) at least a first semantic unit (e.g., the word "thanks") from the handwriting input, wherein the first semantic unit comprises a respective character, word or phrase that is capable of conveying a respective semantic meaning in a respective human language.
  • the user device identifies (1410) a second emoji character (e.g., a "handshake” emoji character) associated with the first semantic unit (e.g., the word "Thanks”) recognized from the handwriting input.
  • the user device displays (1412), in the candidate display area of the handwriting input interface, a second recognition result (e.g., a recognition result showing a "handshake” emoji character followed by the " « " and a emoji characters) comprising at least the second emoji character identified from the first semantic unit (e.g., the word "Thanks”).
  • displaying the second recognition result further includes concurrently displaying the second recognition result with a third recognition result (e.g., a recognition result "Thanks- *”) comprising at least the first semantic unit (e.g., the word "Thanks").
  • the user receives a user input selecting the first recognition result displayed in the candidate display area.
  • the user device in response to the user input, enters text of the selected first recognition result in a text input area of the handwriting input interface, where the text includes at least the first emoji character and the first character from the script of the natural human language.
  • the user is able to enter a mixed-script text input using a single handwriting input (albeit, a handwriting input comprising multiple strokes) in the handwriting input area, without switching between a natural language keyboard and an emoji character keyboard.
  • the handwriting recognition model has been trained on a multi-script training corpus comprising writing samples corresponding to characters of at least three non-overlapping scripts, and the three non-overlapping scripts includes a set of emoji characters, Chinese characters, and Latin script.
  • the user device identifies (1414) a second semantic unit (e.g., the word "Love") corresponding to the first emoji character (e.g., the emoji character) recognized directly from the handwriting input.
  • the user device displays (1416) in the candidate display area of the handwriting input interface, a fourth recognition result (e.g., 1328 in Figure 13D) comprising at least the second semantic unit (e.g., the word "Love") identified from the first emoji character (e.g., the " ⁇ " emoji character).
  • the user device concurrently displays the fourth recognition result (e.g., result 1328 "Thanks! Love") with the first recognition result (e.g., result "Thanks* *”) in the candidate display area, as shown in Figure 13D.
  • the user device allows the user to enter regular text by drawing an emoji character. For example, if the user does not know how to spell the word "elephant,” the user optionally draw a stylized emoji character for "elephant” in the handwriting input area, and if the user device can correctly recognize the handwriting input as an emoji character for "elephant,” the user device optionally also presents the word "elephant” in normal text as one of the recognition results displayed in the candidate display area. In another example, the user may draw a stylized cat in the handwriting input area, in lieu of writing the Chinese character "S3 ⁇ 4".
  • the user device If the user device identifies the emoji character for "cat” based on the handwriting input provided by the user, the user device optionally also presents the Chinese character which means “cat” in Chinese, along with the emoji character for "cat” in the candidate recognition results.
  • the user device By presenting normal text for a recognized emoji character, the user device provides an alternative way of entering complex characters or words using a few stylized strokes commonly associated with a well-known emoji character.
  • the user device stores a dictionary linking emoji characters with their corresponding normal text (e.g., characters, words, phrases, symbols, etc.) in one or more preferred scripts or languages (e.g., English or Chinese).
  • the user device recognizes an emoji character based on a visual resemblance of the emoji character to an image generated from the handwriting input.
  • the handwriting recognition model used on the user device is trained using a training corpus that include both handwriting samples corresponding to characters of a script of a natural human language and also handwriting samples corresponding to a set of artificially designed emoji characters.
  • the emoji characters related to the same semantic concept may have different appearance when used in a mixed input with text of different natural languages.
  • an emoji character for the semantic concept of "Love” may be a "heart” emoji character when presented with normal text of one natural language (e.g., Japanese), and a "kiss” emoji character when presented with normal text of another natural language (e.g., English or French).
  • the handwriting input module when performing recognition of a multi-character handwriting input, performs segmentation of the handwriting input currently accumulated in the handwriting input area, and divides the accumulated strokes into one or more recognition units.
  • One of the parameters used to determine how to segment a handwriting input may be how the strokes are clustered in the handwriting input area and the distance between the different clusters of strokes. Since people have different writing styles. Some people tend to write very sparsely with large distances between strokes or different parts of the same character, while other people tend to write very densely with very small distances between strokes or different characters.
  • a handwritten character may depart from a balanced appearance and be lopsided, stretched, or squeezed in different ways.
  • the multi-script handwriting recognition model provides stroke-order independent recognition, therefore, the user may write characters or parts of characters out of sequence. As a result, spatial uniformity and balance in a handwriting input between characters may be difficult to attain.
  • the handwriting input model described herein provides a way for the user to inform the handwriting input module whether to merge two adjacent recognition units into a single recognition unit or to divide a single recognition unit into two separate recognition units. With the user's help, the handwriting input module can revise the initial segmentation, and generate a result intended by the user.
  • Figures 15A-15J illustrate some exemplary user interfaces and processes in which the user provides a predetermined pinch and expand gestures to modify the recognition units identified by the user device.
  • a user has entered a plurality of handwritten strokes 1502 (e.g., three strokes) in the handwriting input area 806 of a handwriting input interface 802.
  • the user device has identified a single recognition unit based on the currently accumulated handwritten strokes 1502, and presented three candidate characters 1504,
  • Figure 15C shows that the user has further entered a few additional strokes 1510 to the right of the initial handwritten strokes 1502 in the handwriting input area 606.
  • the user device determines (e.g., based on the dimensions and spatial distributions of the plurality of strokes 1502 and 1510) that the strokes 1502 and the strokes 1510 should be considered as two separate recognition units.
  • the user device Based on the division of the recognition units, the user device provides the input images of the first and second recognition units to the handwriting recognition model and obtains two sets of candidate characters.
  • the user device then generates a plurality of recognition results (e.g., 1512, 1514, 1516, and 1518) based on different combinations of the recognized characters.
  • Each recognition result includes a recognized character for the first recognition unit, and a recognized character for the second recognition unit.
  • the plurality of recognition results 1512, 1514, 1516, and 1518 each include two recognized characters.
  • the user in fact intended the handwriting input to be recognized as a single character, but unintentionally left too much space between the left portion (e.g., the left radical and the right portion (e.g., the right radical "W " ) of the handwritten character (e.g., Having seen the results (e.g., 1512, 1514, 1516, and
  • the user would realize that the user device has incorrectly segmented the current handwriting input into two recognition units. Although the segmentation may be based on an objective standard, it would not be desirable for the user to delete the current handwriting input and rewrite the whole character again with a smaller distance between the left and the right portions.
  • the user uses a pinch gesture over the two clusters of the handwritten strokes 1502 and 1510 to indicate to the handwriting input module that the two recognition units identified by the handwriting input module should be merged as a single recognition unit.
  • the pinch gesture is indicated by two contacts 1520 and 1522 on the touch-sensitive surface that are move toward each other.
  • Figure 15E shows that, in response to the user's pinch gesture, the user device has revised the segmentation of the currently accumulated handwriting input (e.g., strokes 1502 and 1510), and merged the handwritten strokes into a single recognition unit.
  • the user device has provided the input image based on the revised recognition unit to the handwriting recognition model, and obtained three new candidate characters 1524, 1526, and 1528 (e.g., -I'M " . " 1", and "H") for the revised recognition unit.
  • the user device optionally adjusts the rendering of the handwriting input within the handwriting input area 806, such that the distance between the left cluster and the right cluster of the handwritten strokes is reduced.
  • the user device does not alter the rendering of the handwriting input shown in the handwriting input area 608 in response to the pinch gesture. In some embodiments, the user device distinguishes the pinch gesture from an input stroke based on the two simultaneous contacts (as opposed to one single contact) detected in the
  • the user has entered two more strokes 1530 to the right of the previously entered handwriting input (i.e., the strokes for the character " ").
  • the user device determines that the newly entered strokes 1530 is a new recognition unit, and recognizes a candidate character (e.g., " f ") for the newly identified recognition unit.
  • the user device then combines the newly identified character (e.g., " F ") with the candidate characters for the earlier identified recognition unit, and presents a number of different recognition results (e.g., results 1532 and 1534) in the candidate display area 806.
  • the user continues to write more strokes 1536 (e.g., three more strokes) to the right of the strokes 1530, as shown in Figure 15G. Since the horizontal distance between the strokes 1530 and the strokes 1536 is very small, the user device determines that the strokes 1530 and the strokes 1536 belong to the same recognition unit, and provides an input image formed by the strokes 1530 and 1536 to the handwriting recognition model.
  • the handwriting recognition model identifies three different candidate characters for the revised recognition unit, and generates two revised recognition results 1538 and 1540 for the currently accumulated handwriting input.
  • Figure 151 shows that, in response to the user's expand gesture, the user device revised the previous segmentation of the currently accumulated handwriting input, and assigned the strokes 1530 and the strokes 1536 into two consecutive recognition units. Based on the input images generated for the two separate recognition units, the user device identifies one or more candidate characters for the first recognition unit based on the strokes 1530, and one or more candidate characters for the second recognition unit based on the strokes 1536. The user device then generates two new recognition results 1546 and 1548 based on different combinations of the recognized characters. In some embodiments, the user device optionally modifies the rendering of the strokes 1536 and 1536 to reflect the division of the previously identified recognition unit.
  • the user has selected (as indicated by the contact 1550) one of the candidate recognition results displayed in the candidate display area 806, and the selected recognition result (e.g., result 1548) has been entered in the text input area 808 of the user interface.
  • the candidate display area 806 and the handwriting input area 804 are both cleared and ready to display subsequent user input.
  • Figures 16A-16B are flow charts of an exemplary process 1600 in which the user uses predetermined gesture (e.g., a pinch gesture and/or an expand gesture) to notify the handwriting input module how to segment or revise an existing segmentation of the current handwriting input.
  • Figures 15J and 15K provide an illustration of the exemplary process 1600 in accordance with some embodiments.
  • the user device receives (1602) a handwriting input from a user.
  • the handwriting input includes a plurality of handwritten strokes provided in a touch-sensitive surface coupled to the device.
  • the user device renders (1604), in real-time, the plurality of handwritten strokes in a handwriting input area (e.g., handwriting input area 806 of Figures 15A-15K) of a handwriting input interface.
  • the user device receiving one of a pinch gesture input and a expand gesture input over the plurality of handwritten strokes, e.g., as shown in Figure 15D and 15H.
  • the user device upon receiving a pinch gesture input, the user device generates (1606) a first recognition result based on the plurality of handwritten strokes by treating the plurality of handwritten strokes as a single recognition unit, e.g., as illustrated in Figures 15C-15E.
  • the user device upon receiving a expand gesture input, the user device generates (1608) a second recognition result based on the plurality of handwritten strokes by treating the plurality of handwritten strokes as two separate recognition units pulled apart by the expand gesture input, e.g., as illustrated in Figures 15G-15I.
  • the user device upon generating a respective one of the first and second recognition results, displays the generated recognition result in a candidate display area of the handwriting input interface, e.g., as shown in Figure 15E, and 151.
  • the pinch gesture input comprises two simultaneous contacts on the touch- sensitive surface that converge toward each other in an area occupied by the plurality of handwritten strokes.
  • the expand gesture input comprises two simultaneous contacts on the touch- sensitive surface that diverge from each other in an area occupied by the plurality of handwritten strokes.
  • the user device identifies (e.g., 1614) two adjacent recognition units from the plurality of handwritten strokes.
  • the user device displays (1616), in the candidate display area, an initial recognition result (e.g., results 1512, 1514, 1516, and 1518 in Figure 15C) comprising respective characters recognized from the two adjacent recognition units, e.g., as illustrated in Figure 15C.
  • an initial recognition result e.g., results 1512, 1514, 1516, and 1518 in Figure 15C
  • the user device when displaying the first recognition result (e.g., result 1524, 1526, or 1528 in Figure 15E) in response to a pinch gesture, the user device replaces (1618) the initial recognition result with the first recognition result in the candidate display area.
  • the user device receives (1620) the pinch gesture input while the initial recognition result is displayed in the candidate display area, as shown in Figure 15D.
  • the user device in response to the pinch gesture input, re-renders (1622) the plurality of handwritten strokes to reduce a distance between the two adjacent recognition units in the handwriting input area, e.g., as shown in figure 15E.
  • the user device identifies (1624) a single recognition unit from the plurality of handwritten strokes.
  • the user device displays (1626), in the candidate display area, an initial recognition result (e.g., result 1538 or 1540 of Figure 15G) comprising a character (e.g., ⁇ ' " " (3 ⁇ 4:'”) recognized from the single recognition unit.
  • an initial recognition result e.g., result 1538 or 1540 of Figure 15G
  • a character e.g., ⁇ ' " " (3 ⁇ 4:'
  • the user device when displaying the second recognition result (e.g., result 1546 or 1548 in Figure 151) in response to the expand gesture, replaces (1628) the initial recognition result (e.g., results 1538 or 1540) with the second recognition result (e.g., result 1546 or 1548) in the candidate display area, e.g., as illustrated in Figures 15H-15I.
  • the user device receives (1630) the expand gesture input while the initial recognition result is displayed in the candidate display area, as illustrated in Figure 15H.
  • the user device in response to the expand gesture input, re- renders (1632) the plurality of handwritten strokes to increase a distance between a first subset of strokes assigned to a first recognition unit and a second subset of handwritten strokes assigned to a second recognition unit in the handwriting input area, as illustrated in Figures 15H and 151.
  • the user optionally provides a pinch gesture to inform the user device to treat a plurality of strokes as a single recognition unit, immediately after the user has provided the strokes and realized that the strokes may be too spread out for a correct segmentation based on a standard segmentation process.
  • the user device can distinguish the pinch gesture from a regular stroke based on the two simultaneous contacts present in a pinch gesture.
  • the user optionally provides an expand gesture to inform the user device to treat a plurality of strokes two separate recognition units, immediately after the user has provided the strokes and realized that the strokes may be too close together for a correct segmentation based on a standard
  • the user device can distinguish the expand gesture from a regular stroke based on the two simultaneous contacts present in a pinch gesture.
  • the motion direction of the pinch or expand gesture is optionally used to provide additional guidance on how to segment the strokes under the gesture. For example, if multi-line handwriting input is enabled for the handwriting input area, a pinch gesture with two contacts moving in the vertical direction can inform the handwriting input module to merge two recognition units identified in two adjacent lines into a single recognition unit (e.g., as a top radical and a bottom radical). Similarly, an expand gesture with two contacts moving in the vertical direction can inform the handwriting input module to divide a single recognition unit into two recognition units in two adjacent lines.
  • the pinch and expand gestures can also provide segmentation guidance in a sub-portion of a character input, e.g., merging two subcomponents or divide a single component in a composite character 3 ⁇ 4
  • handwriting input is particular helpful for the user to input the correct character without having to make several attempts to get at the correct proportions and balance.
  • the handwriting input module allows a user to enter a multi-character handwriting input, and allows out-of-order strokes for the multi-character handwriting input within a character, across multiple characters, and even across multiple phrases, sentences, and/or lines in the handwriting input area.
  • the handwriting input module also provides character-by-character deletion in the handwriting input area, where the order of character deletion is in the reverse writing direction, and independent of when the strokes for each character has been provided in the handwriting input area.
  • the deletion of each recognition unit (e.g., character or radical) in the handwriting input area is optionally performed stroke-by- stroke, where the strokes are deleted in a reverse temporal order by which they were provided within the recognition unit.
  • Figures 17A-17H illustrate exemplary user interfaces for responding to a deletion input from a user and provide character-by-character deletion in a multi-character handwriting input.
  • the user has provided a plurality of handwritten strokes 1702 in the handwriting input area 804 of the handwriting input user interface 802. Based on the currently accumulated strokes 1702, the user device presents three recognition results (e.g., results 1704, 1706, and 1708) in the candidate display area 806. As shown in Figure 17B, the user has provided an additional plurality of strokes 1710 in the handwriting input area 806. The user device recognizes three new output characters, and replaces the three previous recognition results 1704, 1706, and 1708 with the three new recognition results 1712, 1714, and 1716.
  • three recognition results e.g., results 1704, 1706, and 1708
  • the cluster of strokes 1710 does not correspond well to any known characters in the repertoire of the handwriting recognition module.
  • the candidate characters (e.g., " " ⁇ ") identified for the recognition unit comprising strokes 1710 are all have a recognition confidence below a predetermined threshold.
  • the user device presents a partial recognition result (e.g., result 1712) which includes only a candidate character (e.g., " ⁇ ") for the first recognition unit, but not any candidate character for the second recognition unit in the candidate display area 806.
  • the user device further displays a full recognition result (e.g., result 1714 or 1716) which includes a candidate character for both recognition units, regardless of whether the recognition confidence has passed the predetermined threshold.
  • a full recognition result e.g., result 1714 or 1716
  • Providing a partial recognition result informs the user which part of the handwritten input needs revision.
  • the user can also choose to enter the correctly recognized portion of handwriting input first, and then rewrite the portion that was not correctly recognized.
  • Figure 17C shows that the user has continued to provide an additional handwritten stroke 1718 to the left of strokes 1710. Based on the relative location and distance of the stroke 1718, the user device determines that the newly added stroke belong to the same recognition unit as the cluster of handwritten strokes 1702. Based on the revised recognition units recognizes a new character (e.g., "3 ⁇ 4") for the first recognition unit, and generates a set of new recognition results 1720, 1722, and 1724. Again, the first recognition result 1720 is a partial recognition result, because none of the candidate characters identified for the strokes 1710 meet the predetermined confidence threshold.
  • Figure 17D shows that the user has now entered a plurality of new strokes 1726 to between the strokes 1702 and the strokes 1710.
  • the user device assigns the newly entered strokes 1726 to the same recognition unit as the strokes 1710. Now, the user has completed entering all of the handwritten strokes for the two Chinese characters (e.g., "), and the correct recognition result 1728 is shown in the candidate display area 806.
  • Figure 27E shows that the user has entered an initial portion of a deletion input, e.g., by making a light contact 1730 on the delete button 1732. If the user maintains the contact with the delete button 1732, the user can delete the current handwriting input character-by-character (or recognition unit by recognition unit). The deletion is not performed for all of the handwritten input at the same time.
  • the last recognition unit e.g., the recognition unit for the character "M ”
  • the default writing direction e.g., from left to right
  • the other recognition unit(s) concurrently displayed within the handwriting input area 804, as shown in Figure 17E.
  • the user device when the user device detects that the user has maintained the contact 1730 on the delete button 1732 for more than a threshold duration, the user device removes the highlighted recognition unit (e.g., in box 1734) from the handwriting input area 806, as shown in Figure 17F. In addition, the user device also revises the recognition results shown in the candidate display area 608 to delete any output characters generated based on the deleted recognition unit, as shown in Figure 17F.
  • the highlighted recognition unit e.g., in box 1734
  • the user device also revises the recognition results shown in the candidate display area 608 to delete any output characters generated based on the deleted recognition unit, as shown in Figure 17F.
  • Figure 17F further illustrates that, if the user continues to maintain the contact 1730 on the delete button 1732 after the last recognition unit (e.g., the recognition unit for the character "fc ") in the handwriting input area 806 has been deleted, the adjacent recognition unit (e.g., the recognition unit for the character "3 ⁇ 4") to the deleted recognition unit becomes the next recognition unit to be deleted. As shown in Figure 17F, the remaining recognition unit has become visually highlighted (e.g., in a box 1736), and ready to be deleted. In some embodiments, the visual highlighting of the recognition unit provides a preview of the recognition unit that would be deleted if the user continues to maintain the contact with the delete button.
  • the last recognition unit e.g., the recognition unit for the character "fc ”
  • the adjacent recognition unit e.g., the recognition unit for the character "3 ⁇ 4"
  • the remaining recognition unit has become visually highlighted (e.g., in a box 1736), and ready to be deleted.
  • the visual highlighting of the recognition unit provides a
  • the contact intensity e.g., the pressure by which the user has applied the contact 1730 with the touch-sensitive screen
  • the threshold duration is optionally used to adjust the threshold duration to confirm the user' s intent to delete the currently highlighted
  • FIGs 17F and 17G illustrate that the user has broken the contact 1730 on the delete button 1732 before the threshold duration is reached, and the recognition unit for the character is preserved in the handwriting input area 806.
  • the first recognition result e.g., result 1738
  • the text in the first recognition result 1738 is entered into the text input area 808, as shown in Figures 17G-17H.
  • Figures 18A-18B are flow charts of an exemplary process 1800 in which the user device provides character-by-character deletion in a multi-character handwriting input.
  • the deletion of the handwriting input is performed before the characters recognized from the handwriting input have been confirmed and entered into the text input area of the user interface.
  • the deletion of the characters in the handwriting input proceeds according to the reverse spatial order of the recognition units identified from the handwriting input, and is independent of the temporal sequence by which the recognition units are formed.
  • Figures 17A-17H illustrate the exemplary process 1800 in accordance with some embodiments.
  • the user device receives (1802) a handwriting input from a user, the handwriting input including a plurality of handwritten strokes provided in an handwriting input area (e.g., area 804 in Figure 17D) of a handwriting input interface.
  • the user device identifies (1804) a plurality of recognition units from the plurality of handwritten strokes, each recognition unit including a respective subset of the plurality of handwriting strokes.
  • the first recognition unit includes strokes 1702 and 1718
  • the second recognition unit includes strokes 1710 and 1726.
  • the user device generates (1806) a multi-character recognition result (e.g., result 1728 in Figure 17D) comprising respective characters recognized from the plurality of recognition units.
  • the user device displays the multi- character recognition result (e.g., result 1728 in Figure 17D) in a candidate display area of the handwriting input interface.
  • the user device receives (1810) a deletion input (e.g., contact 1730 on delete button 1732) from the user, e.g., as shown in Figure 17E.
  • the user device in response to receiving the deletion input, removes (1812) an end character (e.g., the character "life " that appear at the end of the spatial sequence from the multi-character recognition result (e.g., result 1728) displayed in the candidate display area (e.g., candidate display area 806), e.g., as illustrated in Figures 17E-17F.
  • an end character e.g., the character "life" that appear at the end of the spatial sequence from the multi-character recognition result (e.g., result 1728) displayed in the candidate display area (e.g., candidate display area 806), e.g., as illustrated in Figures 17E-17F.
  • the user device renders (1814) the plurality of handwritten strokes in the handwriting input area of the handwriting input interface as the plurality of handwritten strokes are provided by the user in real-time, e.g., as illustrated in Figures 17A-17D.
  • the user device in response to receiving the deletion input, removes (1816) from the handwriting input area (e.g., handwriting input area 804 in Figure 17E) the respective subset of the plurality of handwritten strokes
  • an end recognition unit e.g., the recognition unit containing strokes 1726 and 1710
  • the end recognition unit corresponds to the end character (e.g., the character "3 ⁇ 4 « ' ") in the multi-character recognition result (e.g., result 1728 in Figure 17E).
  • the end recognition unit does not include (1818) a temporally final handwritten stroke among the plurality of handwritten strokes provided by the user. For example, if the user had provided the stroke 1718 after he or she has provided the strokes 1726 and 1710, the end recognition unit including the strokes 1726 and 1710 would still be deleted first.
  • the user device in response to receiving an initial portion of the deletion input, visually distinguishes (1820) the end recognition unit from other recognition units identified in the handwriting input area, e.g., as illustrated in Figure 17E.
  • the initial portion of the deletion input is (1822) an initial contact detected on a delete button in the handwriting input interface, and the deletion input is detected when the initial contact is sustained for more than a predetermined threshold amount of time.
  • the end recognition unit corresponds to a handwritten Chinese character.
  • the handwriting input is written in a cursive writing style.
  • the handwriting input corresponds to multiple Chinese characters written in a cursive writing style.
  • at least one of the handwritten strokes is divided into two adjacent recognition units of the plurality of recognition units. For example, sometimes, a user may use a long stroke that runs on into multiple characters, and in such cases, the segmentation module of the handwriting input module optionally divides the long stroke into several recognition units.
  • the deletion of handwriting input is performed character-by-character (or recognition unit by recognition unit), only a segment of the long stroke is deleted at a time (e.g., the segment within a corresponding recognition unit).
  • the deletion input is (1824) a sustained contact on a deletion button provided in the handwriting input interface
  • removing the respective subset of the plurality of handwritten strokes further includes removing the subset of handwritten strokes in the end recognition unit from the handwriting input area, stroke- by-stroke in a reverse temporal order by which the subset of handwritten strokes has been provided by the user.
  • the user device generates (1826) a partial recognition result comprising a subset of the respective characters recognized from the plurality of recognition units, where each of the subset of the respective characters meets a
  • the user device displays (1828) the partial recognition result (e.g., result 1712 in Figure 17B and result 1720 in Figure 17C) concurrently with the multi-character recognition result (e.g., results 1714 and 1722) in the candidate display area of the handwriting input interface.
  • the partial recognition result e.g., result 1712 in Figure 17B and result 1720 in Figure 17C
  • the multi-character recognition result e.g., results 1714 and 1722
  • the partial recognition result does not include at least the end character in multi-character recognition result. In some embodiments, the partial recognition result does not include at least an initial character in multi-character recognition result. In some embodiments, the partial recognition result does not include at least an intermediate character in multi-character recognition result.
  • the smallest unit of deletion is a radical
  • handwriting inputs are deleted one radical at a time whenever a radical happens to be the last recognition unit in the handwriting inputs still remaining in the handwriting input area.
  • the user device provides both a horizontal writing mode and a vertical writing mode.
  • the user device allows the user to enter text in one or both of a left-to-right writing direction and a right- to-left direction in the horizontal writing mode.
  • the user device allows the user to enter text in one or both of a top-down writing direction and a bottom-up direction in the vertical writing mode.
  • the user device provides various affordances (e.g., a writing mode or writing direction button) on the user interface to invoke a respective writing mode and/or writing direction for the current handwriting inputs.
  • the text input direction in the text input area is the same as the handwriting input direction in the handwriting input direction by default.
  • the user device allows the user to manually set the input direction in the text input area and the writing direction in the handwriting input area.
  • the text display direction in the candidate display area is the same as the handwriting input direction in the handwriting input area by default.
  • the user device allows the user to manually set the text display direction in the text input area independent of the handwriting input direction in the handwriting input area.
  • the user device associates the writing mode and/or writing direction of a handwriting input interface with a corresponding device orientation, and changes in the device orientation automatically triggers a change in writing mode and/or writing direction.
  • a change in writing direction automatically causes entry of a top-ranked recognition result to be entered into the text input area.
  • Figures 19A-19F illustrate exemplary user interfaces of a user device that provides both a horizontal input mode and a vertical input mode.
  • Figure 19A shows the user device in a horizontal input mode.
  • the horizontal input mode is provided when the user device is in a landscape orientation, as shown in Figure 19A.
  • the horizontal input mode is optionally associated with and provided when the device is operated in the portrait orientation. In different applications, the association between the device orientation and the writing mode may be different.
  • the user can provide handwritten characters in a horizontal writing direction (e.g., with a default writing direction going from left to right, or a default writing direction going from right to left).
  • the user device performs segmentation of the handwriting input into one or more recognition units along the horizontal writing direction.
  • the user device only permits single-line input in the handwriting input area.
  • the user device allows multi-line input (e.g., two lines of input) in the handwriting input area.
  • the user has provided a plurality of handwritten strokes in several rows in the handwriting input area 806. Based on the sequence that the user has provided plurality of handwritten strokes and the relative locations of and distances between the plurality of handwritten strokes, the user device determines that the user has entered two lines of characters. After segmenting the handwriting input into two separate lines, the device determines the recognition unit(s) within each line.
  • the user device has recognized a respective character for each recognition unit identified in the current handwriting input 1902, and generated a number of recognition results 1904 and 1906.
  • the output character e.g. the letter "I”
  • the user device optionally generates a partial recognition result (e.g., result 1906) that only shows the output characters that have a sufficient recognition confidence.
  • the user may realize from the partial recognition result 1906, that the first stroke can be revised or individually deleted or rewritten for the recognition model to produce the correct recognition result.
  • editing of the first recognition unit is not necessary because the first recognition unit 1904 does show the desired recognition result for the first recognition unit.
  • the user has rotated the device to a portrait orientation (e.g., shown in Figure 19B).
  • the handwriting input interface is changed from the horizontal input mode to a vertical input mode, as shown in Figure 19B.
  • the layout of the handwriting input area 804, the candidate display area 806, and the text input area 808 may be different from that shown in the horizontal input mode.
  • the particular layout of the horizontal and the vertical input modes can vary to suit different device shapes and application needs.
  • the user device automatically enters the top-ranked result (e.g., result 1904) as a text input 1910 into the text input area 808.
  • the orientation and position of the cursor 1912 also reflect the change in input mode and writing direction.
  • the change in input mode is optionally triggered by the user touching the special input mode selection affordance 1908.
  • the input mode selection affordance is a graphical user interface element that also shows the current writing mode, the current writing direction, and/or the current paragraph direction.
  • the input mode selection affordance can cycle through all available input modes and writing directions provided by the handwriting input interface 802. As shown in Figure 19A, the affordance 1908 shows that the current input mode is a horizontal input mode, with the writing direction from left to right, and a paragraph direction from top to bottom. In Figure 19B, the affordance 1908 shows that the current input mode is a vertical input mode, with a writing direction from top to bottom, and a paragraph direction from right to left. Other combinations of writing direction and paragraph direction are possible, in accordance with various embodiments.
  • the user has entered plurality of new strokes 1914 (e.g., handwritten strokes for two Chinese characters "#$3 ⁇ 4") in the handwriting input area 804 in the vertical input mode.
  • the handwriting input is written in the vertical writing direction.
  • the user device segments the handwriting input in the vertical direction into two recognition units, and displays two recognition results 1916 and 1918 each including two recognized characters laid out in the vertical direction.
  • Figures 19C-19D illustrate that, when the user selects a displayed recognition result (e.g., result 1916), the selected recognition result is entered into the text input area 808 in the vertical direction.
  • a displayed recognition result e.g., result 1916
  • Figures 19E-19F illustrate that, the user has entered additional lines of handwriting input 1920 in the vertical writing direction.
  • the lines run from left to right in accordance with the paragraph direction of traditional Chinese writing.
  • the candidate display area 806 also shows the recognition results (e.g., results 1922 and 1924) in the same writing direction and paragraph direction as that for the handwriting input area.
  • other writing direction and paragraph direction can be provided by default in accordance with a primary language associated with the user device, or the language (e.g., Arabic, Chinese, Japanese, English, etc.) of a soft keyboard installed on the user device.
  • Figures 19E-19F also show that, when the user has selected a recognition result (e.g., result 1922), the text of the selected recognition result is entered into the text input area 808.
  • the text input currently in the text input area 808 thus includes both text written in a horizontal mode, with a writing direction from left to right, and text written in the vertical mode, with a top-down writing direction.
  • the paragraph direction for the horizontal text is top-down, while the paragraph direction for the vertical text is from right to left.
  • the user device allows the user to separately establish preferred writing directions, paragraph directions for each of the handwriting input area 804, the candidate display area 806, and the text input area 808. In some embodiments, the user device allows the user to establish the preferred writing direction and paragraph direction for each of the handwriting input area 804, the candidate display area 806, and the text input area 808, to be associated with each device orientation.
  • Figures 20A-20C are flow charts of an exemplary process 2000 for changing the text input direction and the handwriting input direction of the user interface.
  • Figures 19A- 19F illustrate the process 2000 in accordance with some embodiments.
  • the user device determines (2002) an orientation of the device.
  • the orientation of the device and the changes in device orientation can be detected by the accelerometer and/or other orientation sensing element in the user device.
  • the user device provides (2004) a handwriting input interface on the device in a horizontal input mode in accordance with the device being in a first orientation.
  • a respective line of handwriting input entered in the horizontal input mode is segmented into one or more respective recognition units along a horizontal writing direction.
  • the device provides (2006) the handwriting input interface on the device in a vertical input mode in accordance with the device in a second orientation.
  • a respective line of handwriting input entered in the vertical input mode is segmented into one or more respective recognition units along a vertical writing direction.
  • the device while operating in the horizontal input mode (2008): the device detects (2010) a change in device orientation from the first orientation to the second orientation. In some embodiments, in response to the change in device orientation, the device switches (2012) from the horizontal input mode to the vertical input mode. This is illustrated, for example, in Figures 19A-19B.
  • the user device while operating in the vertical input mode (2014): the user device detects (2016) a change in device orientation from the second orientation to the first orientation. In some embodiments, in response to the change in device orientation, the user device switches (2016) from the vertical input mode to the horizontal input mode. In some embodiments, the association between device orientation and the input mode may be opposite of that described above.
  • the user device while operating in the horizontal input mode (2020): receives (2022) a first multi-word handwriting input from the user. In response to the first multi-word handwriting input, the user device presents (2024) a first multi-word recognition result in a candidate display area of the handwriting input interface in accordance with the horizontal writing direction. This is illustrated, for example, in Figure 19A.
  • the user device while operating in the vertical input mode (2026): receives (2028) a second multi-word handwriting input from the user. In response to the second multi-word handwriting input, the user device presents (2030) a second multi-word recognition result in the candidate display area in accordance with the vertical writing direction. This is illustrated, for example, in Figures 19C and 19E.
  • the user device receives (2032) a first user input selecting the first multi-word recognition result, e.g., as shown in Figures 19A-19B where the selection is made implicitly with an input (e.g., rotation of device or selection of affordance 1908) for changing the input direction.
  • the user device receives (2034) a second user input selecting the second multi-word recognition result, e.g., as shown in Figure 19C or Figure 19E.
  • the user device concurrently displays (2036) respective text of the first multi-word recognition result and the second multi-word recognition result in a text input area of the handwriting input interface, where the respective text of the first multi-word recognition result is displayed in accordance with the horizontal writing direction, and the respective text of the second multi-word recognition result is displayed in accordance with the vertical writing direction.
  • This is illustrated in text input area 808 in Figure 19F, for example.
  • the handwriting input area accepts multiple lines of handwriting input in the horizontal writing direction and has a default top-down paragraph direction.
  • the horizontal writing direction is from left to right.
  • the horizontal writing direction is from right to left.
  • the handwriting input area accepts multiple lines of handwriting input in the vertical writing direction and has a default left-to-right paragraph direction.
  • the handwriting input area accepts multiple lines of handwriting input in the vertical writing direction and has a default right-to-left paragraph direction.
  • the vertical writing direction is from top to bottom.
  • the first orientation is a landscape orientation by default
  • the second orientation is a portrait orientation by default.
  • the user device provides a respective affordance in the handwriting input interface for manually switching between the horizontal input mode and the vertical input mode, irrespective of the device orientation. In some embodiments, the user device provides a respective affordance in the handwriting input interface for manually switching between two alternative writing directions. In some embodiments, the user device provides a respective affordance in the handwriting input interface for manually switching between two alternative paragraph directions. In some embodiments, the affordance is a toggle button that rotates through each available combination of the input and paragraph directions, when invoked one or more consecutive times.
  • the user device receives (2038) a handwriting input from a user.
  • the handwriting input includes a plurality of handwritten strokes provided in the handwriting input area of the handwriting input interface.
  • the user device displays (2040) one or more recognition results in a candidate display area of the handwriting input interface. While the one or more recognition results are displayed in the candidate display area, the user device detects (2042) a user input for switching from a current handwriting input mode to an alternative handwriting input mode.
  • the user device switches (2046) from the current handwriting input mode to the alternative handwriting input mode.
  • the user device clears (2048) the handwriting input from the handwriting input area.
  • the user device automatically enters (2050) a top-ranked recognition result of the one or more recognition results displayed in the candidate display area into a text input area of the handwriting input interface.
  • the current handwriting input mode is the horizontal input mode
  • the alternative handwriting input mode is the vertical input mode
  • the current handwriting input mode is the vertical input mode
  • the alternative handwriting input mode is the horizontal input mode.
  • the current handwriting input mode and the alternative handwriting input mode are modes under which any two different handwriting input directions or paragraph directions are provided.
  • the user input is (2052) rotation of the device from a current orientation to a different orientation.
  • the user input is invocation of an affordance to manually switch the current handwriting input mode to the alternative handwriting input mode.
  • the handwriting input module allows the user to enter handwritten strokes and/or characters in any temporal order.
  • deleting an individual handwritten character in a multi-character handwriting input, and rewriting the same or a different handwritten character in the same location as the deleted character is
  • Figures 20A-20H illustrate exemplary user interfaces for visually highlighting and/or deleting a recognition unit identified in a plurality of handwritten strokes currently accumulated in the handwriting input area. Allowing the user to individually select, view, and delete any one of multiple recognition units identified in a plurality inputs is particularly useful when multi-character, and even multi-line handwriting input is permitted by the user device. By allowing the user to delete a particular recognition unit in the beginning or the middle of the handwriting inputs allows to the user to make corrections to a long input, without requiring the user to delete all recognition units positioned after an undesirable recognition unit.
  • the user has provided a plurality of handwritten strokes (e.g., strokes 2102, 2104, and 2106) in the handwriting input area 804 of the handwriting input user interface 802. While the user continues to provide additional strokes to the handwriting input area 804, the user device updates the recognition units identified from the handwriting input currently accumulated in the handwriting input area, and revises the recognition results according to the output characters recognized from the updated recognition units. As shown in Figure 20C, the user device has identified two recognition units from the current handwriting input, and presented three recognition results (e.g., 2108, 2010, and 2112) each including two Chinese characters.
  • three recognition results e.g., 2108, 2010, and 2112
  • the user device when the user provides a tap gesture (e.g., a contact followed with an immediate lift-off at the same location) on the touch-sensitive display, the user device interprets the tap gesture as an input to cause visually highlighting of individual recognition units that are currently identified in the handwriting input area.
  • another predetermined gesture e.g., a multi-finger wiping gesture over the handwriting input area
  • a tap gesture is sometimes preferred because it is relatively easy to distinguish from a handwritten stroke, which usually involves a sustained contact of a longer duration and with movement of the contact within the handwriting input area 804.
  • a multi-touch gesture is sometimes preferred because it is relatively easy to distinguish from a handwritten stroke, which usually involves a single contact within the handwriting input area 804.
  • the user device provides an affordance 2112 in the user interface that can be invoked (e.g., via a contact 2114) by the user to cause the individual recognition units to be visually highlighted (e.g., as shown by boxes 2108 and 2110).
  • the affordance is preferred when there is sufficient screen space to accommodate such an affordance.
  • the affordance can be invoked multiple consecutive times by the user, which causes the user device to visually highlight recognition unit(s) identified according to a different segmentation chain in the segmentation lattice and for the highlighting to be turned off when all segmentation chains have been shown.
  • FIG. 21D when the user has provided the necessary gesture to highlight the individual recognition units in the handwriting input area 804, the user device further displays a respective deleting affordance (e.g., small delete buttons 2116 and 2118) over each highlighted recognition unit.
  • Figures 21E-21F show that when the user touches (e.g., via a contact 2120) the deleting affordance of a respective recognition unit (e.g., the delete button 2116 for the first recognition unit in box 2118), the respective recognition unit (e.g., in box 2118) is removed from the handwriting input area 804.
  • the deleted recognition unit is not the last entered recognition unit temporally, nor is it the spatially last recognition unit along the writing direction.
  • Figure 2 IF shows that, in response to the deletion of the first recognition unit in the handwriting input area, the user device also updates the recognition results displayed in the candidate display area 806. As shown in Figure 21F, the user device has also deleted candidate character corresponding to the deleted recognition unit from the recognition results. As a result, a new recognition result 2120 is shown in the candidate display area 806.
  • FIG. 21G-21H show that after the first recognition unit has been removed from the handwriting input interface 804, the user has provided a plurality of new handwritten strokes 2122 in the area that was previously occupied by the deleted recognition unit.
  • the user device has re-segmented the currently accumulated handwriting input in the handwriting input area 804. Based on the recognition units identified from the handwriting input, the user device regenerated recognition results (e.g., results 2124 and 2126) in the candidate display area 806.
  • Figure 21G-21H show that the user has selected (e.g., via the contact 2128) one of the recognition results (e.g., result 2124), and text of the selected recognition result is entered into the text input area 808.
  • Figures 22A-22B are flow charts for an exemplary process 2200 in which individual recognition units identified in the current handwriting input is visually presented, and can be individually deleted, regardless of the temporal order by which the recognition units are formed.
  • Figures 21A-21H illustrate the process 2200 in accordance with some embodiments.
  • the user device receives (2202) a handwriting input from a user.
  • the handwriting input includes a plurality of handwritten strokes provided on a touch-sensitive surface coupled to the device.
  • the user device renders (2204) the plurality of handwritten strokes in a handwriting input area (e.g., handwriting input area 804) of a handwriting input interface.
  • the user device segments (2206) the plurality of handwritten strokes into two or more recognition units, each recognition unit comprising a respective subset of the plurality of handwritten strokes.
  • the user device receives (2208) an edit request from the user.
  • the edit request is (2210) a contact detected over a
  • predetermined affordance e.g., affordance 2112 in Figure 21D
  • the edit request is (2212) a tap gesture detected over a predetermined area in the handwriting input interface.
  • the predetermined area is within the handwriting input area of the
  • the predetermined area is outside of the handwriting input area of the handwriting input interface.
  • another predetermined gesture e.g., a cross gesture, a horizontal swipe gesture, a vertical swipe gesture, a slanted swipe gesture
  • Gestures outside of the handwriting input area can be easily distinguished from a handwritten stroke, since it is provided outside of the handwriting input area.
  • the user device in response to the edit request, visually distinguishes (2214) the two or more recognition units in the handwriting input area, e.g., using the boxes 2108 and 2110 in Figure 21D. In some embodiments, visually
  • distinguishing the two or more recognition units further includes (2216) highlighting respective boundaries between the two or more recognition units in the handwriting input area.
  • (2216) highlighting respective boundaries between the two or more recognition units in the handwriting input area.
  • different ways of visually distinguishing the recognition units identified in the current handwriting input may be used.
  • the user device provides (2218) a means for individually deleting each of the two or more recognition units from the handwriting input area.
  • the means for individually deleting each of the two or more recognition units is a respective delete button displayed in proximity to the each recognition unit, e.g., as shown by delete buttons 2116 and 2118 in Figure 21D.
  • the means for individually deleting each of the two or more recognition units is a means for detecting a predetermined deletion gesture input over the each recognition unit.
  • the user device does not visibly display the individual deletion affordance over the highlighted recognition units. Instead, in some embodiments, the user is allowed to use a deletion gesture to delete a respective recognition unit underneath the deletion gesture. In some embodiments, as the user device is displaying the recognition units in a visually highlighted manner, the user device does not accept additional handwritten strokes in the handwriting input area. Instead, a predetermined gesture or any gesture detected over a visually highlighted recognition unit will cause the user device to remove the recognition unit from the handwriting input area, and revise the recognition results displayed in the candidate display area accordingly. In some embodiments, a tap gesture causes the user device to visually highlight the individual recognition units identified in the handwriting recognition area, and the user can then use the delete button to delete the individual recognition units one by one in the reverse writing direction.
  • the user device receives (2224), from the user and through the provided means, a deletion input for individually deleting a first recognition unit of the two or more recognition units from the handwriting input area, e.g., as shown in Figure 21E.
  • the user device removes (2226) the respective subset of handwritten strokes in the first recognition unit from the handwriting input area, e.g., as shown in Figure 21F.
  • the first recognition unit is a spatially initial recognition unit in the two or more recognition units.
  • the first recognition unit is a spatially intermediate recognition unit among the two or more recognition units, e.g., as shown in Figures 21E-21F.
  • the first recognition unit is a spatially end recognition unit among the two or more recognition units.
  • the user device generates (2228) a segmentation lattice from the plurality of handwritten strokes, the segmentation lattice includes a plurality of alternative segmentation chains each representing a respective set of recognition units identified from the plurality of handwriting strokes.
  • Figure 21G shows the recognition results 2024 and 2026, where the recognition result 2024 is generated from one segmentation chain with two recognition units, and the recognition result 2026 is generated from another segmentation chain with three recognition units.
  • the user device receives (2230) two or more consecutive edit requests from the user.
  • the two or more consecutive edit request can be several consecutive taps on the affordance 2112 in Figure 21G.
  • the user device in response to each of the two or more consecutive edit requests, visually distinguishes (2232) the respective set of recognition units from a different one of the plurality of alternative segmentation chains in the handwriting input area. For example, in response to a first tap gesture, two recognition units (e.g., for the characters " t'N” and respectively) are highlighted in the handwriting input area 804, and in response to a second tap gesture, three recognition units (e.g., for the characters "W "" . and " ', respectively). In some embodiments, in response to a third tap gesture, the visual highlighting is optionally removed from all recognition units, and the handwriting input area is returned to the normal state ready to accept additional strokes.
  • two recognition units e.g., for the characters " t'N" and respectively
  • three recognition units e.g., for the characters "W "" . and " ', respectively.
  • the visual highlighting is optionally removed from all recognition units, and the handwriting input area is returned to the normal state ready to accept additional
  • the user device provides (2234) a means for individually deleting each of the respective set of recognition units currently represented in the handwriting input area.
  • the means is an individual delete button for each highlighted recognition unit.
  • the means is a means for detecting of a predetermined deletion gesture over each highlighted recognition unit, and for invoking a function to delete the highlighted recognition unit under the
  • the user device provides a continuous input mode in the handwriting input area. Since the area of the handwriting input area is limited on a portable user device, it is sometimes desirable to provide a way to cache the handwriting inputs provided by the user, and allow the user to reuse the screen space without commit the previously provided handwriting inputs.
  • the user device provides a scrolling handwriting input area, where input area gradually shifts by a certain amount (e.g., one recognition unit at a time) when the user is getting sufficiently close to the end of the handwriting input area.
  • the existing recognition units in the handwriting input area may interfere with the user's writing process, and possibly interfere with the correct segmentation of the recognition units, it is sometimes advantageous to recycle a previously used region of the input area without dynamically shifting the recognition units.
  • a top recognition result for the handwriting input area is
  • the user is allowed to write over an existing handwriting input that is still shown in the handwriting input area.
  • temporal information is used to determine whether a new stroke is part of an earlier recognition unit or a new recognition unit.
  • Such temporal-information dependent systems place stringent requirements on the speed and tempo by which the user provides the handwriting input, which is difficult to meet by many users.
  • the visual rendering of the handwriting input can be a jumble that is difficult to for the user to decipher.
  • the writing process can be frustrating, and confusing for the user, leading to a bad user experience.
  • a fading process is used to indicate when the user can reuse an area occupied by a previously written recognition unit, and continue writing in the handwriting input area.
  • the fading process gradually reduces the visibility of each recognition unit that has been provided in the handwriting input area for a threshold amount of time, such that when new strokes are written over it, the existing text does not visually compete with the new strokes.
  • writing over a faded recognition unit automatically causes a top-ranked recognition result for the recognition unit to be entered into the text input area, without requiring the user to stop writing and to explicitly provide a selection input for the top-ranked recognition result.
  • writing over a faded recognition unit does not cause automatic selection of the top-ranked search result.
  • the faded recognition units are cached in a handwriting input stack, and combined with the new handwriting input as the current handwriting input. The user can see a recognition results generated based on all of the recognition units accumulated in the handwriting input stack before making a selection.
  • Figures 23A-23J illustrate exemplary user interfaces and processes in which recognition units provided in different region of the handwriting input area are gradually faded out from their respective regions, e.g., after a predetermined amount of time and after the fade-out has occurred in a particular region, the user is allowed to provide new handwritten strokes in that region.
  • the user has provided a plurality of handwritten strokes 2302 (e.g., three handwritten strokes for the capital letter "I") in the handwriting input area 804.
  • the handwritten strokes 2302 are identified by the user device as a recognition unit.
  • the handwriting input currently shown in the handwriting input area 804 is cached in a first layer in the handwriting input stack of the user device.
  • a number of recognition results generated based on the identified recognition unit are provided in the candidate display area 804.
  • Figure 23B shows that, when the user continues to write one or more strokes 2302 to the right of the strokes 2304, the handwritten strokes 2302 in the first recognition unit starts to fade out gradually in the handwriting input area 804.
  • an animation is displayed to mimic the gradual fading or dissipation of the visual rendering of the first recognition unit.
  • the animation may produce a visual effect of ink evaporating from a white board.
  • the fading of the recognition unit is not uniform across the entire recognition unit. In some embodiments, the fading of the recognition unit increases over time and eventually the recognition unit is completely invisible in the handwriting area.
  • the invisible recognition unit remains at the top of the handwriting input stack, and the recognition results generated from the recognition unit continue to be displayed in the candidate display area.
  • a faded recognition unit is not completely removed from view until new handwriting input has been written over it.
  • the user device allows new handwriting input to be provided over the region occupied by a faded recognition unit immediate upon the start of the fading animation. In some embodiments, the user device allows new handwriting input to be provided over the region occupied by a faded recognition unit only after the fading has progressed to a certain stage (e.g., to the faintest level or until the recognition is completely invisible in the region).
  • Figure 23C shows that the first recognition unit (i.e., strokes 2302) has completed its fading process (e.g., the ink color has stabilized at very faint level or has become invisible).
  • the user device has identified additional recognition units from the additional handwritten strokes provided by the user (e.g., the recognition units for the handwritten letters "a" and "m"), and updated the recognition results presented in the candidate display area 804.
  • Figures 22D-22F illustrate that, as time goes on, and the user has provided a plurality of addition handwritten strokes (e.g., 2304 and 2306) in the handwriting input area 804. At the same time, the previously identified recognition units gradually fade away from the handwriting input area 804. In some embodiments, it takes a predetermined amount of time for each recognition unit to start its own fading process after the recognition unit has been identified. In some embodiments, the fading process for each recognition unit does not start until the user has started inputting a second recognition unit downstream from it.
  • a predetermined amount of time for each recognition unit does not start until the user has started inputting a second recognition unit downstream from it.
  • a single stroke e.g., stroke 2304 or stroke 2306
  • multiple recognition units e.g., recognition unit for each handwritten letter in the word "am” or "back
  • Figure 22G illustrate that, even after a recognition unit has started its fading process, the user can bring it back to the un-faded state by a predetermined revival input, e.g., a tap gesture (e.g., as indicated by a contact 2308 followed by an immediate lift-off) on a delete button 2310.
  • a predetermined revival input e.g., a tap gesture (e.g., as indicated by a contact 2308 followed by an immediate lift-off) on a delete button 2310.
  • a tap gesture e.g., as indicated by a contact 2308 followed by an immediate lift-off
  • the revival of faded recognition units is made character-by-character in the reverse writing direction in the handwriting input area 804.
  • the revival of faded recognition units is made word-by- word in the handwriting input area 804.
  • the recognition units corresponding to the word "back” has been revived from a completed faded state to a completely un-faded state.
  • the clock for starting the fading process is reset each recognition unit when the recognition unit is revived into the un-faded state.
  • Figure 22H shows that, a sustained contact on the delete button causes the last recognition unit (e.g., the recognition unit for the letter “k” in the word “back") in the default writing direction to be deleted from the handwriting input area 804. As the deletion input is continually maintained, more recognition units (e.g., the recognition units for the letters "c", "a", "b” in the word “back") are deleted one by one in the reverse writing direction. In some embodiments, the deletion of the recognition unit is word by word, and all letters of the handwritten word "back” is deleted from the handwriting input area 804 are removed at the same time.
  • Figure 22H also shows that, as the contact 2308 is maintained on the delete button 2310 after the deletion of the recognition unit for the letter "b” in the handwritten word “back”, the previously faded recognition unit "m” is revived as well.
  • Figure 231 shows that, if the delete input ceases before the deletion of the revived recognition unit "m" in the handwritten word "am” occurs, the revived recognition unit gradually fades again.
  • the state e.g., a state selected from a set of one or more faded states and the un-faded state
  • the state of each recognition unit is maintained and updated in the handwriting input stack.
  • Figure 23 J illustrate that, when the user has provided one or more strokes 2312 over the area occupied by a faded recognition unit (e.g., the recognition unit for the letter ' ⁇ ') in the handwriting input area, in some embodiments, text of the top-ranked recognition result (e.g., result 2314) for the handwriting input made before the strokes 2312 are automatically entered into the text input area 808, as shown in Figures 23I-23J. As shown in Figure 23J, the text "I am" is no longer shown as being tentative, but instead, has been committed in the text input area 808. In some embodiments, once a text input has been made for a fully or partially faded handwriting input, the handwriting input is removed from the handwriting input stack. The newly entered strokes (e.g., strokes 2312) become the current input in the handwriting input stack.
  • a faded recognition unit e.g., the recognition unit for the letter ' ⁇ '
  • the text "I am” is no longer shown as being tentative, but instead, has been committed in the text input area 808.
  • the handwriting input is removed from the handwriting input stack.
  • the newly entered strokes (e.g., strokes 2312) become the current input in the handwriting input stack.
  • the strokes 2312 when the strokes 2312 is provided over the area occupied by a faded recognition unit (e.g., the recognition unit for the letter "I") in the handwriting input area, the text of the top-ranked recognition result (e.g., result 2314) for the strokes 2312 is provided over the area occupied by a faded recognition unit (e.g., the recognition unit for the letter "I") in the handwriting input area, the text of the top-ranked recognition result (e.g., result 2314) for the
  • the currently handwriting input (both faded and un-faded) in the handwriting input area 804 is cleared, and cached in the handwriting input stack.
  • the user device determines the recognition results based on the entirety of the handwriting input currently accumulated in the handwriting input stack.
  • the recognition results are displayed in the candidate display area. In other words, even though only a part of the currently accumulated handwriting input is shown in the handwriting input area 804, the recognition results are generated based on the entire handwriting input cached in the handwriting input stack (both the portion that is visible and the portions that are no longer visible).
  • Figure 23K shows that the user has entered more strokes 2316 in the
  • Figure 23L shows that a new stroke 2318 written over the faded strokes 2312 and 2316, has caused text of the top recognition result 2320 for the faded strokes 2312 and 2316 to be entered into the text input area 808.
  • the user optionally provides a handwriting input in multiple lines.
  • the same fading process can be used to clear the handwriting input area for a new handwriting input, when multi-line input is enabled.
  • Figures 24A-24B are flow charts of an exemplary process 2400 for providing a fading process in the handwriting input area of a handwriting input interface.
  • Figures 23 A- 23K illustrate the process 2400 in accordance with some embodiments.
  • the device receives (2402) a first handwriting input from a user.
  • the first handwriting input includes a plurality of handwritten strokes, and the plurality of handwritten strokes form multiple recognition units distributed along a respective writing direction associated with a handwriting input area of a handwriting input interface.
  • the user device renders (2404) each of the plurality of handwritten strokes in the handwriting input area as the handwritten stroke is provided by the user.
  • the user device starts (2406) a respective fading process for each of the multiple recognition units after the recognition unit is completely rendered. In some embodiments, during the respective fading process, the rendering of the
  • the user device receives (2408) a second handwriting input from the user over a region of the handwriting input area occupied by a faded recognition unit of the multiple recognition unit, e.g., as illustrated in Figures 231-23 J, and 23K-23L.
  • the user device in response to receiving the second handwriting input (2410): the user device renders (2412) the second handwriting input in the handwriting input area and clears (2414) all faded recognition units from the handwriting input area.
  • all recognition units that were entered in the handwriting input area before the second handwriting input are cleared from the handwriting input area, regardless of whether the recognition unit has started its fading process. This is illustrated in Figure 23I-23J, and 23K-23L, for example.
  • the user device generates (2416) one or more recognition results for the first handwriting input.
  • the user device displays (2418) the one or more recognition results in a candidate display area of the handwriting input interface.
  • the user device in response to receiving the second handwriting input, enters (2420) a top-ranked recognition result displayed in the candidate display area into a text input area of the handwriting input interface. This is illustrated in Figure 23T23J and 23K-23L, for example.
  • the user device stores (2422) an input stack including the first handwriting input and the second handwriting input.
  • the user device generates (2424) one or more multi-character recognition results each comprising a respective spatial sequence of characters recognized from a concatenation of the first handwriting input and the second handwriting input.
  • the user device displays (2426) the one or more multi-character recognition results in a candidate display area of the handwriting input interface, while the rendering of the second handwriting input has replaced the rendering of the first handwriting input in the handwriting input area.
  • the respective fading process for each recognition unit is started when a predetermined time period has elapsed after the recognition unit is completed by the user.
  • fading process for each recognition unit is started when the user has started inputting the strokes for a next recognition unit after the recognition unit.
  • an end state of the respective fading process for each recognition unit is a state with a predetermined minimum visibility for the recognition unit.
  • an end state of the respective fading process for each recognition unit is a state with zero visibility for the recognition unit.
  • the user device receives (2428) a predetermined revival input from the user.
  • the user device reverts (2430) the last recognition unit from the faded state to an un-faded state.
  • the predetermined revival input is an initial contact detected on a deletion button provided in the handwriting input interface.
  • a sustained contact detected on the deletion button deletes the last recognition unit from the handwriting input area and revives the second to last recognition unit from the faded state to the un-faded state. This is illustrated in Figures 23G-23H, for example.
  • the multi- script handwriting recognition model performs stroke-order independent, and stroke-direction independent recognition of handwritten characters.
  • the recognition model is trained on only spatially-derived features contained in flat images of writing samples corresponding to different characters in the vocabulary of the handwriting recognition model. Since the images of the writing sample do not contain any temporal information related to individual strokes contained in the images, the resulting recognition model is stroke-order independent and stroke-direction independent.
  • stroke-order and stroke-direction independent handwriting recognition provide many advantages over conventional recognition systems that rely on information related to the temporal generation of the characters (e.g., temporal sequences of strokes in the characters).
  • temporal information related to individual strokes is available, and it is sometimes beneficial to utilize this information to improve recognition accuracy of the handwriting recognition system.
  • the follow describe a technique that integrates temporally-derived stroke-distribution information into the spatial feature extraction of a handwriting recognition model, where the use of the temporally-derived stroke-distribution information does not destroy the stroke-order and/or stroke direction independence of the handwriting recognition system. Based on the stroke-distribution information related to different characters, disambiguation between similar-looking characters that are produced with distinctively different set of strokes becomes possible.
  • a handwriting input is converted to an input image (e.g., an input bitmap image) for the handwriting recognition model (e.g., a CNN)
  • the temporal information associated with individual strokes is lost.
  • an input image e.g., an input bitmap image
  • eight strokes e.g., labeled #1 - #8 in Figure 27
  • the sequence and direction of the strokes for the character provides some unique features associated with the character.
  • a naive way to capture the stroke-order and stroke-direction information, without destroying the stroke-order and stroke-direction independence of the recognition system is to explicitly enumerate all possible permutations in stroke order and stroke direction in the training samples. But even for a character of only moderate complexity, this would amount to over one billion possibilities, which makes it unfeasible if not impossible to implement in practice.
  • a stroke- distribution profile is generated for each writing sample, which abstract out the
  • the stroke- distribution profiles of writing samples are trained to extract a set of temporally-derived features which are subsequently combined with the spatially-derived features (e.g., from input bitmap images), to improve recognition accuracy without impacting the stroke-order and stroke direction independence of the handwriting recognition system.
  • the temporal information associated with a character is extracted by computing a variety of pixel distributions to characterize each handwritten stroke. Every handwritten stroke of a character gives rise to a deterministic pattern (or profile) when projected onto a given direction. While this pattern in and of itself may be insufficient to recognize the stroke unambiguously, when combined with other similar patterns, it may be adequate to capture certain characteristics inherent to this particular stroke. Integrating this kind of stroke representation alongside spatial feature extraction (e.g., feature extraction based on input images in a CNN) in turn provides orthogonal information that can be useful to disambiguate between similar-looking characters in the repertoire of the handwriting recognition model.
  • Figures 25A-25B are flow charts of an exemplary process 2500 for integrating temporally-derived features and spatially-derived features of handwriting samples during training of a handwriting recognition model, where the resulting recognition model remains stroke-order and stroke direction independent.
  • the exemplary process 2500 is performed on a server device that provides the trained recognition model to a user device (e.g., a portable device 100).
  • the server device includes one or more processors and memory containing instructions for performing the process 2500 when executed by the one or more processors.
  • the device separately trains (2502) a set of spatially-derived features and a set of temporally-derived features of a handwriting recognition model, where the set of spatially-derived features are trained on a corpus of training images each being an image of a handwriting sample for a respective character of an output character set, and the set of temporally-derived features are trained on a corpus of stroke-distribution profiles, each stroke-distribution profile numerically characterizing a spatial distribution of a plurality of strokes in a handwriting sample for a respective character of the output character set.
  • separately training the set of spatially-derived features further includes (2504) training a convolutional neural network having an input layer, an output layer, and a plurality of convolutional layers including a first convolutional layer, a last convolutional layer, zero or more intermediate convolutional layers between the first convolutional layer and the last convolutional layer, and a hidden layer between the last convolutional layer and the output layer.
  • An exemplary convolutional network 2602 is shown in Figure 26.
  • the exemplary convolutional network 2602 can be implemented in substantially the same way as the convolutional network 602 shown in Figure 6.
  • the convolutional network 2602 includes an input layer 2606, an output layer 2608, a plurality of convolutional layers including a first convolutional layer 2610a, zero or more intermediate convolutional layers, and a last convolutional layer 261 On, and a hidden layer 2614 between the last convolutional layer and the output layer 2608.
  • the convolutional network 2602 also includes kernel layers 2616 and sub-sampling layers 2612 in accordance with the arrangement shown in Figure 6.
  • the training of the convolutional network is based on images 2614 of writing samples in the training corpus 2604. Spatially-derived features are obtained and respective weights associated with the different features are determined by minimizing the recognition errors for the training samples in the training corpus. The same features and weights, once trained, are used for recognition of new handwriting samples not present in the training corpus.
  • separately training the set of temporally-derived features further includes (2506) providing the plurality of stroke-distribution profiles to a statistical model to determine a plurality of temporally-derived parameters and respective weights for the plurality of temporally-derived parameters for classifying the respective characters of the output character set.
  • a stroke-distribution profile 2620 is derived from each writing sample in a training corpus 2622.
  • the training corpus 2622 optionally includes the same writing samples as the corpus 2604, but also includes temporal information associated with stroke generation in each writing sample.
  • the stroke-distribution profiles 2622 are provided to a statistical modeling process 2624, during which temporally-derived features are extracted and respective weights for the different features are determined by minimizing a recognition or classification error based on a statistical modeling method (e.g., a CNN, K-Nearest Neighbor, etc.).
  • a statistical modeling method e.g., a CNN, K-Nearest Neighbor, etc.
  • the set of temporally derived features and respective weights are converted to a set of feature vectors (e.g., feature vectors 2626 or feature vectors 2628) and injected into a respective layer in the convolutional neural network 2602.
  • the resulting network thus includes spatially-derived parameters and temporally-derived parameters that are orthogonal to each other, and together contribute to the recognition of characters.
  • the device combines (2508) the set of spatially-derived features and the set of temporally-derived features in the handwriting recognition model. In some embodiments, combining the set of spatially-derived features and the set of temporally-derived features in the handwriting recognition model includes (2510) injecting the plurality of spatially-derived parameters and the plurality of temporally-derived parameters into one of the convolutional layers or the hidden layer of the convolutional neural network. In some embodiments, the plurality of temporally-derived parameters and respective weights for the plurality temporally-derived parameters are injected into the last convolutional layer (e.g., the last convolutional layer 2610n in Figure 26) of the convolutional neural network for handwriting recognition. In some embodiments, the plurality of temporally-derived parameters and respective weights for the plurality temporally-derived parameters are injected into the hidden layer (e.g., the hidden layer 2614 in Figure 26) of the convolutional neural network for handwriting recognition.
  • the last convolutional layer e.g
  • the device provides (2512) real-time handwriting recognition for a user's handwriting input using the handwriting recognition model.
  • the device generates (2514) the corpus of stroke- distribution profiles from a plurality writing samples.
  • each of the plurality of handwriting samples corresponds (2516) to a character in the output character set and separately preserves respective spatial information for each constituent stroke of the handwriting sample as it was written.
  • the device performs (2518) the following steps:
  • the device For each of the plurality of handwriting samples (2520): the device identifies (2522) constituent strokes in the handwriting sample; for each of the identified strokes of the handwriting sample, the device calculates (2524) a respective occupancy ratio along each of a plurality of predetermined directions, occupancy ratio being a ratio between a projected span of said each stroke direction and a maximum projected span of said writing sample; for each of the identified strokes of the handwriting sample, the device also calculates (2526) a respective saturation ratio for said each stroke based on a ratio between a respective number of pixels within said each stroke and an overall number of pixels within said writing sample.
  • the user device then generates (2528) a feature vector for the handwriting sample as the stroke-distribution profile of the writing sample, the feature vector including the respective occupancy ratios and the respective saturation ratio of at least N strokes in the handwriting sample, wherein N is a predetermined natural number. In some embodiments, N is less than a maximum stroke count observed in any single writing sample within the plurality of writing samples.
  • the device sorts the respective occupancy ratios of the identified strokes in each of the predetermined directions in a descending order; and includes only N top-ranked occupancy ratios and saturation ratios of writing sample in the feature vector of the writing sample.
  • the plurality of predetermined directions include a horizontal direction, a vertical direction, a positive 45 degree direction, and a negative 45 degree direction of the writing sample.
  • the device to provide real-time handwriting recognition for a user's handwriting input using the handwriting recognition model, the device receives the user' s handwriting input; and in response to receiving the user' s handwriting input, provides a handwriting recognition output to the user substantially contemporaneously with the receipt of the handwriting input.
  • each input image of a handwritten character is optionally normalized into a square.
  • the span of each individual handwritten stroke (e.g., stroke #1, #2, ... , and #8) is measured when projected onto the horizontal, vertical, the +45 degree diagonal, and the -45 degree diagonal of the square.
  • the spans of each stroke Si are recorded as xspan(i), yspan(i), cspan(i), and dspan(i), respectively, for the four projection directions.
  • the maximum spans observed across the entire image is also recorded.
  • the maximum spans of character are recorded as xspan, yspan, cspan, and dspan, respectively, for the four projection directions.
  • four directions of projection are optionally considered here, although in principle any arbitrary set of projections may be used in various embodiments.
  • the maximum spans e.g., denoted as xspan, yspan, cspan, and dspan
  • the spans e.g., denoted as xspan(4), yspan(4), cspan(4), and dspan(4)
  • one of the strokes e.g., stroke #4
  • the respective occupancy ratio along each projection direction is computed.
  • the occupancy ratios of all strokes in each direction is sorted separately in decreasing order, and a respective ranking of all strokes in the input image is thus obtained for each projection direction in terms of their occupancy ratios in that direction.
  • the ranking of strokes in each projection direction reflects the relative importance of each stroke along the associated projection direction. This relative importance is irrespective of the order and the direction by which the stroke has been produced in the writing sample.
  • this ranking based on occupancy ratios is temporally- derived information that is stroke-order and stroke-direction independent.
  • a relative weight is given to each stroke, indicating the importance of the stroke relative to the entire character.
  • the weight is measured by the ratio of the number of pixels in each stroke to the overall number of pixels in the character. This ratio is referred to as a saturation ratio associated with each stroke.
  • a feature vector can be created for each stroke. For each character, a set of feature vectors including 5S number of features is created. This set of features is referred to as a stroke-distribution profile of the character.
  • only a predetermined number of top-ranked strokes are used in constructing the stroke-distribution profile of each character.
  • the predetermined number of strokes is 10.
  • 50 stroke-derived features can be generated for each character. In some embodiments, these features are injected either at the last convolutional layer of a convolutional neural network, or at the subsequent hidden layer.
  • an input image of a recognition unit is provided to the handwriting recognition mode that has been trained with both the spatially-derived features and the temporally-derived features. The input image is processed through each layer of the handwriting recognition model shown in Figure 26. When the processing of the input image reaches the layer (e.g., the last convolutional layer or the hidden layer) in which the stroke-distribution profile input is needed, the layer (e.g., the last convolutional layer or the hidden layer) in which the stroke-distribution profile input is needed, the layer (e.g., the last convolutional layer or the hidden layer) in which the stroke-distribution profile input is needed,
  • the stroke-distribution profile of the recognition unit is injected into that layer.
  • the processing of the input image and the stroke-distribution profile continues until an output classification (e.g., one or more candidate characters) is provided in output layer 2608.
  • the stroke-distribution profiles of all recognition units are computed, and provided to the handwriting recognition model as input, together with the input images of the recognition units.
  • the input image of a recognition unit goes through the handwriting recognition model (without the benefit of the temporally-trained features) initially.
  • the stroke-distribution profiles of the recognition unit is then injected into the handwriting recognition model at the layer that has been trained with the temporally-derived features (e.g., the last convolutional layer, or the hidden layer).
  • the temporally-derived features e.g., the last convolutional layer, or the hidden layer.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Character Discrimination (AREA)
  • User Interface Of Digital Computer (AREA)
  • Document Processing Apparatus (AREA)
  • Character Input (AREA)
  • Image Analysis (AREA)

Abstract

L'invention concerne des procédés, des systèmes et des supports lisibles par ordinateur liés à une technique destinée à mettre en place une fonctionnalité de saisie manuscrite sur un dispositif d'utilisateur. Un module de reconnaissance d'écriture manuscrite fait l'objet d'un apprentissage pour disposer d'un répertoire comportant des écritures multiples ne se chevauchant pas et être capable de reconnaître des dizaines de milliers de caractères en utilisant un seul modèle de reconnaissance d'écriture manuscrite. Le module de saisie manuscrite assure une reconnaissance d'écriture manuscrite en temps réel, indépendante de l'ordre des traits et de la direction des traits, pour une saisie manuscrite multi-caractères. En particulier, une reconnaissance d'écriture manuscrite en temps réel, indépendante de l'ordre des traits et de la direction des traits est assurée pour la reconnaissance multi-caractère, ou au niveau des phrases, d'une écriture manuscrite en chinois. Des interfaces d'utilisateurs destinées à mettre en place la fonctionnalité de saisie manuscrite sont également décrites.
PCT/US2014/040417 2013-06-09 2014-05-30 Gestion de reconnaissance d'écriture manuscrite en temps réel WO2014200736A1 (fr)

Priority Applications (11)

Application Number Priority Date Filing Date Title
KR1020197021958A KR102121487B1 (ko) 2013-06-09 2014-05-30 실시간 필기 인식 관리
KR1020217005264A KR102347064B1 (ko) 2013-06-09 2014-05-30 실시간 필기 인식 관리
KR1020217043310A KR20220003662A (ko) 2013-06-09 2014-05-30 실시간 필기 인식 관리
CN201480030897.0A CN105247540B (zh) 2013-06-09 2014-05-30 管理实时手写识别
CN201811217822.XA CN109614847B (zh) 2013-06-09 2014-05-30 管理实时手写识别
JP2016518366A JP6154550B2 (ja) 2013-06-09 2014-05-30 リアルタイム手書き認識の管理
KR1020157033627A KR101892723B1 (ko) 2013-06-09 2014-05-30 실시간 필기 인식 관리
CN201811217768.9A CN109614845B (zh) 2013-06-09 2014-05-30 管理实时手写识别
KR1020207016098A KR102221079B1 (ko) 2013-06-09 2014-05-30 실시간 필기 인식 관리
KR1020187024261A KR102005878B1 (ko) 2013-06-09 2014-05-30 실시간 필기 인식 관리
HK16108185.0A HK1220276A1 (zh) 2013-06-09 2016-07-12 管理實時手寫識別

Applications Claiming Priority (18)

Application Number Priority Date Filing Date Title
US201361832942P 2013-06-09 2013-06-09
US201361832908P 2013-06-09 2013-06-09
US201361832921P 2013-06-09 2013-06-09
US201361832934P 2013-06-09 2013-06-09
US61/832,934 2013-06-09
US61/832,921 2013-06-09
US61/832,908 2013-06-09
US61/832,942 2013-06-09
US14/290,945 2014-05-29
US14/290,945 US9465985B2 (en) 2013-06-09 2014-05-29 Managing real-time handwriting recognition
US14/290,935 US9898187B2 (en) 2013-06-09 2014-05-29 Managing real-time handwriting recognition
US14/290,935 2014-05-29
US14/291,722 2014-05-30
US14/292,138 2014-05-30
US14/291,865 2014-05-30
US14/291,865 US9495620B2 (en) 2013-06-09 2014-05-30 Multi-script handwriting recognition using a universal recognizer
US14/291,722 US20140363082A1 (en) 2013-06-09 2014-05-30 Integrating stroke-distribution information into spatial feature extraction for automatic handwriting recognition
US14/292,138 US20140361983A1 (en) 2013-06-09 2014-05-30 Real-time stroke-order and stroke-direction independent handwriting recognition

Publications (1)

Publication Number Publication Date
WO2014200736A1 true WO2014200736A1 (fr) 2014-12-18

Family

ID=52022661

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2014/040417 WO2014200736A1 (fr) 2013-06-09 2014-05-30 Gestion de reconnaissance d'écriture manuscrite en temps réel

Country Status (5)

Country Link
JP (8) JP6154550B2 (fr)
KR (6) KR20220003662A (fr)
CN (4) CN109614845B (fr)
HK (1) HK1220276A1 (fr)
WO (1) WO2014200736A1 (fr)

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016200592A1 (fr) * 2015-06-06 2016-12-15 Apple Inc. Systèmes et procédés permettant de générer et de fournir une heure intelligente pour laisser des rappels
JP2017215859A (ja) * 2016-06-01 2017-12-07 日本電信電話株式会社 文字列認識装置、方法、及びプログラム
WO2018075191A1 (fr) * 2016-10-17 2018-04-26 Google Llc Techniques de planification de modèles de langage et modèles de reconnaissance de caractères pour des entrées d'écriture manuscrite
US10884617B2 (en) 2016-06-12 2021-01-05 Apple Inc. Handwriting keyboard for screens
US11016658B2 (en) 2013-06-09 2021-05-25 Apple Inc. Managing real-time handwriting recognition
EP3839706A1 (fr) * 2019-12-20 2021-06-23 The Swatch Group Research and Development Ltd Procede et dispositif de determination de position d'un objet sur une surface donnee
US20210271939A1 (en) * 2019-05-16 2021-09-02 Boe Technology Group Co., Ltd. Method and system for converting font of chinese character in image, computer device and medium
US11112968B2 (en) 2007-01-05 2021-09-07 Apple Inc. Method, system, and graphical user interface for providing word recommendations
CN113673415A (zh) * 2021-08-18 2021-11-19 山东建筑大学 一种手写汉字身份鉴别方法及系统
US11188158B2 (en) 2017-06-02 2021-11-30 Samsung Electronics Co., Ltd. System and method of determining input characters based on swipe input
US11194467B2 (en) 2019-06-01 2021-12-07 Apple Inc. Keyboard management user interfaces
WO2022125342A1 (fr) * 2020-12-10 2022-06-16 Microsoft Technology Licensing, Llc Détection de gestes d'encre sur la base d'un traitement de données spatiales et d'image
WO2022131793A1 (fr) * 2020-12-18 2022-06-23 Samsung Electronics Co., Ltd. Procédé et appareil de reconnaissance d'entrées d'écriture manuscrite dans un environnement multi-utilisateur
US11437000B2 (en) 2017-05-19 2022-09-06 Semiconductor Energy Laboratory Co., Ltd. Machine learning method, machine learning system, and display system
EP4057182A1 (fr) * 2021-03-09 2022-09-14 Société BIC Retour sur l'écriture manuscrite
US11531454B2 (en) 2020-12-10 2022-12-20 Microsoft Technology Licensing, Llc Selecting content in ink documents using a hierarchical data structure
EP4145264A1 (fr) * 2021-09-07 2023-03-08 Ricoh Company, Ltd. Appareil d'affichage, moyen de support et procédé d'affichage
CN116646911A (zh) * 2023-07-27 2023-08-25 成都华普电器有限公司 应用于数字化电源并联模式的电流均流分配方法及系统

Families Citing this family (41)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10013603B2 (en) * 2016-01-20 2018-07-03 Myscript System and method for recognizing multiple object structure
KR102482850B1 (ko) * 2016-02-15 2022-12-29 삼성전자 주식회사 전자 장치 및 전자 장치의 필체 교정 기능 제공 방법
CN107220655A (zh) * 2016-03-22 2017-09-29 华南理工大学 一种基于深度学习的手写、印刷文本的分类方法
US20170308289A1 (en) * 2016-04-20 2017-10-26 Google Inc. Iconographic symbol search within a graphical keyboard
JP6728993B2 (ja) * 2016-05-31 2020-07-22 富士ゼロックス株式会社 筆記システム、情報処理装置、プログラム
CN106126092A (zh) * 2016-06-20 2016-11-16 联想(北京)有限公司 一种信息处理方法及电子设备
TWI633463B (zh) * 2016-06-20 2018-08-21 鴻海精密工業股份有限公司 文字輸入法
CN106527875B (zh) * 2016-10-25 2019-11-29 北京小米移动软件有限公司 电子记事方法及装置
KR102474245B1 (ko) * 2017-06-02 2022-12-05 삼성전자주식회사 스와이프 입력에 기초하여 입력 문자를 판단하는 시스템 및 방법
US10481791B2 (en) * 2017-06-07 2019-11-19 Microsoft Technology Licensing, Llc Magnified input panels
US20190155895A1 (en) * 2017-11-20 2019-05-23 Google Llc Electronic text pen systems and methods
CN107861684A (zh) * 2017-11-23 2018-03-30 广州视睿电子科技有限公司 书写识别方法、装置、存储介质及计算机设备
KR102008845B1 (ko) * 2017-11-30 2019-10-21 굿모니터링 주식회사 비정형 데이터의 카테고리 자동분류 방법
KR102053885B1 (ko) * 2018-03-07 2019-12-09 주식회사 엘렉시 판서 분석 시스템 및 방법과 어플리케이션
CN108710882A (zh) * 2018-05-11 2018-10-26 武汉科技大学 一种基于卷积神经网络的屏幕渲染文本识别方法
JP7298290B2 (ja) * 2018-06-19 2023-06-27 株式会社リコー 手書き入力表示装置、手書き入力表示方法およびプログラム
KR101989960B1 (ko) 2018-06-21 2019-06-17 가천대학교 산학협력단 복수 개의 기계학습 모델을 사용한 실시간 필기 인식 방법, 이를 실행시키기 위한 프로그램을 기록한 컴퓨터 판독 가능한 기록 매체 및 실시간 필기 인식 시스템
US11270486B2 (en) * 2018-07-02 2022-03-08 Apple Inc. Electronic drawing with handwriting recognition
CN109446780B (zh) * 2018-11-01 2020-11-27 北京知道创宇信息技术股份有限公司 一种身份认证方法、装置及其存储介质
CN109471587B (zh) * 2018-11-13 2020-05-12 掌阅科技股份有限公司 基于Java虚拟机的手写内容的显示方法及电子设备
CN109858323A (zh) * 2018-12-07 2019-06-07 广州光大教育软件科技股份有限公司 一种字符手写识别方法及系统
CN110009027B (zh) * 2019-03-28 2022-07-29 腾讯科技（深圳）有限公司 图像的比对方法、装置、存储介质及电子装置
CN110362247A (zh) * 2019-07-18 2019-10-22 江苏中威科技软件系统有限公司 一套在电子文件上放大签批的模式
KR20210017090A (ko) * 2019-08-06 2021-02-17 삼성전자주식회사 필기 입력을 텍스트로 변환하는 방법 및 전자 장치
CN110942089B (zh) * 2019-11-08 2023-10-10 东北大学 一种基于多级决策的击键识别方法
EP3828685B1 (fr) 2019-11-29 2022-09-28 MyScript Reconnaissance de course de geste dans une entrée d'interface utilisateur tactile
US20200251217A1 (en) * 2019-12-12 2020-08-06 Renee CASSUTO Diagnosis Method Using Image Based Machine Learning Analysis of Handwriting
CN111078073B (zh) * 2019-12-17 2021-03-23 科大讯飞股份有限公司 一种笔迹放大的方法和相关装置
CN111355715B (zh) * 2020-02-21 2021-06-04 腾讯科技（深圳）有限公司 待决议事件的处理方法、系统、装置、介质及电子设备
CN111736751B (zh) * 2020-08-26 2021-03-26 深圳市千分一智能技术有限公司 笔画重绘方法、设备及可读存储介质
US11627799B2 (en) * 2020-12-04 2023-04-18 Keith McRobert Slidable work surface
JP2022148901A (ja) * 2021-03-24 2022-10-06 カシオ計算機株式会社 文字認識装置、文字認識方法およびプログラム
KR20220135914A (ko) * 2021-03-31 2022-10-07 삼성전자주식회사 학습 기반의 필기 입력을 처리하는 전자 장치, 그 동작 방법 및 저장매체
CN113190161A (zh) * 2021-04-25 2021-07-30 无锡乐骐科技有限公司 一种基于卷积神经网络的电子书写练习方法
KR20220147832A (ko) * 2021-04-28 2022-11-04 삼성전자주식회사 필기 입력을 처리하는 전자 장치 및 이의 동작 방법
KR102366052B1 (ko) * 2021-05-28 2022-02-23 (유)벨류이 지연시간 감소 처리를 이용한 필기 시스템 및 방법, 그리고 이를 위한 첩 스프레드 스펙트럼 기반 저 복잡도 거리측정 알고리즘
KR102468713B1 (ko) * 2022-07-07 2022-11-21 주식회사 에이치투케이 학습자의 한글 필기에 대한 인공지능 기반의 획순 인식 장치 및 방법
WO2024014655A1 (fr) * 2022-07-14 2024-01-18 삼성전자 주식회사 Dispositif électronique et procédé d'identification de phrase exprimée par des traits
CN115291791A (zh) * 2022-08-17 2022-11-04 维沃移动通信有限公司 文本识别方法、装置、电子设备及存储介质
KR20240065997A (ko) * 2022-11-07 2024-05-14 삼성전자주식회사 필기 입력을 인식하는 방법 및 장치
CN117037186B (zh) * 2023-10-09 2024-01-30 山东维克特信息技术有限公司 一种患者数据管理系统

Family Cites Families (51)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0614372B2 (ja) * 1984-01-23 1994-02-23 日本電信電話株式会社 文字読取方法
JPS61272890A (ja) * 1985-05-29 1986-12-03 Canon Inc 手書文字認識装置
JP3353954B2 (ja) * 1993-08-13 2002-12-09 ソニー株式会社 手書き入力表示方法および手書き入力表示装置
JPH07160827A (ja) * 1993-12-09 1995-06-23 Matsushita Electric Ind Co Ltd 手書きストローク編集装置及び方法
JPH07200723A (ja) * 1993-12-29 1995-08-04 Canon Inc 文字認識方法及び装置
JPH0855182A (ja) * 1994-06-10 1996-02-27 Nippon Steel Corp 手書き文字入力装置
US5737443A (en) * 1994-11-14 1998-04-07 Motorola, Inc. Method of joining handwritten input
CN1102275C (zh) * 1994-11-14 2003-02-26 摩托罗拉公司 分解手写体输入的装置和方法
JP3333362B2 (ja) * 1995-04-11 2002-10-15 株式会社日立製作所 文字入力装置
TW338815B (en) 1995-06-05 1998-08-21 Motorola Inc Method and apparatus for character recognition of handwritten input
JP4115568B2 (ja) * 1996-12-18 2008-07-09 シャープ株式会社 文章入力装置
JPH10307675A (ja) * 1997-05-01 1998-11-17 Hitachi Ltd 手書き文字認識方法及び装置
US6970599B2 (en) * 2002-07-25 2005-11-29 America Online, Inc. Chinese character handwriting recognition system
JP4663903B2 (ja) * 2000-04-20 2011-04-06 パナソニック株式会社 手書き文字認識装置及び手書き文字認識プログラム並びに手書き文字認識プログラムを記録したコンピュータ読み取り可能な記録媒体
US7336827B2 (en) 2000-11-08 2008-02-26 New York University System, process and software arrangement for recognizing handwritten characters
US7286141B2 (en) 2001-08-31 2007-10-23 Fuji Xerox Co., Ltd. Systems and methods for generating and controlling temporary digital ink
JP4212270B2 (ja) * 2001-12-07 2009-01-21 シャープ株式会社 文字入力装置、文字入力方法および文字を入力するためのプログラム
US6986106B2 (en) 2002-05-13 2006-01-10 Microsoft Corporation Correction widget
JP2004213269A (ja) * 2002-12-27 2004-07-29 Toshiba Corp 文字入力装置
US8479112B2 (en) * 2003-05-13 2013-07-02 Microsoft Corporation Multiple input language selection
JP2005341387A (ja) * 2004-05-28 2005-12-08 Nokia Corp リアルタイムコミュニケーションシステム、及びリアルタイムコミュニケーションに用いる送受信装置及び方法
JP2006323502A (ja) 2005-05-17 2006-11-30 Canon Inc 情報処理装置及びその制御方法、プログラム
US7496547B2 (en) * 2005-06-02 2009-02-24 Microsoft Corporation Handwriting recognition using a comparative neural network
US7720316B2 (en) * 2006-09-05 2010-05-18 Microsoft Corporation Constraint-based correction of handwriting recognition errors
KR100859010B1 (ko) * 2006-11-01 2008-09-18 노키아 코포레이션 필기 인식을 위한 장치 및 방법
CN101311887A (zh) * 2007-05-21 2008-11-26 刘恩新 一种计算机手写输入系统及输入方法和编辑方法
CN101123044A (zh) * 2007-09-13 2008-02-13 无敌科技(西安)有限公司 汉字书写学习方法
JP2009110092A (ja) * 2007-10-26 2009-05-21 Alps Electric Co Ltd 入力処理装置
CN101178633A (zh) * 2007-12-13 2008-05-14 深圳华为通信技术有限公司 一种手写屏错误纠正的方法、系统及装置
US8116569B2 (en) 2007-12-21 2012-02-14 Microsoft Corporation Inline handwriting recognition and correction
US9355090B2 (en) * 2008-05-30 2016-05-31 Apple Inc. Identification of candidate characters for text input
CN101676838B (zh) * 2008-09-16 2012-05-23 夏普株式会社 输入装置
US20100166314A1 (en) * 2008-12-30 2010-07-01 Microsoft Corporation Segment Sequence-Based Handwritten Expression Recognition
US8391613B2 (en) * 2009-06-30 2013-03-05 Oracle America, Inc. Statistical online character recognition
JP2011065623A (ja) * 2009-08-21 2011-03-31 Sharp Corp 情報検索装置、情報検索装置の制御方法
CN101893987A (zh) * 2010-06-01 2010-11-24 华南理工大学 一种电子设备的手写输入方法
WO2012024829A1 (fr) 2010-08-24 2012-03-01 Nokia Corporation Procédé et appareil permettant de segmenter les traits d'une écriture manuscrite avec chevauchement en un ou plusieurs groupes
JP2012108871A (ja) 2010-10-26 2012-06-07 Nec Corp 情報処理装置およびその手書き入力処理方法
JP2013546081A (ja) * 2010-12-02 2013-12-26 ノキア コーポレイション 重ね書き入力に関する方法，装置，コンピュータプログラム製品
JP5550598B2 (ja) * 2011-03-31 2014-07-16 パナソニック株式会社 手書き文字入力装置
WO2012140935A1 (fr) * 2011-04-11 2012-10-18 Ｎｅｃカシオモバイルコミュニケーションズ株式会社 Dispositif d'entrée d'informations
CN102135838A (zh) * 2011-05-05 2011-07-27 汉王科技股份有限公司 分区输入手写字符串的方法和系统
US8977059B2 (en) 2011-06-03 2015-03-10 Apple Inc. Integrating feature extraction via local sequential embedding for automatic handwriting recognition
US8094941B1 (en) * 2011-06-13 2012-01-10 Google Inc. Character recognition for overlapping textual user input
EP2535844A3 (fr) * 2011-06-13 2016-10-19 Google, Inc. Reconnaissance de caractères pour entrée utilisateur textuelle superposée
US20130002553A1 (en) * 2011-06-29 2013-01-03 Nokia Corporation Character entry apparatus and associated methods
JP5330478B2 (ja) * 2011-10-14 2013-10-30 株式会社エヌ・ティ・ティ・ドコモ 入力支援装置、プログラム及び絵文字の入力支援方法
JP2013089131A (ja) * 2011-10-20 2013-05-13 Kyocera Corp 装置、方法及びプログラム
CN102566933A (zh) * 2011-12-31 2012-07-11 广东步步高电子工业有限公司 一种全屏手写有效区分命令手势和字符的方法
JP6102374B2 (ja) * 2013-03-15 2017-03-29 オムロン株式会社 読取文字訂正用のプログラムおよび文字読取装置
GB201704729D0 (en) 2017-03-24 2017-05-10 Lucite Int Uk Ltd Method of producing methyl methacrylate or methacrylic acid

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
NATARAJAN P ET AL: "MULTILINGUAL MACHINE PRINTED OCR", INTERNATIONAL JOURNAL OF PATTERN RECOGNITION AND ARTIFICIAL INTELLIGENCE (IJPRAI), WORLD SCIENTIFIC PUBLISHING, SI, vol. 15, no. 1, 1 February 2001 (2001-02-01), pages 43 - 63, XP001030221, ISSN: 0218-0014, DOI: 10.1142/S0218001401000745 *
PREM NATARAJAN ET AL: "Multi-lingual Offline Handwriting Recognition Using Hidden Markov Models: A Script-Independent Approach", 27 September 2006, ARABIC AND CHINESE HANDWRITING RECOGNITION; [LECTURE NOTES IN COMPUTER SCIENCE], SPRINGER BERLIN HEIDELBERG, BERLIN, HEIDELBERG, PAGE(S) 231 - 250, ISBN: 978-3-540-78198-1, XP019087058 *
SANTOSH K C ET AL: "A COMPREHENSIVE SURVEY ON ON-LINE HANDWRITING RECOGNITION TECHNOLOGY AND ITS REAL APPLICATION TO THE NEPALESE NATURAL HANDWRITING", 1 January 2009 (2009-01-01), pages 31 - 55, XP055141093, Retrieved from the Internet <URL:http://hal.archives-ouvertes.fr/docs/00/36/09/22/PDF/KUSET.pdf> [retrieved on 20140918] *
YOUSRI KESSENTINI ET AL: "A Multi-Stream HMM-based Approach for Off-line Multi-Script Handwritten Word Recognition", PROCEEDINGS OF THE ICFHR 2008, 1 January 2011 (2011-01-01), pages 147 - 152, XP055140677, Retrieved from the Internet <URL:http://www.cenparmi.concordia.ca/ICFHR2008/Proceedings/papers/cr1037.pdf> [retrieved on 20140916] *

Cited By (39)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11416141B2 (en) 2007-01-05 2022-08-16 Apple Inc. Method, system, and graphical user interface for providing word recommendations
US11112968B2 (en) 2007-01-05 2021-09-07 Apple Inc. Method, system, and graphical user interface for providing word recommendations
US11016658B2 (en) 2013-06-09 2021-05-25 Apple Inc. Managing real-time handwriting recognition
US11816326B2 (en) 2013-06-09 2023-11-14 Apple Inc. Managing real-time handwriting recognition
US11182069B2 (en) 2013-06-09 2021-11-23 Apple Inc. Managing real-time handwriting recognition
WO2016200592A1 (fr) * 2015-06-06 2016-12-15 Apple Inc. Systèmes et procédés permettant de générer et de fournir une heure intelligente pour laisser des rappels
CN107660291B (zh) * 2015-06-06 2021-12-03 苹果公司 用于生成并提供智能出发时间提醒的系统和方法
US11487421B2 (en) 2015-06-06 2022-11-01 Apple Inc. Systems and methods for generating and providing intelligent time to leave reminders
CN107660291A (zh) * 2015-06-06 2018-02-02 苹果公司 用于生成并提供智能出发时间提醒的系统和方法
US11880561B2 (en) 2015-06-06 2024-01-23 Apple Inc. Systems and methods for generating and providing intelligent time to leave reminders
JP2017215859A (ja) * 2016-06-01 2017-12-07 日本電信電話株式会社 文字列認識装置、方法、及びプログラム
US11941243B2 (en) 2016-06-12 2024-03-26 Apple Inc. Handwriting keyboard for screens
US20210124485A1 (en) * 2016-06-12 2021-04-29 Apple Inc. Handwriting keyboard for screens
US11640237B2 (en) 2016-06-12 2023-05-02 Apple Inc. Handwriting keyboard for screens
US10884617B2 (en) 2016-06-12 2021-01-05 Apple Inc. Handwriting keyboard for screens
WO2018075191A1 (fr) * 2016-10-17 2018-04-26 Google Llc Techniques de planification de modèles de langage et modèles de reconnaissance de caractères pour des entrées d'écriture manuscrite
US10325018B2 (en) 2016-10-17 2019-06-18 Google Llc Techniques for scheduling language models and character recognition models for handwriting inputs
US11437000B2 (en) 2017-05-19 2022-09-06 Semiconductor Energy Laboratory Co., Ltd. Machine learning method, machine learning system, and display system
US11188158B2 (en) 2017-06-02 2021-11-30 Samsung Electronics Co., Ltd. System and method of determining input characters based on swipe input
US11587343B2 (en) * 2019-05-16 2023-02-21 Boe Technology Group Co., Ltd. Method and system for converting font of Chinese character in image, computer device and medium
US20210271939A1 (en) * 2019-05-16 2021-09-02 Boe Technology Group Co., Ltd. Method and system for converting font of chinese character in image, computer device and medium
US11194467B2 (en) 2019-06-01 2021-12-07 Apple Inc. Keyboard management user interfaces
US11842044B2 (en) 2019-06-01 2023-12-12 Apple Inc. Keyboard management user interfaces
US11620046B2 (en) 2019-06-01 2023-04-04 Apple Inc. Keyboard management user interfaces
EP3839706A1 (fr) * 2019-12-20 2021-06-23 The Swatch Group Research and Development Ltd Procede et dispositif de determination de position d'un objet sur une surface donnee
US11301079B2 (en) 2019-12-20 2022-04-12 The Swatch Group Research And Development Ltd Method and device for determining the position of an object on a given surface
WO2022125342A1 (fr) * 2020-12-10 2022-06-16 Microsoft Technology Licensing, Llc Détection de gestes d'encre sur la base d'un traitement de données spatiales et d'image
US11587346B2 (en) 2020-12-10 2023-02-21 Microsoft Technology Licensing, Llc Detecting ink gestures based on spatial and image data processing
US11531454B2 (en) 2020-12-10 2022-12-20 Microsoft Technology Licensing, Llc Selecting content in ink documents using a hierarchical data structure
US11972097B2 (en) 2020-12-10 2024-04-30 Microsoft Technology Licensing, Llc Selecting content in ink documents using a hierarchical data structure
WO2022131793A1 (fr) * 2020-12-18 2022-06-23 Samsung Electronics Co., Ltd. Procédé et appareil de reconnaissance d'entrées d'écriture manuscrite dans un environnement multi-utilisateur
US11847401B2 (en) 2020-12-18 2023-12-19 Samsung Electronics Co., Ltd. Method and apparatus for recognizing handwriting inputs in multiple-user environment
US11675495B2 (en) 2021-03-09 2023-06-13 SOCIéTé BIC Handwriting feedback
EP4057182A1 (fr) * 2021-03-09 2022-09-14 Société BIC Retour sur l'écriture manuscrite
CN113673415B (zh) * 2021-08-18 2022-03-04 山东建筑大学 一种手写汉字身份鉴别方法及系统
CN113673415A (zh) * 2021-08-18 2021-11-19 山东建筑大学 一种手写汉字身份鉴别方法及系统
EP4145264A1 (fr) * 2021-09-07 2023-03-08 Ricoh Company, Ltd. Appareil d'affichage, moyen de support et procédé d'affichage
CN116646911A (zh) * 2023-07-27 2023-08-25 成都华普电器有限公司 应用于数字化电源并联模式的电流均流分配方法及系统
CN116646911B (zh) * 2023-07-27 2023-10-24 成都华普电器有限公司 应用于数字化电源并联模式的电流均流分配方法及系统

Also Published As

Publication number Publication date
JP7011747B2 (ja) 2022-01-27
KR101892723B1 (ko) 2018-08-29
KR20190090887A (ko) 2019-08-02
JP7361156B2 (ja) 2023-10-13
KR20200068755A (ko) 2020-06-15
JP6154550B2 (ja) 2017-06-28
KR20180097790A (ko) 2018-08-31
CN109614846A (zh) 2019-04-12
CN109614845A (zh) 2019-04-12
JP2017208101A (ja) 2017-11-24
CN109614845B (zh) 2023-08-04
JP6802876B2 (ja) 2020-12-23
KR102005878B1 (ko) 2019-07-31
KR102347064B1 (ko) 2022-01-04
CN109614847A (zh) 2019-04-12
JP2023182718A (ja) 2023-12-26
JP2016523406A (ja) 2016-08-08
CN105247540B (zh) 2018-10-16
CN105247540A (zh) 2016-01-13
JP2021168159A (ja) 2021-10-21
JP6903808B2 (ja) 2021-07-14
JP2022062064A (ja) 2022-04-19
JP2019164801A (ja) 2019-09-26
JP2021051759A (ja) 2021-04-01
KR102121487B1 (ko) 2020-06-11
JP2022116104A (ja) 2022-08-09
KR102221079B1 (ko) 2021-03-02
KR20160003112A (ko) 2016-01-08
JP7078808B2 (ja) 2022-05-31
JP6559184B2 (ja) 2019-08-14
KR20210024210A (ko) 2021-03-04
CN109614847B (zh) 2023-08-04
KR20220003662A (ko) 2022-01-10
HK1220276A1 (zh) 2017-04-28

Similar Documents

Publication Publication Date Title
US11182069B2 (en) Managing real-time handwriting recognition
US9934430B2 (en) Multi-script handwriting recognition using a universal recognizer
JP7361156B2 (ja) リアルタイム手書き認識の管理
US20140363082A1 (en) Integrating stroke-distribution information into spatial feature extraction for automatic handwriting recognition
US20140361983A1 (en) Real-time stroke-order and stroke-direction independent handwriting recognition

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14733034

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 20157033627

Country of ref document: KR

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2016518366

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14733034

Country of ref document: EP

Kind code of ref document: A1