JP2022116104A - Managing real-time handwriting recognition - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide methods, systems and computer-readable media for improving handwriting input functionality on a user device.

SOLUTION: In an operating environment of a handwriting input system, a handwriting recognition module is trained to have a repertoire comprising multiple non-overlapping scripts and capable of recognizing tens of thousands of characters using a single handwriting recognition model. The handwriting input module provides real-time, stroke-order and stroke-direction independent handwriting recognition for multi-character handwriting input. In particular, real-time, stroke-order and stroke-direction independent handwriting recognition is provided for multi-character or sentence level Chinese handwriting recognition.

SELECTED DRAWING: Figure 5

COPYRIGHT: (C)2022,JPO&INPIT

Description

TECHNICAL FIELD This specification relates to providing handwriting input functionality on computing devices, and more particularly, real-time, multi-script, stroke-order independent, handwriting recognition and input on computing devices. to provide the functionality of

Handwriting input is an important alternative input method for computing devices equipped with touch-sensitive surfaces (eg, touch-sensitive display screens or touchpads). Many users, especially those in some Asian or Arab countries, are accustomed to writing in cursive and may feel comfortable writing in normal handwriting as opposed to typing on a keyboard. can.

For certain logogram writing systems, such as Chinese Kanji and Japanese Kanji (also referred to as Chinese characters), alternative syllabic input methods (e.g., pinyin or kana) are used to write the corresponding logograms. If the user does not know how to phonetically spell the logogram, even if it is available for entering the characters of the writing system, and the incorrect phonetic expression of the logogram When using spelling, such a syllabic input method is inappropriate. Therefore, being able to use handwriting input on a computing device becomes extremely important for users who are unable to pronounce words well or not at all with respect to the relevant logogram writing system.

Although handwriting input functionality has become somewhat prevalent within certain regions of the world, there is still a need for improvement. Specifically, human handwriting is highly variable (eg, in terms of stroke order, size, typeface, etc.), and high-quality handwriting recognition software is complex and requires extensive training. Therefore, providing efficient real-time handwriting recognition on mobile devices with limited memory and computing resources has become a challenging task.

Furthermore, in today's multicultural world, users in many countries use multiple languages and write in more than one script (e.g. mentioning movie titles in English, writing messages in Chinese). ) may be required frequently. However, manually switching the recognition system to the desired script or language while writing is cumbersome and inefficient. Furthermore, the usefulness of conventional multi-script handwriting recognition techniques is severely limited, as extending a device's recognition capabilities to handle multiple scripts simultaneously increases the complexity of its recognition system. This is because it greatly increases the demand on computer resources, and on computer resources.

Furthermore, traditional handwriting techniques rely heavily on language or script-specific peculiarities to achieve recognition accuracy. Such peculiarities are not easily portable to other languages or scripts. Therefore, adding handwriting input capability for new languages or scripts is a difficult task that is not easily undertaken by software and device vendors. As a result, multilingual users are not provided with significant alternative input methods for their electronic devices.

A conventional user interface for providing handwriting input includes an area for accepting handwriting input from a user and an area for displaying handwriting recognition results. Significant improvements in user interfaces are still needed on portable devices with small form factors to improve efficiency, accuracy, and overall user experience.

This specification describes techniques for providing multi-script handwriting recognition using a universal recognizer. This universal recognizer is trained using a large multi-script corpus of written samples for characters in different languages or scripts. The training of this universal recognizer is language independent, script independent, stroke order independent, and stroke direction independent. Therefore, the same recognizer can recognize mixed-language, mixed-script handwritten input without requiring manual switching between input languages during use. Furthermore, this universal recognizer is lightweight enough to be deployed as a stand-alone module on mobile devices, allowing handwriting input in different languages and scripts used in different regions of the world. It becomes possible.

Furthermore, since this general recognizer is trained on spatially derived features of stroke order-independent and stroke direction-independent formulas and does not require temporal or sequence information at the stroke level, This universal recognizer provides a number of additional features and advantages over conventional time-based recognition methods (eg, Hidden Markov Method (HMM) based recognition methods). For example, a user can enter strokes of one or more characters, phrases, and sentences in any order and still get the same recognition result. Therefore, out-of-order multiple character entry and out-of-order modification (eg, addition or rewriting) of previously entered characters is now possible.

Furthermore, this universal recognizer can also be used for real-time handwriting recognition, where temporal information about each stroke is available and the ambiguity of the handwriting input is evaluated before character recognition is performed by the universal recognizer. is optionally used to remove or split handwriting input. The real-time, stroke-order-independent recognition described herein is distinct from traditional offline recognition methods (e.g., optical character recognition (OCR)), and is more accurate than traditional offline recognition methods. It can provide good performance. Furthermore, the general recognizer described herein explicitly expresses the salient features of various variations (e.g., variations in speed, slowness, stroke order, stroke direction, stroke continuity, etc.) within the recognition system. The recognition system's overall complexity is reduced.

As described herein, in some embodiments, temporally derived stroke distribution information is optionally reintroduced within the general purpose recognizer to improve recognition accuracy and Ambiguity between similar looking recognition outputs for images is removed. Although the reintroduction of temporally derived stroke distribution information does not compromise the stroke order and stroke direction independence of the universal recognizer, this is a significant difference between temporally derived features and spatially derived features. features are obtained through a separate training process and are combined in the handwriting recognition model only after the separate training process is completed. Furthermore, the temporally derived stroke distribution information captures the salient temporal properties of similar-looking characters without relying on explicit knowledge of the differences in stroke order of similar-looking characters. carefully designed to

A user interface for providing handwriting input functionality is also described herein.

In some embodiments, a method of providing multi-script handwriting recognition is training a multi-script handwriting recognition model based on spatially derived features of a multi-script training corpus, wherein the multi-script training training, wherein the corpus contains handwriting samples each corresponding to characters of at least three non-overlapping scripts; and that multiscript handwriting recognition is trained on spatially derived features of the multiscript training corpus. and using the model to provide real-time handwriting recognition for the user's handwriting input.

In some embodiments, a method of providing multi-script handwriting recognition is to receive a multi-script handwriting recognition model, the multi-script recognition model on spatially derived features of a multi-script training corpus. Receiving a multiscript handwriting recognition model that has been trained, wherein the multiscript training corpus includes respective handwriting samples corresponding to characters of at least three non-overlapping scripts; and receiving handwriting input from a user. receiving handwriting input, the handwriting input comprising one or more handwriting strokes provided on a touch-sensitive surface coupled to a user device; and in response to receiving the handwriting input, and providing one or more handwriting recognition results to the user in real-time based on a multi-script handwriting recognition model that has been trained on spatially derived features of the multi-script training corpus.

In some embodiments, a method of providing real-time handwriting recognition is receiving a plurality of handwriting strokes from a user, the plurality of handwriting strokes corresponding to a handwritten character. generating an input image based on the plurality of handwritten strokes; and providing the input image to a handwriting recognition model to perform real-time recognition of the handwritten characters, The handwriting recognition model provides stroke order independent handwriting recognition, providing an input image and receiving a plurality of handwritten strokes regardless of the respective order of the handwritten strokes received from a user. and displaying the same first output character in real time.

In some embodiments, the method includes receiving a second plurality of handwritten strokes from the user, wherein the second plurality of handwritten strokes correspond to the second handwritten character. and generating a second input image based on the second plurality of handwritten strokes; and providing the second input image to a handwriting recognition model to perform real-time recognition of the second handwritten character. and displaying, in real-time upon receiving the second plurality of handwritten strokes, a second output character corresponding to the second plurality of handwritten strokes, wherein the first output character and the second are simultaneously displayed in a spatial arrangement independent of the respective order of the first plurality of handwritten inputs and the second plurality of handwritten inputs provided by the user. and displaying.

In some embodiments, the second plurality of handwriting strokes spatially follows the first plurality of handwriting strokes along a default writing direction of the handwriting input interface of the user device; The character follows the first output character in spatial alignment along its default writing direction, and the method is to receive a third handwritten stroke from the user to correct the handwritten character. receiving a third handwritten stroke, wherein the third handwritten stroke is received later in time than the first plurality of handwritten strokes and the second plurality of handwritten strokes; In response to receiving the handwritten stroke, based on the relative proximity of the third handwritten stroke to the first plurality of handwritten strokes, the handwritten stroke into the same recognition unit as the first plurality of handwritten strokes. generating a corrected input image based on the first plurality of handwritten strokes and the third handwritten strokes; and performing real-time recognition of the corrected handwritten characters, the corrected handwritten characters. providing the corrected input image to a handwriting recognition model; and in response to receiving the third handwriting input, displaying a third output character corresponding to the corrected input image, the third Displaying a third output character, wherein the output character replaces the first output character and is displayed simultaneously with the second output character in spatial alignment along the default writing direction.

In some embodiments, the method includes providing a delete input from the user while the third output character and the second output character are simultaneously displayed as recognition results within the candidate display area of the handwriting input interface. and, in response to the deletion input, deleting the second output character from the recognition result while maintaining the third output character within the recognition result.

In some embodiments, within the handwriting input area of the handwriting input interface, a first plurality of handwriting strokes, a second plurality of handwriting strokes, and a third handwriting stroke are generated by the user. rendering in real-time as provided; and deleting respective renderings of the second plurality of handwritten strokes from the handwriting input area in response to receiving a deletion input, while deleting the first plurality of handwritten strokes. and maintaining respective renderings of the third handwriting stroke within the handwriting input area.

In some embodiments, a method of providing real-time handwriting recognition is receiving handwriting input from a user, the handwriting input being provided within a handwriting input area of a handwriting input interface. receiving handwritten input comprising handwritten strokes; identifying a plurality of output characters for the handwritten input based on a handwriting recognition model; and determining the plurality of output characters based on predefined categorization criteria. , into two or more categories, and displaying in an initial view of a candidate display area of a handwriting input interface each output character in a first of the two or more categories. a user selecting an affordance for displaying an output character and for invoking this extended view, wherein an initial view of this candidate display area is provided concurrently with an affordance for invoking an extended view of the candidate display area; receiving input and, in response to the user input, each output character in the first category in the expanded view of the candidate display area and previously displayed in the initial view of the candidate display area; and displaying each output character in at least a second of the two or more categories that was not.

In some embodiments, a method of providing real-time handwriting recognition is receiving handwriting input from a user, wherein the handwriting input is provided within a handwriting input area of a handwriting input interface. and recognizing a plurality of output characters from the handwritten input based on a handwriting recognition model, wherein the output characters are at least a first glyph and a natural human language recognizing a plurality of output characters comprising at least a first character from a script of a handwriting input interface, and outputting a recognition result comprising those first glyphs and a first character from a script of a natural human language to a handwriting input interface and displaying within the candidate display area of the

In some embodiments, a method of providing handwriting recognition is receiving handwriting input from a user, wherein the handwriting input is provided within a touch-sensitive surface coupled to the device, a plurality of handwriting strokes. receiving handwriting input, including rendering the handwriting strokes in real-time within the handwriting area of the handwriting input interface; pinch gesture input on the handwriting strokes; A first method based on the plurality of handwritten strokes by receiving one of the magnifying gesture input and, upon receiving the pinch gesture input, processing the plurality of handwritten strokes as a single recognition unit. generating a recognition result; and upon receiving the magnification gesture input, processing the plurality of handwriting strokes as two separate recognition units separated by the magnification gesture input, based on the plurality of handwriting strokes; generating the second recognition result; and displaying the generated recognition result in a candidate display area of the handwriting input interface upon generating each one of the first recognition result and the second recognition result. including doing.

In some embodiments, a method of providing handwriting recognition is receiving handwriting input from a user, the handwriting input comprising a plurality of handwriting strokes provided within a handwriting input area of a handwriting input interface. and identifying a plurality of recognition units from the plurality of handwritten strokes, each recognition unit including a corresponding subset of the plurality of handwritten strokes. generating a multi-character recognition result including each character recognized from the plurality of recognition units; and displaying the multi-character recognition result in a candidate display area of the handwriting input interface. , receiving a deletion input from the user while the recognition result of the plurality of characters is displayed in the candidate display area; and removing the last character from the recognition result.

In some embodiments, a method of providing real-time handwriting recognition includes determining an orientation of a device and providing a handwriting input interface on a device in a horizontal input mode according to the device being in a first orientation. providing a horizontal input mode in which each line of handwriting input entered in this horizontal input mode is divided into one or more respective recognition units along the horizontal writing direction and providing a handwriting input interface on the device in a vertical input mode according to the device being in a second orientation, each line of handwriting input entered in the vertical input mode comprising: and providing in a vertical input mode, divided into one or more respective recognition units along a vertical writing direction.

In some embodiments, a method of providing real-time handwriting recognition is receiving handwriting input from a user, wherein the handwriting input is provided on a touch-sensitive surface coupled to a device. receiving handwriting input, including strokes; rendering the plurality of handwriting strokes within a handwriting input area of a handwriting input interface; and rendering the plurality of handwriting strokes into two or more recognition units. dividing into two or more recognition units, each recognition unit including a corresponding subset of the plurality of handwritten strokes; receiving an edit request from a user; Means for visually identifying two or more recognition units within the handwriting input area and individually deleting each of the two or more recognition units from the handwriting input area in response to the edit request. and providing

In some embodiments, a method of providing real-time handwriting recognition is receiving a first handwriting input from a user, the first handwriting input comprising a plurality of handwriting strokes, the plurality of handwriting strokes receiving a first handwriting input, wherein the handwriting strokes form a plurality of recognition units distributed along respective writing directions associated with a handwriting input area of a handwriting input interface; rendering each of the plurality of handwriting strokes within the handwriting input area as provided by the user and respective fading for each of the plurality of recognition units after the recognition unit has been fully rendered; initiating a respective fading process, wherein rendering of recognition units in the first handwritten input gradually fades during each fading process; and a plurality of recognition units. receiving a second handwriting input from the user over the area of the handwriting input area occupied by the faded recognition units of the handwriting input area; Rendering the second handwriting within the handwriting and erasing all faded recognition units from the handwriting input area.

In some embodiments, a method of providing handwriting recognition is to separately train a set of spatially derived features and a set of temporally derived features of a handwriting recognition model. , the set of spatially derived features is trained on a corpus of training images, each an image of a handwriting sample for each character in the output character set, and the set of temporally derived features is the stroke training on a corpus of distribution profiles, each stroke distribution profile numerically characterizing the spatial distribution of a plurality of strokes within a handwriting sample for each character of the output character set; and in a handwriting recognition model, Combining a set of spatially derived features and a set of temporally derived features, training separately and using the handwriting recognition model to provide real-time handwriting recognition for a user's handwriting input Including things.

The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the present subject matter will become apparent from the description, drawings, and claims.

1 is a block diagram illustrating a portable multifunction device with a touch sensitive display, according to some embodiments; FIG.

1 illustrates a portable multifunction device with a touch-sensitive display, according to some embodiments;

1 is a block diagram of an exemplary multifunction device with a display and a touch sensitive surface, according to some embodiments; FIG.

4 illustrates an exemplary user interface for a multifunction device with a touch-sensitive surface separate from the display, according to some embodiments;

1 is a block diagram of an operating environment for a handwriting input system, according to some implementations; FIG.

1 is a block diagram of a multi-script handwriting recognition model, according to some embodiments; FIG.

FIG. 4 is a flow diagram of an exemplary process for training a multi-script handwriting recognition model, according to some embodiments; FIG.

1 illustrates an exemplary user interface showing real-time multi-script handwriting recognition and input on a portable multifunction device, according to some embodiments; 1 illustrates an exemplary user interface showing real-time multi-script handwriting recognition and input on a portable multifunction device, according to some embodiments;

FIG. 4 is a flow diagram of an exemplary process for providing real-time multi-script handwriting recognition and input on a portable multifunction device; FIG. FIG. 4 is a flow diagram of an exemplary process for providing real-time multi-script handwriting recognition and input on a portable multifunction device; FIG.

FIG. 4 is a flow diagram of an exemplary process for providing real-time, stroke-order independent handwriting recognition and input on a portable multifunction device, according to some embodiments; FIG. 4 is a flow diagram of an exemplary process for providing real-time, stroke-order independent handwriting recognition and input on a portable multifunction device, according to some embodiments; 1 is a flow diagram of an exemplary process for providing real-time, stroke-order independent handwriting recognition and input on a portable multifunction device, according to some embodiments;

Selectively displaying one category of recognition results in the normal view of the candidate display area and selectively displaying another category of recognition results in the expanded view of the candidate display area, according to some embodiments 1 illustrates an exemplary user interface for Selectively displaying one category of recognition results in the normal view of the candidate display area and selectively displaying another category of recognition results in the expanded view of the candidate display area, according to some embodiments 1 illustrates an exemplary user interface for Selectively displaying one category of recognition results in the normal view of the candidate display area and selectively displaying another category of recognition results in the expanded view of the candidate display area, according to some embodiments 1 illustrates an exemplary user interface for Selectively displaying one category of recognition results in the normal view of the candidate display area and selectively displaying another category of recognition results in the expanded view of the candidate display area, according to some embodiments 1 illustrates an exemplary user interface for Selectively displaying one category of recognition results in the normal view of the candidate display area and selectively displaying another category of recognition results in the expanded view of the candidate display area, according to some embodiments 1 illustrates an exemplary user interface for Selectively displaying one category of recognition results in the normal view of the candidate display area and selectively displaying another category of recognition results in the expanded view of the candidate display area, according to some embodiments 1 illustrates an exemplary user interface for Selectively displaying one category of recognition results in the normal view of the candidate display area and selectively displaying another category of recognition results in the expanded view of the candidate display area, according to some embodiments 1 illustrates an exemplary user interface for Selectively displaying one category of recognition results in the normal view of the candidate display area and selectively displaying another category of recognition results in the expanded view of the candidate display area, according to some embodiments 1 illustrates an exemplary user interface for Selectively displaying one category of recognition results in the normal view of the candidate display area and selectively displaying another category of recognition results in the expanded view of the candidate display area, according to some embodiments 1 illustrates an exemplary user interface for Selectively displaying one category of recognition results in the normal view of the candidate display area and selectively displaying another category of recognition results in the expanded view of the candidate display area, according to some embodiments 1 illustrates an exemplary user interface for Selectively displaying one category of recognition results in the normal view of the candidate display area and selectively displaying another category of recognition results in the expanded view of the candidate display area, according to some embodiments 1 illustrates an exemplary user interface for

Selectively displaying one category of recognition results in the normal view of the candidate display area and selectively displaying another category of recognition results in the expanded view of the candidate display area, according to some embodiments 1 is a flow diagram of an exemplary process for doing; Selectively displaying one category of recognition results in the normal view of the candidate display area and selectively displaying another category of recognition results in the expanded view of the candidate display area, according to some embodiments 1 is a flow diagram of an exemplary process for doing;

4 illustrates an exemplary user interface for entering glyphs through handwriting input, according to some embodiments; 4 illustrates an exemplary user interface for entering glyphs through handwriting input, according to some embodiments; 4 illustrates an exemplary user interface for entering glyphs through handwriting input, according to some embodiments; 4 illustrates an exemplary user interface for entering glyphs through handwriting input, according to some embodiments; 4 illustrates an exemplary user interface for entering glyphs through handwriting input, according to some embodiments;

FIG. 4 is a flow diagram of an exemplary process for entering glyphs through handwriting input according to some embodiments; FIG.

Using a pinch gesture or a magnifying gesture, according to some embodiments, to inform the handwriting input module how to separate the currently accumulated handwriting input into one or more units of recognition. 1 shows an exemplary user interface for Using a pinch gesture or a magnifying gesture, according to some embodiments, to inform the handwriting input module how to separate the currently accumulated handwriting input into one or more units of recognition. 1 shows an exemplary user interface for Using a pinch gesture or a magnifying gesture, according to some embodiments, to inform the handwriting input module how to separate the currently accumulated handwriting input into one or more units of recognition. 1 shows an exemplary user interface for Using a pinch gesture or a magnifying gesture, according to some embodiments, to inform the handwriting input module how to separate the currently accumulated handwriting input into one or more units of recognition. 1 shows an exemplary user interface for Using a pinch gesture or a magnifying gesture, according to some embodiments, to inform the handwriting input module how to separate the currently accumulated handwriting input into one or more units of recognition. 1 shows an exemplary user interface for Using a pinch gesture or a magnifying gesture, according to some embodiments, to inform the handwriting input module how to separate the currently accumulated handwriting input into one or more units of recognition. 1 shows an exemplary user interface for Using a pinch gesture or a magnifying gesture, according to some embodiments, to inform the handwriting input module how to separate the currently accumulated handwriting input into one or more units of recognition. 1 shows an exemplary user interface for Using a pinch gesture or a magnifying gesture, according to some embodiments, to inform the handwriting input module how to separate the currently accumulated handwriting input into one or more units of recognition. 1 shows an exemplary user interface for Using a pinch gesture or a magnifying gesture, according to some embodiments, to inform the handwriting input module how to separate the currently accumulated handwriting input into one or more units of recognition. 1 shows an exemplary user interface for Using a pinch gesture or a magnifying gesture, according to some embodiments, to inform the handwriting input module how to separate the currently accumulated handwriting input into one or more units of recognition. 1 shows an exemplary user interface for Using a pinch gesture or a magnifying gesture, according to some embodiments, to inform the handwriting input module how to separate the currently accumulated handwriting input into one or more units of recognition. 1 shows an exemplary user interface for

Using a pinch gesture or a magnifying gesture, according to some embodiments, to inform the handwriting input module how to separate the currently accumulated handwriting input into one or more units of recognition. 1 is a flow diagram of an exemplary process for Using a pinch gesture or a magnifying gesture, according to some embodiments, to inform the handwriting input module how to separate the currently accumulated handwriting input into one or more units of recognition. 1 is a flow diagram of an exemplary process for

4 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input, according to some embodiments. 4 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input, according to some embodiments. 4 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input, according to some embodiments. 4 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input, according to some embodiments. 4 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input, according to some embodiments. 4 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input, according to some embodiments. 4 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input, according to some embodiments. 4 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input, according to some embodiments.

FIG. 4 is a flow diagram of an exemplary process for providing character-by-character deletion of a user's handwriting input, according to some embodiments; FIG. 4 is a flow diagram of an exemplary process for providing character-by-character deletion of a user's handwriting input, according to some embodiments;

4 illustrates an exemplary user interface for switching between vertical writing mode and horizontal writing mode, according to some embodiments. 4 illustrates an exemplary user interface for switching between vertical writing mode and horizontal writing mode, according to some embodiments. 4 illustrates an exemplary user interface for switching between vertical writing mode and horizontal writing mode, according to some embodiments. 4 illustrates an exemplary user interface for switching between vertical writing mode and horizontal writing mode, according to some embodiments. 4 illustrates an exemplary user interface for switching between vertical writing mode and horizontal writing mode, according to some embodiments. 4 illustrates an exemplary user interface for switching between vertical writing mode and horizontal writing mode, according to some embodiments.

FIG. 6 is a flow diagram of an exemplary process for switching between vertical and horizontal write modes according to some embodiments; FIG. FIG. 6 is a flow diagram of an exemplary process for switching between vertical and horizontal write modes according to some embodiments; FIG. FIG. 6 is a flow diagram of an exemplary process for switching between vertical and horizontal write modes according to some embodiments; FIG.

4 illustrates a user interface for displaying and providing means for selectively deleting individual recognition units identified within a user's handwriting input, according to some embodiments. 4 illustrates a user interface for displaying and providing means for selectively deleting individual recognition units identified within a user's handwriting input, according to some embodiments. 4 illustrates a user interface for displaying and providing means for selectively deleting individual recognition units identified within a user's handwriting input, according to some embodiments. 4 illustrates a user interface for displaying and providing means for selectively deleting individual recognition units identified within a user's handwriting input, according to some embodiments. 4 illustrates a user interface for displaying and providing means for selectively deleting individual recognition units identified within a user's handwriting input, according to some embodiments. 4 illustrates a user interface for displaying and providing means for selectively deleting individual recognition units identified within a user's handwriting input, according to some embodiments. 4 illustrates a user interface for displaying and providing means for selectively deleting individual recognition units identified within a user's handwriting input, according to some embodiments. 4 illustrates a user interface for displaying and providing means for selectively deleting individual recognition units identified within a user's handwriting input, according to some embodiments.

FIG. 4 is a flow diagram of an exemplary process for providing means for displaying and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments; FIG. FIG. 4 is a flow diagram of an exemplary process for providing means for displaying and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments; FIG.

Implicit confirmation input for entering new handwriting input provided over existing handwriting input in the handwriting input area, according to some embodiments, with displayed recognition results for that existing handwriting input 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for entering new handwriting input provided over existing handwriting input in the handwriting input area, according to some embodiments, with displayed recognition results for that existing handwriting input 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for entering new handwriting input provided over existing handwriting input in the handwriting input area, according to some embodiments, with displayed recognition results for that existing handwriting input 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for entering new handwriting input provided over existing handwriting input in the handwriting input area, according to some embodiments, with displayed recognition results for that existing handwriting input 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for entering new handwriting input provided over existing handwriting input in the handwriting input area, according to some embodiments, with displayed recognition results for that existing handwriting input 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for entering new handwriting input provided over existing handwriting input in the handwriting input area, according to some embodiments, with displayed recognition results for that existing handwriting input 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for entering new handwriting input provided over existing handwriting input in the handwriting input area, according to some embodiments, with displayed recognition results for that existing handwriting input 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for entering new handwriting input provided over existing handwriting input in the handwriting input area, according to some embodiments, with displayed recognition results for that existing handwriting input 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for entering new handwriting input provided over existing handwriting input in the handwriting input area, according to some embodiments, with displayed recognition results for that existing handwriting input 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for entering new handwriting input provided over existing handwriting input in the handwriting input area, according to some embodiments, with displayed recognition results for that existing handwriting input 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for entering new handwriting input provided over existing handwriting input in the handwriting input area, according to some embodiments, with displayed recognition results for that existing handwriting input 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for entering new handwriting input provided over existing handwriting input in the handwriting input area, according to some embodiments, with displayed recognition results for that existing handwriting input 1 illustrates an exemplary user interface for use as a .

Implicit confirmation input for entering new handwriting input provided over existing handwriting input in the handwriting input area, according to some embodiments, with displayed recognition results for that existing handwriting input 1 is a flow diagram of an exemplary process for use as a. Implicit confirmation input for entering new handwriting input provided over existing handwriting input in the handwriting input area, according to some embodiments, with displayed recognition results for that existing handwriting input 1 is a flow diagram of an exemplary process for use as a.

Integrate temporally derived stroke distribution information into a spatially derived feature-based handwriting recognition model without compromising the stroke order and stroke direction dependencies of the handwriting recognition model, according to some embodiments. 1 is a flow diagram of an exemplary process for doing; Integrate temporally derived stroke distribution information into a spatially derived feature-based handwriting recognition model without compromising the stroke order and stroke direction dependencies of the handwriting recognition model, according to some embodiments. 1 is a flow diagram of an exemplary process for doing;

FIG. 4 is a block diagram illustrating separate training of spatially derived features and temporally derived features and subsequent integration of an exemplary handwriting recognition system, according to some embodiments;

FIG. 4 is a block diagram illustrating an exemplary method for computing stroke distribution profiles of characters;

Like reference numbers refer to corresponding parts throughout the drawings.

Many electronic devices have a graphical user interface with a soft keyboard for entering characters. On some electronic devices, the user may also be able to install or enable a handwriting input interface that allows the user to use a touch-sensitive display coupled to the device. Characters can be entered via handwriting on a screen or touch-sensitive surface. Conventional handwriting recognition input methods and user interfaces have several problems and shortcomings. for example,
• In general, traditional handwriting input functionality is enabled per language or per script. Each additional input language requires installation of a separate handwriting recognition model, which consumes separate storage space and memory. Combining handwriting recognition models for different languages provides little synergy, and mixed-language or mixed-script handwriting recognition traditionally takes a very long time due to the complex disambiguation process. It was something.
• Moreover, conventional handwriting recognition systems rely heavily on language-specific or script-specific properties for character recognition. Recognition of mixed-language handwritten input had low accuracy. Furthermore, the available combinations of recognized languages are very limited. Most systems required the user to manually specify the desired language-specific handwriting recognizer before providing handwriting input in each non-default language or script.
• Many existing real-time handwriting recognition models require temporal or sequence information at the stroke-by-stroke level, which leads to high variability in how characters can be written (e.g. typeface or individual (high variability in stroke shape, length, speed, segmentation, order, and direction due to quirks) can lead to inaccurate recognition results. Some systems also allow users to set rigid spatial and temporal criteria (e.g., with built-in assumptions on the size, alignment, and timeframe of each character input) when providing handwriting input. It also requires compliance. Any deviation from these criteria caused inaccurate recognition results that were difficult to correct.
• Currently, most real-time handwriting input interfaces only allow users to input a small number of characters at a time. Input of long phrases or sentences is broken into short segments and input separately. Not only does this formal input impose a cognitive burden on the user to maintain compositional flow, but it also makes it difficult for the user to modify or correct previously entered characters or phrases.

The embodiments described below address these and related issues.

Figures 1-4 below provide a description of an exemplary device. Figures 5, 6 and 26, 27 illustrate exemplary handwriting recognition and input systems. 8A, 8B, 11A-11K, 13A-13E, 15A-15K, 17A-17F, 19A-19F, 21A-21H, and 23A-12L are drawn by hand. 4 illustrates an exemplary user interface for recognition and input; 7, 9A, 9B, 10A-10C, 12A, 12B, 14, 16A, 16B, 18A, 18B, 20A-20C, 22A, 22B, 24A , FIG. 24B, and FIG. 25 train a handwriting recognition model, provide real-time handwriting recognition results, provide a means to enter and correct handwriting input, and input recognition results as text input. Fig. 3 is a flow diagram showing a method of enabling handwriting recognition and input including providing means; 8A, 8B, 11A-11K, 13A-13E, 15A-15K, 17A-17F, 19A-19F, 21A-21H, 23A-12L 7, 9A, 9B, 10A-10C, 12A, 12B, 14, 16A, 16B, 18A, 18B, 20A-20C, 22A, 22B , are used to illustrate the process in FIGS. 24A, 24B, and 25 .
Exemplary device

Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

It is also understood that, although the terms first, second, etc. may be used herein to describe various elements, those elements should not be limited by these terms. will be done. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the present invention. The first contact and the second contact are both contacts, but they are not the same contact.

The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the description of the present invention and in the appended claims, the singular forms "a," "an," and "the" are also used in the plural unless the context clearly indicates otherwise. intended to include As used herein, the term "and/or" refers to and includes any and all possible combinations of one or more of the associated listed items, also including will be understood. The terms "includes," "including," "comprises," and/or "comprising," as used herein, refer to the features, integers, steps, or , acts, elements and/or components, but excludes the presence or addition of one or more other features, integers, steps, acts, elements, components and/or groups thereof It will be further understood that no

As used herein, the term "in the event of" is, depending on the context, "when" or "when" or "in response to a determination" or "on detection of may be construed to mean "according to". Similarly, the phrases "when determined to" or "when (the condition or event described) is detected" are replaced by "when determined to" or "when determined to" or "when (the condition or event described) is detected" or "on detection of (the condition or event described)" or "on detection of (the condition or event described)" .

Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device such as a mobile phone that also includes PDA functionality and/or other functionality such as music player functionality. An exemplary embodiment of a portable multifunction device is the Apple Inc. (Cupertino, Calif.), iPhone®, iPod Touch®, and iPad® devices. Other portable electronic devices, such as laptop computers or tablet computers with touch-sensitive surfaces (eg, touch screen displays and/or touch pads) can also be used. It should also be appreciated that in some embodiments the device is a desktop computer with a touch-sensitive surface (eg, a touchscreen display and/or a touchpad) rather than a portable communication device.

The following discussion describes electronic devices that include displays and touch-sensitive surfaces. However, it should be appreciated that the electronic device may include one or more other physical user interface devices such as a physical keyboard, mouse and/or joystick.

The device is typically used for drawing applications, presentation applications, word processing applications, website creation applications, disc authoring applications, spreadsheet applications, gaming applications, telephony applications, video conferencing applications, e-mail applications, instant messaging applications, Various applications are supported, such as one or more of a training support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application.

Various applications that can run on this device can use at least one common physical user interface device, such as a touch-sensitive surface. One or more features of the touch-sensitive surface and corresponding information displayed on the device may be adjusted and/or changed from application to application and/or within each application. In this manner, the device's common physical architecture (such as the touch-sensitive surface) can support a variety of applications with a user interface that is intuitive and transparent to the user.

Attention is now directed to embodiments of portable devices with touch-sensitive displays. FIG. 1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display 112, according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a "touchscreen" for convenience, and is also known or sometimes referred to as a touch-sensitive display system. Device 100 includes memory 102 (which may include one or more computer-readable storage media), memory controller 122, one or more processing units (CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry 110, It may include speaker 111 , microphone 113 , input/output (I/O) subsystem 106 , other input or control devices 116 as well as external ports 124 . Device 100 may include one or more photosensors 164 . These components may communicate via one or more communication buses or signal lines 103 .

Device 100 is only one example of a portable multifunction device and device 100 may have more or fewer components than shown and may include two or more components. It should be understood that they can be combined or have different configurations or arrangements of their components. The various components shown in FIG. 1 can be implemented as hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and/or application specific integrated circuits. .

Memory 102 may include high speed random access memory and may also include non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile semiconductor memory devices. Access to memory 102 by other components of device 100 , such as CPU 120 and peripherals interface 118 , may be controlled by memory controller 122 .

Peripheral interface 118 may be used to couple the device's input and output peripherals to CPU 120 and memory 102 . One or more processors 120 run or execute various software programs and/or instruction sets stored in memory 102 to perform various functions and process data for device 100 .

In some embodiments, peripheral interface 118, CPU 120, and memory controller 122 may be implemented on a single chip, such as chip 104. FIG. In some other embodiments they may be implemented on separate chips.

RF (Radio Frequency) circuitry 108 transmits and receives RF signals, also called electromagnetic signals. RF circuitry 108 converts electrical signals to/from electromagnetic signals and communicates via the electromagnetic signals with communications networks and other communications devices.

Audio circuitry 110 , speaker 111 , and microphone 113 provide an audio interface between a user and device 100 . Audio circuitry 110 receives audio data from peripherals interface 118 , converts the audio data to electrical signals, and transmits the electrical signals to speaker 111 . The speaker 111 converts the electrical signal into human audible sound waves. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113 . Audio circuitry 110 converts the electrical signal to audio data and transmits the audio data to peripherals interface 118 for processing. Audio data may be retrieved from memory 102 and/or RF circuitry 108 and/or transmitted to memory 102 and/or RF circuitry 108 by peripherals interface 118 . In some embodiments, audio circuitry 110 also includes a headset jack (eg, 212 in FIG. 2).

I/O subsystem 106 couples input/output peripherals on device 100 , such as touch screen 112 and other input control devices 116 , to peripherals interface 118 . The I/O subsystem 106 may include a display controller 156 as well as one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive/send electrical signals to/from other input or control devices 116 . Other input control devices 116 may include physical buttons (eg, push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and the like. In some alternative embodiments, input controller 160 may be coupled to any (or none) of a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. ). One or more buttons (eg, 208 in FIG. 2) may include up/down buttons for volume control of speaker 111 and/or microphone 113 . One or more buttons may include a push button (eg, 206 in FIG. 2).

Touch-sensitive display 112 provides an input and output interface between the device and the user. Display controller 156 receives and/or transmits electrical signals to and from touch screen 112 . Touch screen 112 displays visual output to the user. This visual output may include graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of these visual outputs may correspond to user interface objects.

Touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on tactile and/or tactile contact. Touchscreen 112 and display controller 156 (together with any associated modules and/or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on touchscreen 112 and Contacts are translated into interactions with user interface objects (eg, one or more softkeys, icons, web pages, or images) displayed on touch screen 112 . In an exemplary embodiment, the contact point between touch screen 112 and the user corresponds to the user's finger.

The touch screen 112 may use LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, although other display technologies may be used in other embodiments. can. Touch screen 112 and display controller 156 can detect contact, and any movement or interruption of that contact, using any of a number of now known or later developed touch sensing technologies; These techniques include capacitive, resistive, infrared, and surface ultrasonic techniques, as well as other proximity sensor arrays or other sensors for determining one or more points of contact with the touchscreen 112. Elements include, but are not limited to. In an exemplary embodiment, Apple Inc. (Cupertino, Calif.), projected mutual capacitive sensing technology is used, such as that found in the iPhone®, iPod Touch®, and iPad®.

Touch screen 112 may have a video resolution greater than 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi. A user may contact touch screen 112 using any suitable object or attachment, such as a stylus, finger, or the like. In some embodiments, user interfaces are designed to work primarily with finger-based contacts and gestures, but these are less susceptible to stylus-based input due to the greater finger contact area on touchscreens. may be less accurate than In some embodiments, the device converts coarse finger-based input into precise pointer/cursor positions or commands to perform actions desired by the user. Handwriting input can be provided on the touch screen 112 through finger-based contact or stylus-based contact location and movement. In some embodiments, the touch screen 112 renders finger-based or stylus-based input as immediate visual feedback to its current handwriting input and provides a writing surface using a writing instrument (e.g., pen). Provides a visual effect of actual writing on (eg, a sheet of paper).

In some embodiments, in addition to the touchscreen, device 100 may include a touchpad (not shown) for activating or deactivating certain functions. In some embodiments, the touchpad is the touch-sensitive area of the device that displays no visual output, unlike a touchscreen. The touchpad can be a touch-sensitive surface separate from the touchscreen 112 or an extension of the touch-sensitive surface formed by the touchscreen.

Device 100 also includes a power system 162 for powering the various components. The power system 162 includes a power management system, one or more power sources (eg, batteries, alternating current (AC)), recharging systems, power failure detection circuitry, power converters or inverters, power status indicators (eg, light emitting diodes (LEDs)). , and any other components associated with the generation, management, and distribution of power within the portable device.

Device 100 may also include one or more optical sensors 164 . FIG. 1 shows a photosensor coupled to photosensor controller 158 in I/O subsystem 106 . Photosensor 164 may include a charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS) phototransistor. Optical sensor 164 receives light from the environment projected through one or more lenses and converts the light into data representing an image. In conjunction with imaging module 143 (also called a camera module), optical sensor 164 can capture still images or video.

Device 100 may also include one or more proximity sensors 166 . FIG. 1 shows proximity sensor 166 coupled to peripherals interface 118 . Alternatively, proximity sensor 166 may be coupled to input controller 160 within I/O subsystem 106 . In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call), the proximity sensor is disabled by turning off the touchscreen 112. do.

Device 100 may also include one or more accelerometers 168 . FIG. 1 shows accelerometer 168 coupled to peripherals interface 118 . Alternatively, accelerometer 168 may be coupled to input controller 160 within I/O subsystem 106 . In some embodiments, information is displayed on the touch screen display in portrait or landscape orientation based on analysis of data received from one or more accelerometers. In addition to the accelerometer 168, the device 100 optionally includes a magnetometer (not shown) and a GPS (or , GLONASS or other global navigation system) receiver (not shown).

In some embodiments, the software components stored in memory 102 include operating system 126, communication module (or instruction set) 128, touch/motion module (or instruction set) 130, graphics module (or instruction set), ) 132 , a text input module (or instruction set) 134 , a global positioning system (GPS) module (or instruction set) 135 , and an application (or instruction set) 136 . Additionally, in some embodiments, memory 102 stores handwriting input module 157, as shown in FIGS. Handwriting input module 157 includes a handwriting recognition model and provides handwriting recognition and input functionality to the user of device 100 (or device 300). Further details of handwriting input module 157 are provided in connection with FIGS. 5-27 and their accompanying descriptions.

Operating system 126 (e.g., Darwin®, RTXC®, LINUX®, UNIX®, OS X®, WINDOWS®, VxWorks®, etc.) embedded operating systems) contain various software components and/or drivers for controlling and managing common system tasks (e.g., memory management, storage device control, power management, etc.), and various hardware Facilitates communication between components and software components.

Communications module 128 facilitates communication with other devices via one or more external ports 124 and includes various software for processing data received by RF circuitry 108 and/or external ports 124 . Also includes components. External port 124 (e.g., Universal Serial Bus (USB), FIREWIRE®, etc.) is for direct coupling to other devices or indirectly via a network (e.g., Internet, wireless LAN, etc.) adapted to

Contact/motion module 130 can detect contact with touch screen 112 (in conjunction with display controller 156) and contact with other touch-sensitive devices (eg, a touchpad or physical click wheel). The contact/motion module 130 determines whether contact has occurred (e.g., detecting a finger down event), determines whether there is movement of the contact, and movement across the touch-sensitive surface. (e.g., detecting one or more finger dragging events) and determining whether contact has ceased (e.g., detecting a finger up event or interruption of contact) and various software components for performing various operations related to contact detection. Contact/motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point, represented by a series of contact data, can be done by measuring the speed (magnitude), velocity (magnitude and direction), and/or acceleration (change in magnitude and/or direction) of the contact point. ). These operations can be applied to single contacts (eg, single-finger contacts) or multiple simultaneous contacts (eg, "multi-touch"/multi-finger contacts). In some embodiments, contact/motion module 130 and display controller 156 detect contact on the touchpad.

The contact/motion module 130 can detect gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns. Therefore, gestures can be detected by detecting specific touch patterns. For example, detecting a finger tap gesture involves detecting a finger down event followed by a finger at the same position (or substantially the same position) as the finger down event (e.g., at an icon position). , including detecting an event that raises (lifts off) . As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger down event, followed by detecting one or more finger dragging events, followed by a finger dragging event. , including detecting an event that raises (lifts off) .

The contact/motion module 130 detects a touch within the handwriting input area of the handwriting input interface displayed on the touch-sensitive display screen 112 (or, in FIG. 3, corresponding to the handwriting input area displayed on the display 340). (within the area of pad 355) is optionally utilized by handwriting input module 157 to register input of handwritten strokes. In some embodiments, the locations, motion paths, and intensities associated with the contact during the first finger down event, the last finger up event, and any time in between are recorded as handwritten strokes. be. Based on such information, handwritten strokes can be rendered on the display as feedback regarding that user input. Additionally, one or more input images can be generated based on handwriting strokes registered by contact/motion module 130 .

Graphics module 132 includes various known software components for rendering and displaying graphics on touch screen 112 or other displays, including components for changing the brightness of displayed graphics. . As used herein, the term "graphic" includes any object that can be displayed to a user, including text, web pages, icons (user interface objects including soft keys) etc.), digital images, videos, animations, etc.

In some embodiments, graphics module 132 stores data representing the graphics used. Each graphic can be assigned a corresponding code. Graphics module 132 receives one or more codes specifying the graphics to be displayed, along with coordinate data and other graphics characteristic data as appropriate, such as from an application, and then outputs to display controller 156 . Generate screen image data.

The text input module 134 can be a component of the graphics module 132 and can be used for various applications such as contacts 137, email 140, IM 141, browser 147, and any other application requiring text input. ) to provide a soft keyboard for entering text. In some embodiments, handwriting input module 157 is optionally invoked through the user interface of text input module 134, eg, through a keyboard selection affordance. In some embodiments, the same or similar keyboard selection affordances are also provided within the handwriting input interface to invoke text input module 134 .

The GPS module 135 determines the location of the device and passes this information for use in various applications (e.g., to the phone 138 for use in location-based dialing, as photo/video metadata for camera 143 and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and map/navigation widgets).

Application 136 may include the following modules (or instruction sets), or subsets or supersets thereof: contact module 137 (sometimes called address book or contact list), phone module 138; videoconferencing module; instant messaging (IM) module 141; training support module 142; camera module 143 for still and/or video images; image management module 144; browser module 147; -1, Stock Widget 149-2, Calculator Widget 149-3, Alarm Clock Widget 149-4, Dictionary Widget 149-5, and other widgets obtained by the user, and one of User Created Widgets 149-6. a widget creation module 150 for creating user-created widgets 149-6; a search module 151; a video and music player module 152, which may consist of a video player module and a music player module; a notes module 153; a map module 154; and/or an online video module 155.

Examples of other applications 136 that may be stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA enabled applications, cryptography, digital writing. Rights management, speech recognition, and speech replication.

In conjunction with touch screen 112 , display controller 156 , contact module 130 , graphics module 132 , handwriting input module 157 , and text input module 134 , contact module 137 (eg, contact module in memory 102 or memory 370 137 (stored within the application internal state 192 of 137)) to manage the address book or contact list, including adding names to the address book, deleting names from the address book, and so on. associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; sorting and sorting names; or providing a phone number or email address to initiate and/or facilitate communication via IM 141 or the like.

In conjunction with RF circuitry 108 , audio circuitry 110 , speaker 111 , microphone 113 , touch screen 112 , display controller 156 , touch module 130 , graphics module 132 , handwriting input module 157 and text input module 134 , telephony module 138 . enters a series of characters corresponding to a phone number, accesses one or more phone numbers in the address book 137, modifies the entered phone number, dials each phone number, and conducts a conversation. and can be used to disconnect or hang up when the conversation is complete. As noted above, this wireless communication may use any of a number of communication standards, communication protocols, and communication techniques.

RF circuitry 108 , audio circuitry 110 , speaker 111 , microphone 113 , touch screen 112 , display controller 156 , optical sensor 164 , optical sensor controller 158 , touch module 130 , graphics module 132 , handwriting input module 157 , text input module 134 . , contact list 137, and telephony module 138, the videoconferencing module 139 initiates, conducts, and ends videoconferences between the user and one or more other participants according to the user's instructions. contains executable instructions for

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, email client module 140, in response to user instructions, Contains executable instructions for creating, sending, receiving, and managing email. In conjunction with image management module 144 , email client module 140 greatly facilitates composing and sending emails with still or video images captured by camera module 143 .

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, instant messaging module 141 generates a series of characters corresponding to an instant message. to modify previously entered characters (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for telephone-based instant messaging, or Internet-based instant messaging contains executable instructions for sending instant messages, receiving instant messages, and viewing received instant messages, respectively (using XMPP, SIMPLE, or IMPS for ). In some embodiments, the instant messages sent and/or received are graphics, photos, audio files, video files, and/or other May contain attachments. As used herein, "instant messaging" refers to telephone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., using XMPP, SIMPLE, or IMPS). (messages sent as

Training in conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, GPS module 135, map module 154, and music player module 146. Support module 142 is used to create workouts (eg, with time, distance, and/or calorie expenditure goals), communicate with workout sensors (sports devices), receive workout sensor data, and monitor workouts. includes executable instructions for calibrating the sensors used, selecting and playing training music, and displaying, storing, and transmitting training data.

In conjunction with touch screen 112, display controller 156, light sensor 164, light sensor controller 158, touch module 130, graphics module 132, and image management module 144, camera module 143 can capture still images or video (including video streams). , store them in memory 102 , modify the characteristics of still images or video, or delete still images or video from memory 102 .

In conjunction with touch screen 112, display controller 156, touch module 130, graphics module 132, handwriting input module 157, text input module 134, and camera module 143, image management module 144 arranges still and/or video images. contains executable instructions to modify (eg, edit) or otherwise manipulate, label, delete, present (eg, in a digital slideshow or album), and store.

In conjunction with RF circuitry 108, touch screen 112, display system controller 156, touch module 130, graphics module 132, handwriting input module 157, and text input module 134, browser module 147 may be used to view web pages or portions thereof, and Contains executable instructions for browsing the Internet according to user instructions, including retrieving, linking, receiving, and displaying attachments and other files linked to web pages.

Calendar module 148 is associated with RF circuitry 108 , touch screen 112 , display system controller 156 , touch module 130 , graphics module 132 , handwriting input module 157 , text input module 134 , email client module 140 and browser module 147 . contains executable instructions for creating, displaying, modifying, and storing calendars and data associated with calendars (eg, calendar items, to-do lists, etc.) according to user instructions.

In conjunction with RF circuitry 108, touch screen 112, display system controller 156, touch module 130, graphics module 132, handwriting input module 157, text input module 134, and browser module 147, widget module 149 is downloaded by the user. , mini-applications (eg, weather widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) that can be used, or created by the user. It is a mini-application (eg, user-created widget 149-6) that can In some embodiments, widgets include HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, widgets include XML (extensible markup language) files and JavaScript files (eg, Yahoo!® widgets).

In conjunction with RF circuitry 108, touch screen 112, display system controller 156, touch module 130, graphics module 132, handwriting input module 157, text input module 134, and browser module 147, widget creation module 150 creates widgets. (eg, turn user-specified portions of a web page into widgets).

In conjunction with touch screen 112 , display system controller 156 , contact module 130 , graphics module 132 , handwriting input module 157 , and text input module 134 , search module 151 performs one or more search criteria (e.g., , one or more user-specified search criteria) to search for text, music, audio, images, video, and/or other files in memory 102 .

In conjunction with the touch screen 112, display system controller 156, touch module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, the video and music player module 152 supports MP3 or AAC Executable instructions that allow users to download and play prerecorded music and other audio files stored in one or more file formats, such as files, and videos (e.g., touch screen 112 and executable instructions for displaying, presenting, or otherwise playing on the computer or on a display connected externally via the external port 124). In some embodiments, device 100 may include MP3 player functionality such as an iPod (trademark of Apple Inc.).

In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, notes module 153 creates and displays notes, to-do lists, etc., according to user instructions. Contains executable instructions for managing.

In conjunction with RF circuitry 108, touch screen 112, display system controller 156, touch module 130, graphics module 132, handwriting input module 157, text input module 134, GPS module 135, and browser module 147, map module 154 includes: Receive maps and data associated with maps (e.g., data about driving directions, stores and other points of interest at or near a particular location, and other location-based data) at the user's direction; , can be used to display, modify and store.

Touch screen 112 , display system controller 156 , touch module 130 , graphics module 132 , audio circuitry 110 , speaker 111 , RF circuitry 108 , handwriting input module 157 , text input module 134 , email client module 140 , and browser module 147 . In conjunction with , the online video module 155 enables a user to access, view, receive (eg, by streaming and/or downloading), and receive (eg, by streaming and/or downloading), and (eg, on a touch screen or through the external port 124) a particular online video. (on an external display connected via H.265), send an e-mail with a link to the online video, and in other ways. It contains instructions that allow for managing online video in one or more file formats, such as H.264. In some embodiments, rather than email client module 140, instant messaging module 141 is used to send links to specific online videos.

Each of the modules and applications identified above may perform one or more of the functions described above and the methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). ), which corresponds to a set of executable instructions. These modules (i.e., instruction sets) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or otherwise implemented in various embodiments. can be reconfigured in the following manner. In some embodiments, memory 102 may store a subset of the modules and data structures identified above. Additionally, memory 102 may store additional modules and data structures not described above.

In some embodiments, device 100 is a device in which operations of a predefined set of functions on the device are performed exclusively through a touchscreen and/or touchpad. By using a touchscreen and/or touchpad as the primary input control device for operation of device 100, the number of physical input control devices (push buttons, dials, etc.) on device 100 can be reduced.

FIG. 2 illustrates portable multifunction device 100 having touch screen 112, according to some embodiments. This touch screen can display one or more graphics in a user interface (UI) 200 . In this embodiment, as well as other embodiments described below, a user may, for example, use one or more fingers 202 (not drawn to scale in the figures) or one or more styluses 203 (as shown). not drawn to scale) can be used to select one or more of the graphics by performing a gesture on the graphic. In some embodiments, selection of one or more graphics occurs when the user breaks contact with one or more graphics. In some embodiments, this gesture is one or more taps, one or more swipes (left to right, right to left, up and/or down), and/or contact with device 100 ( right-to-left, left-to-right, up, and/or down) finger rolling. In some embodiments, accidental contact with a graphic may not select that graphic. For example, if the gesture corresponding to selection is a tap, a swipe gesture sweeping over an application icon may not select the corresponding application.

Device 100 may also include one or more physical buttons, such as “home” or menu button 204 . As previously mentioned, the menu button 204 may be used to navigate to any application 136 within the set of applications capable of running on the device 100 . Alternatively, in some embodiments, menu buttons are implemented as softkeys within a GUI displayed on touch screen 112 .

In one embodiment, the device 100 includes a touch screen 112, a menu button 204, push buttons 206 for powering the device on/off and locking the device, volume control buttons 208, and a subscriber identity module (SIM) card slot. 210, a headset jack 212, and an external port 124 for docking/charging. The push button 206 turns power on and off on the device by depressing the button and holding the button depressed for a predetermined time interval, depressing the button and turning off the power on the device before the predetermined time interval elapses. It can be used to lock the device and/or unlock the device or initiate the unlocking process by releasing the button on. In alternative embodiments, device 100 may also accept verbal input via microphone 113 regarding the activation or deactivation of some functions.

FIG. 3 is a block diagram of an exemplary multifunction device with a display and touch-sensitive surface, according to some embodiments; Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, telephone device, or control device ( For example, home or commercial controllers). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more processors for interconnecting these components. Includes communication bus 320 . Communication bus 320 may include circuitry (sometimes called a chipset) that interconnects and controls communications between system components. Device 300 includes input/output (I/O) interface 330 with display 340, which is typically a touch screen display. I/O interface 330 may also include keyboard and/or mouse (or other pointing device) 350 and touch pad 355 . Memory 370 includes high speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access semiconductor memory devices, and may also be one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other devices. may include non-volatile memory, such as non-volatile semiconductor storage devices. Memory 370 may optionally include one or more storage devices located remotely from CPU 310 . In some embodiments, memory 370 stores programs, modules and data structures similar to, or similar to, programs, modules and data structures stored in memory 102 of portable multifunction device 100 (FIG. 1). Store the subset. Additionally, memory 370 may store additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100 . For example, memory 370 of device 300 may store drawing module 380, presentation module 382, word processing module 384, website creation module 386, disc authoring module 388, and/or spreadsheet module 390, although only one of So, memory 102 of portable multifunction device 100 (FIG. 1) may not store these modules.

Each of the above-identified elements of FIG. 3 can be stored in one or more of the aforementioned memory devices. Each of the modules identified above correspond to a set of instructions that perform the functions described above. The modules or programs (i.e., instruction set) identified above need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be used in various embodiments. They can be combined or rearranged in other ways. In some embodiments, memory 370 may store a subset of the modules and data structures identified above. Additionally, memory 370 may store additional modules and data structures not described above.

FIG. 4 is an illustration on a device (eg, device 300 of FIG. 3) with a touch-sensitive surface 451 (eg, tablet or touchpad 355 of FIG. 3) separate from display 450 (eg, touchscreen display 112). a typical user interface. Although many of the examples below are described with reference to input on a touch screen display 112 (where the touch sensitive surface and display are combined), in some embodiments the device 4, detects input on a touch-sensitive surface separate from the display. In some embodiments, this touch-sensitive surface (eg, 451 in FIG. 4) has a major axis (eg, 453 in FIG. 4) that corresponds to a major axis (eg, 453 in FIG. 4) on the display (eg, 450). 452). According to these embodiments, the device provides touch-sensitive surface 451 and touch-sensitive surface 451 at corresponding locations on the display (e.g., 460 corresponds to 468 and 462 corresponds to 470 in FIG. 4). (eg, 460 and 462 in FIG. 4). In this manner, if the touch-sensitive surface is separate from the display, user input detected by the device on the touch-sensitive surface (eg, 451 in FIG. 4) (eg, contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the multifunction device's display (eg, 450 in FIG. 4). It should be appreciated that similar methods can be used with other user interfaces described herein.

Attention is now directed to embodiments of handwriting input methods and user interfaces (“UIs”) that can be implemented on a multifunction device (eg, device 100).

FIG. 5 illustrates, in interaction with an I/O interface module 500 (eg, I/O interface 330 in FIG. 3 or I/O subsystem 106 in FIG. 1), an on-device device, according to some implementations. 15 is a block diagram showing an exemplary handwriting input module 157 that provides handwriting input capabilities to the . As shown in FIG. 5 , handwriting input module 157 includes input processing module 502 , handwriting recognition module 504 , and result generation module 506 . In some embodiments, input processing module 502 includes segmentation module 508 and normalization module 510 . In some embodiments, result generation module 506 includes radical clustering module 512 and one or more language models 514 .

In some embodiments, input processing module 502 communicates with I/O interface module 500 (eg, I/O interface 330 in FIG. 3 or I/O subsystem 106 in FIG. 1) to provide user receive handwriting input from Handwriting is entered via any suitable means, such as touch-sensitive display system 112 in FIG. 1 and/or touchpad 355 in FIG. Those handwritings contain data representing each stroke provided by the user within a predefined handwriting area within the handwriting UI. In some embodiments, the data representing each stroke of handwriting input includes the start and end of sustained contact (e.g., contact of a user's finger or stylus with a touch-sensitive surface of a device) within the handwriting input area. Includes data such as location, intensity profile, and path of motion. In some embodiments, the I/O interface module 500 passes an array of handwritten strokes 516 with associated temporal and spatial information to the input processing module 502 in real time. At the same time, the I/O interface module also provides a real-time rendering 518 of those handwriting strokes within the handwriting input area of the handwriting input user interface as visual feedback for this user's input.

In some embodiments, as data representing each handwritten stroke is received by the input processing module 502, temporal and spatial information associated with multiple consecutive strokes is also recorded. For example, those data optionally include the shape, size, spatial saturation of individual strokes with their respective stroke sequence numbers, and their relative values along the writing direction of the overall handwriting input. It contains a stack, which indicates the spatial position of the object, etc. In some embodiments, the input processing module 502 renders the received strokes on the device's display 518 (eg, display 340 in FIG. 3 or touch-sensitive display 112 in FIG. 1). Returns the command to the /O interface module 500. In some embodiments, the rendering of received strokes is animated to mimic the actual progress of writing on a writing surface (e.g., a sheet of paper) using a writing instrument (e.g., pen). , visual effects are provided. In some embodiments, the user is optionally allowed to specify the nib style, color, texture, etc. of the rendered strokes.

」は、２つの部首、すなわち、左の部首「

」及び右の部首「

」を使用して構築されている。 In some embodiments, the input processing module 502 processes strokes currently accumulated within the handwriting input area and assigns them to one or more recognition units. In some embodiments, each recognition unit corresponds to a particular character recognized by handwriting recognition model 504 . In some embodiments, each recognition unit corresponds to a particular output character or radical recognized by handwriting recognition model 504 . Radicals are reproducible components found within multiple compound logograms. A compound logogram may include two or more radicals arranged according to a common layout (eg, side-to-side layout, top-to-bottom layout, etc.). In one example, a single Chinese character "

” has two radicals, namely the left radical “

” and the right radical “

is built using .

」に関するストロークを分割する場合、分割モジュール５０８は、任意選択的に、手書き入力の左側にクラスタ化されるストロークを、１つの（すなわち、左の部首「

」に関する）認識単位に割り当て、手書き入力の右側にクラスタ化されるストロークを、別の（すなわち、右の部首「

」に関する）認識単位に割り当てる。あるいは、分割モジュール５０８はまた、これらのストロークの全てを、単一の（すなわち、文字「

」に関する）認識単位に割り当てることもできる。 In some embodiments, the input processing module 502 relies on a segmentation module to assign or separate currently accumulated handwriting strokes into one or more units of recognition. For example, the handwritten character "

, the splitting module 508 optionally divides the strokes clustered on the left side of the handwritten input into one (i.e., the left radical "

) to recognize strokes that are clustered on the right side of the handwritten input to another (i.e., the right radical "

) to recognition units. Alternatively, the segmentation module 508 also divides all of these strokes into a single (i.e., the character "

) can also be assigned to recognition units.

In some embodiments, segmentation module 508 segments the currently accumulated handwritten input (eg, one or more handwritten strokes) into groups of recognition units in several different ways, resulting in segmented bundles 520 . produce. For example, assume that a total of nine strokes have so far accumulated in the handwriting input area. Strokes 1, 2, 3 are grouped into a first recognition unit 522, and strokes 4, 5, 6 are grouped into a second recognition unit 526, according to the first split chain of the split bundle 520. be. According to the second split chain of split bundle 520 , all strokes 1-9 are grouped into one recognition unit 526 .

In some embodiments, each split chain is given a split score to assess the likelihood that the particular split chain is the correct split for the current handwritten input. In some embodiments, factors optionally used to calculate the split score for each split chain include the absolute and/or relative size of the stroke, various directions (e.g., x-direction, y-direction, z-direction), mean and/or variation of stroke saturation level, absolute and/or relative distance to adjacent strokes, stroke absolute and/or or the relative location, the order or arrangement in which the strokes are entered, the duration of each stroke, the average and/or variation in the velocity (or slowness) at which each stroke was entered, the intensity of each stroke along its length. profile and the like. In some embodiments, one or more functions or transformations are optionally applied to one or more of these factors to generate splitting scores for various splitting strands within splitting bundle 520. be.

In some embodiments, after the segmentation module 508 segments the current handwriting input 516 received from the user, the segmentation module 508 passes the segmentation bundle 520 to the normalization module 510 . In some embodiments, normalization module 510 generates an input image (eg, input image 528) for each recognition unit (eg, recognition units 522, 524, and 526) specified in segment bundle 520. FIG. In some embodiments, this normalization module applies any required or desired normalization (e.g., stretching, cropping, etc.) to the input image so that it can be provided as input to handwriting recognition model 504. , downsampling, or upsampling). In some embodiments, each input image 528 includes strokes assigned to a respective recognition unit and corresponds to a single character or radical recognized by handwriting recognition module 504 .

In some embodiments, the input image produced by the input processing module 502 does not contain any temporal information associated with individual strokes, but rather spatial information (e.g., location and density of pixels in the input image). ) is preserved in the input image. A handwriting recognition model trained purely on the spatial information of training writing samples is capable of handwriting recognition based only on spatial information. As a result, the handwriting recognition model is stroke order and stroke direction independent, and during training stroke order and stroke direction for all characters in the vocabulary of the handwriting recognition model (i.e., all output classes). It is not an exhaustive list of all possible permutations of directions. Indeed, in some embodiments, the handwriting recognition module 502 does not distinguish between pixels belonging to one stroke and pixels belonging to another stroke in the input image.

As will be described in more detail below (eg, in connection with FIGS. 25A-27), in some embodiments some temporally derived stroke distribution information is purely spatial handwriting. Reintroduced into the recognition model, recognition accuracy is improved without compromising the stroke order and stroke direction independence of the recognition model.

In some embodiments, the input image generated by the input processing module 502 for one recognition unit does not overlap the input image for any other recognition unit within the same split chain. In some embodiments, input images generated for different recognition units may have some degree of overlap. In some embodiments, some degree of overlap between input images includes handwritten input written in a cursive typeface, and/or contiguous characters (e.g., one stroke connects two adjacent characters). Acceptable for recognizing handwritten input.

In some embodiments, some degree of normalization is performed prior to splitting. In some embodiments, the functions of segmentation module 508 and normalization module 510 may be performed by the same module or by two or more other modules.

In some embodiments, when the input image 528 for each recognition unit is provided as input to the handwriting recognition model 504, the handwriting recognition model 504 determines that the recognition unit corresponds to the repertoire or vocabulary of the handwriting recognition model 504 (i.e., Generate an output consisting of different likelihoods of being each output character in the list of all characters and radicals recognizable by the handwriting recognition module 504 . As will be described in more detail below, the handwriting recognition model 504 is trained to recognize multiple characters in multiple scripts (eg, at least three non-overlapping scripts encoded by the Unicode standard). It is Examples of non-overlapping scripts include Latin scripts, Chinese scripts, Arabic scripts, Persian scripts, Cyrillic scripts, and artificial scripts such as glyphs. In some embodiments, the handwriting recognition model 504 generates one or more output characters for each input image (i.e., for each recognition unit), and based on the confidence level associated with that character recognition, each output character assign a recognition score for each.

In some embodiments, the handwriting recognition model 504 generates candidate bundles 530 according to the split bundle 520, corresponding to each arc in the split chain within the split bundle 520 (eg, each recognition unit 522, 524, 526). is expanded into one or more candidate arcs (eg, arcs 532, 534, 536, 538, 540, each corresponding to a respective output character) in candidate bundle 530. FIG. Each candidate strand in candidate bundle 530 is scored according to the respective split score of the candidate strand's underlying split strand and the recognition score associated with the output character in that candidate strand.

In some embodiments, after the handwriting recognition model 504 has generated output characters from the input image 528 of the recognition unit, the candidate bundle 530 is passed to the result generation module 506 and is associated with the currently accumulated handwriting input 516 . One or more recognition results are generated.

In some embodiments, the result generation module 506 utilizes a radical clustering module 512 to combine one or more radicals in the candidate chain into compound characters. In some embodiments, result generation module 506 uses one or more language models 514 to ensure that the character chains in candidate bundle 530 are aligned in the proper arrangement for the particular language represented by the language model. Determine whether or not there is In some embodiments, result generation module 506 generates corrected candidate bundle 542 by deleting certain arcs or combining two or more arcs in candidate bundle 530 .

In some embodiments, result generation module 506 combines the joint recognition score for each character sequence (eg, character sequences 544 and 546) still maintained in correction candidate bundle 542 with radical clustering module 512 and language. Generate based on the recognition scores of the output characters in those character sequences as modified (eg, increased or decreased) by model 514 . In some embodiments, result generation module 506 ranks the various character sequences maintained in candidate correction bundle 542 based on their combined recognition scores.

」及び「

」）を、手書き入力インタフェースの候補表示区域内に表示する。一部の実施形態では、Ｉ／Ｏインタフェースモジュールは、複数の認識結果（例えば、「

」及び「

」）をユーザに対して表示し、ユーザが、認識結果を選択し、関連するアプリケーションに関するテキスト入力として入力することを可能にする。一部の実施形態では、Ｉ／Ｏインタフェースモジュールは、最上ランクの認識結果（例えば、「

」）を、その認識結果のユーザ確認の、他の入力又は指示に応じて、自動的に入力する。最上ランクの結果の有効な自動入力により、入力インタフェースの効率を改善し、より良好なユーザエクスペリエンスを提供することができる。 In some embodiments, the result generation module 506 sends the highest ranked character sequences as ranked recognition results 548 to the I/O interface module 500 for display to the user. In some embodiments, the I/O interface module 500 receives recognition results 548 (e.g., "

"as well as"

”) is displayed in the candidate display area of the handwriting input interface. In some embodiments, the I/O interface module uses multiple recognition results (e.g., "

"as well as"

) is displayed to the user and allows the user to select a recognition result and enter it as a text input for the associated application. In some embodiments, the I/O interface module selects the highest ranking recognition result (e.g., "

”) are automatically input in response to other inputs or instructions for user confirmation of the recognition result. Efficient auto-filling of the highest ranking results can improve the efficiency of the input interface and provide a better user experience.

In some embodiments, the result generation module 506 uses other factors to modify the candidate strand's combined recognition score. For example, in some embodiments, result generation module 506 optionally maintains a log of the most frequently used characters for a particular user or multiple users. The result generation module 506 calculates the joint recognition score for a particular candidate character or character sequence if the particular candidate character or character sequence is found in the list of most frequently used characters or character sequences; Optionally raise.

In some embodiments, handwriting input module 157 provides real-time updates regarding recognition results displayed to the user. For example, in some embodiments, for each additional stroke by the user, the input processing module 502 optionally subdivides the currently accumulated handwriting input and divides the segmented bundles provided to the handwriting recognition model 504 . and correct the input image. Similarly, handwriting recognition model 504 optionally corrects the candidate bundle provided to result generation module 506 . As a result, result generation module 506 optionally updates the recognition results presented to the user. As used herein, real-time handwriting recognition refers to handwriting recognition in which handwriting recognition results are presented to the user immediately or within a short period of time (eg, within tens of milliseconds to seconds). Real-time handwriting recognition starts from recorded images that are saved for later retrieval, rather than all at once after the current user session, where recognition begins immediately and occurs substantially simultaneously with the receipt of handwritten input It differs from offline recognition (eg, as in offline optical character recognition (OCR) applications) in that the Furthermore, since offline character recognition is performed without using any temporal information about individual strokes and sequences of strokes, segmentation is performed without the benefit of such information. Further disambiguation between similar looking candidate characters also does not benefit from such temporal information.

In some embodiments, handwriting recognition model 504 is implemented as a convolutional neural network (CNN). FIG. 6 shows an exemplary convolutional neural network 602 trained on a multi-script training corpus 604 containing writing samples for characters in multiple non-overlapping scripts.

As shown in FIG. 6, convolutional neural network 602 includes input surface 606 and output surface 608 . Between the input surface 606 and the output surface 608 are multiple convolutional layers 610 (including, for example, an initial convolutional layer 610a, zero or more intermediate convolutional layers (not shown), and a final convolutional layer 610n). Each convolutional layer 610 is followed by a respective subsampling layer 612 (eg, an initial subsampling layer 612a, zero or more intermediate subsampling layers (not shown), and a final subsampling layer 612n). After these convolutional and subsampling layers, just before the output surface 608 is a hidden layer 614 . Hidden layer 614 is the final layer before output face 608 . In some embodiments, a kernel layer 616 (eg, including an initial kernel layer 616a, zero or more intermediate kernel layers (not shown), and a final kernel layer 612n) is inserted before each convolutional layer 610. which improves computational efficiency.

As shown in FIG. 6, input surface 606 receives an input image 614 of handwritten recognition units (e.g., handwritten characters or radicals), and output surface 608 displays the recognition units in corresponding output classes ( For example, output a set of probabilities indicating the likelihood of belonging to a particular character in the set of output characters that the neural network is configured to recognize. The entire neural network output class (or set of neural network output characters) is also referred to as the repertoire or vocabulary of the handwriting recognition model. The convolutional neural networks described herein can be trained to have repertoires of tens of thousands of characters.

As the input image 614 is processed through various layers of the neural network, various spatial features embedded within the input image 614 are extracted by the convolutional layers 610 . Each convolutional layer 610, also referred to as a set of feature maps, acts as a filter to find specific features in the input image 614 to distinguish images corresponding to different characters. The subsampling layer 612 ensures that features are captured at increasing scales from the input image 614 . In some embodiments, subsampling layer 612 is implemented using a maximum pooling technique. This max pooling layer, which produces position invariance over a larger local region, downsamples the output image of the previous convolutional layer by a factor of Kx and Ky along each direction (Kx and Ky is the size of the max pooling rectangle). Max pooling results in faster convergence speed and improves generalization ability by choosing top invariant features. In some embodiments, subsampling is accomplished using other methods.

In some embodiments, after the convolutional layers 610n and the final set of subsamplings 612n, there is a fully connected or hidden layer 614 before the output surface 608. FIG. A fully connected hidden layer 614 is a multilayer perceptron that fully connects the nodes in the final subsampling layer 612 n and the nodes in the output surface 608 . Hidden layer 614 takes the output image received from the previous layer and through logistic regression arrives at one of the output characters in output layer 608 .

During training of the convolutional neural network 602, the features in the convolutional layer 610 and their respective weights associated with those features and the weights associated with the parameters in the hidden layer 614 are extracted from the known output classes in the training corpus 604. is adjusted so that the classification error is minimized for written samples with Once the convolutional neural network 602 has been trained and an optimal set of parameters and associated weights established for the various layers within the network, the convolutional neural network 602 can be used to measure real-time handwritten input received from a user. New written samples 618 that are not part of the training corpus 604, such as input images generated based on , can be recognized.

As described herein, the convolutional neural network of the handwriting input interface is trained using a multi-script training corpus to enable multi-script or mixed-script handwriting recognition. In some embodiments, a convolutional neural network is trained to recognize a wide repertoire of 30,000 to over 60,000 characters (eg, all characters encoded by the Unicode standard). Most state-of-the-art handwriting recognition systems are based on stroke-order dependent Hidden Markov Methods (HMM). Furthermore, most existing handwriting recognition models are language-specific, and from a small repertoire of a few dozen characters (e.g., characters from the English alphabet, the Greek alphabet, all ten digits, etc.), Contains up to thousands of characters (eg the set of most commonly used Chinese characters). Therefore, the universal recognizer described herein can handle several orders of magnitude more characters than most existing systems.

Some conventional handwriting systems may include several individually trained handwriting recognition models, each adapted for a particular language or subset of characters. Writing samples are propagated through different recognition models until they can be classified. For example, handwriting samples can be provided to a series of concatenated language-specific or script-specific character recognition models, and if the handwriting samples cannot be conclusively classified by the first recognition model, then , and this next recognition model attempts to classify that handwriting sample within its own repertoire. This approach to classification is time consuming and the memory requirements grow rapidly with each additional recognition model that needs to be employed.

Other state-of-the-art models require the user to specify a preferred language and use the selected handwriting recognition model to classify the current input. Such implementations are cumbersome to use, consume significant memory, and cannot be used to recognize mixed-language input. It is not practical to require the user to switch language settings in the middle of entering mixed languages or scripts.

The multiscript or universal recognizer described herein addresses at least some of the above problems with conventional recognition systems. FIG. 7 illustrates using a large multiscript training corpus to train a handwriting recognition module (e.g., a convolutional neural network) to transform the handwriting recognition module into real-time multilingual and multiscript handwriting on a user's handwriting input. 7 is a flow diagram of an exemplary process 700 for subsequent use to provide recognition.

In some embodiments, training of the handwriting recognition model is performed on the server device and then the trained handwriting recognition model is provided to the user device. This handwriting recognition model optionally performs real-time handwriting recognition locally on the user device without requiring further assistance from a server. In some embodiments, both training and recognition are provided on the same device. For example, a server device can receive a user's handwriting input from a user device, perform handwriting recognition, and send the recognition results to the user device in real time.

In exemplary process 700, in a device having one or more processors and memory, the device generates multiscript handwriting based on spatially derived features (e.g., stroke order-independent features) of a multiscript training corpus. A recognition model is trained (702). In some embodiments, the spatially derived features of the multiscript training corpus are stroke order independent and stroke direction independent (704). In some embodiments, training the multiscript handwriting recognition model is independent of the temporal information associated with each stroke in the handwriting sample (706). Specifically, the images of the handwriting samples are normalized to a predetermined size, and they do not contain any information about the order in which individual strokes are input to form the image. Furthermore, the images also do not contain any information about the direction in which the individual strokes were input to form the image. In fact, during training, features are extracted from these handwritten images regardless of how the images are temporally shaped by individual strokes. Therefore, no temporal information relating to individual strokes is required during recognition. As a result, the recognition robustly provides consistent recognition results despite delayed out-of-order strokes and arbitrary stroke directions within handwritten input.

In some embodiments, the multiscript training corpus includes handwriting samples corresponding to characters of at least three non-overlapping scripts. As shown in FIG. 6, the multiscript training corpus contains handwriting samples collected from many users. Each handwriting sample corresponds to one character of the respective script as represented in the handwriting recognition model. In order to properly train a handwriting recognition model, the training corpus contains a large number of written samples for each character of the script represented in the handwriting recognition model.

In some embodiments, the at least three non-overlapping scripts include Chinese characters, glyphs, and Latin script (708). In some embodiments, the multi-script handwriting recognition model has at least 30,000 output classes representing 30,000 characters across at least 3 non-overlapping scripts (710).

In some embodiments, the multiscript training corpus includes all or substantially all Chinese characters encoded in the Unicode standard (eg, all CJK (Chinese-Japanese-Korean) integrated Chinese characters) part) for each character. The Unicode standard defines a total of approximately 74,000 CJK integrated Chinese characters. The CJK Unified Hanzi Basic Block (4E00-9FFF) contains 20,941 basic Chinese characters used in Chinese, as well as Japanese, Korean, and Vietnamese. In some embodiments, the multiscript training corpus contains writing samples for all characters in the basic block of CJK integrated Chinese characters. In some embodiments, the multiscript training corpus further includes written samples for CJK radicals that can be used to construct one or more compound Chinese characters. In some embodiments, the multiscript training corpus further includes written samples for less frequently used Chinese characters, such as Chinese characters encoded in one or more of the CJK Unified Hanzi extensions.

In some embodiments, the multiscript training corpus further includes respective written samples for each character of every character in the Latin script encoded by the Unicode standard. Characters in the basic Latin script include Latin uppercase and lowercase letters, as well as the basic symbols and digits commonly used on standard Latin keyboards. In some embodiments, the multiscript training corpus further includes characters in extended Latin scripts (eg, various stressed forms of basic Latin characters).

」に関する手書きサンプルである。同様に、手描きの笑顔（例えば、上を向いた弧の上方の２つの点）は、訓練コーパス内の絵文字「

」に関する手書きサンプルである。他の絵文字としては、種々の感情（例えば、歓喜、悲哀、怒り、当惑、驚愕、笑い、泣き、苛立ちなど）、種々の物体及びキャラクタ（例えば、猫、犬、ウサギ、心臓、果物、目、唇、贈り物、花、ろうそく、月、星など）、及び種々の行為（例えば、握手、キス、走る、踊る、跳び上がる、眠る、食べる、会う、愛する、好む、投票するなど）などを示すアイコンのカテゴリが挙げられる。一部の実施形態では、絵文字に対応する手書きサンプル内のストロークは、対応する絵文字を形成する実際の線の、単純化及び／又は様式化された線である。一部の実施形態では、各デバイス又はアプリケーションは、同じ絵文字に関して、異なるデザインを使用することができる。例えば、女性のユーザに提示される笑顔の絵文字は、男性のユーザに提示される笑顔の絵文字とは、それらの２人のユーザから受信される手書き入力が実質的に同じ場合であっても、異なるものとすることができる。 In some embodiments, the multiscript training corpus includes written samples corresponding to each character of an artificial script not associated with any natural human language. For example, in some embodiments, a set of glyphs is optionally defined in the glyph script, and writing samples corresponding to each of those glyphs are included in the multiscript training corpus. For example, the hand-drawn heart symbol is the glyph "

is a handwritten sample about Similarly, a hand-drawn smiley face (e.g., two points above an arc pointing up) is the glyph "

is a handwritten sample about Other glyphs include various emotions (e.g., joy, sadness, anger, embarrassment, surprise, laughter, cry, irritation, etc.), various objects and characters (e.g., cats, dogs, rabbits, hearts, fruits, eyes, etc.). lips, gifts, flowers, candles, moon, stars, etc.) and various actions (e.g. handshake, kiss, run, dance, jump, sleep, eat, meet, love, like, vote, etc.) category. In some embodiments, the strokes in the handwriting sample corresponding to the glyph are simplified and/or stylized lines of the actual lines forming the corresponding glyph. In some embodiments, each device or application may use different designs for the same emoji. For example, a smiley emoji presented to a female user may not be the same as a smiley emoji presented to a male user, even if the handwriting input received from those two users is substantially the same. can be different.

In some embodiments, the multiscript training corpus also includes Greek script (e.g., including Greek letters and symbols), Cyrillic script, Hebrew script, and one or more others encoded according to the Unicode standard. Also includes writing samples for characters in other scripts, such as the script of In some embodiments, the at least three non-overlapping scripts included in the multiscript training corpus include Chinese characters, glyphs, and characters in Latin script. Chinese characters, glyphs, and characters in Latin scripts are inherently non-overlapping scripts. Many other scripts may overlap each other, at least on some characters. For example, some letters in Latin script (eg, A, Z) can be found in many other scripts (eg, Greek and Cyrillic). In some embodiments, the multiscript training corpus includes Chinese script, Arabic script, and Latin script. In some embodiments, the multi-script training corpus includes other combinations of overlapping scripts and/or non-overlapping scripts. In some embodiments, the multiscript training corpus contains writing samples for all characters encoded by the Unicode standard.

As shown in FIG. 7, in some embodiments, to train a multi-script handwriting recognition model, the device trains a single convolutional neural network with a single input surface and a single output surface. , provides handwriting samples of the multiscript training corpus (712). the device comprises spatially derived features (e.g., stroke order independent formula features) of those handwriting samples for distinguishing characters of at least three non-overlapping scripts represented in the multiscript training corpus; and their respective weights for their spatially derived features are determined using a convolutional neural network (714). This multi-script handwriting recognition model has a single input surface and a single output surface, in that a single handwriting recognition model is trained using all the samples in the multi-script training corpus, It differs from traditional multi-script handwriting recognition models. A single convolutional neural network is divided into separate sub-networks (e.g., sub-networks each trained on a particular script or character used within a particular language), each handling a small subset of the training corpus. It is trained to identify all characters represented in the multiscript training corpus without dependence. Furthermore, this single convolutional neural network can be used for multiple non-overlapping scripts, rather than a small number of overlapping scripts such as Latin and Greek scripts (e.g., with overlapping letters A, B, E, Z, etc.). trained to identify a large number of characters over

In some embodiments, the device provides real-time handwriting recognition for user handwriting input using a multiscript handwriting recognition model that has been trained on spatially derived features of a multiscript training corpus ( 716). In some embodiments, providing real-time handwriting recognition for a user's handwriting input continues recognition output for the user's handwriting input as the user continues to provide additions and corrections to the user's handwriting input. including correcting In some embodiments, providing real-time handwriting recognition for the user's handwriting input further includes providing a multi-script handwriting recognition model to the user device (718), wherein the user device receives handwriting input from the user. Receive and locally perform handwriting recognition on the handwritten input based on the multi-script handwriting recognition model.

In some embodiments, the device provides a multi-script handwriting recognition model to multiple devices where there is no overlap in their respective input languages, and the multi-script handwriting recognition model is associated with each of those user devices. Each of these multiple devices is used for handwriting recognition in different languages. For example, if a multi-script handwriting recognition model is trained to recognize characters in many different scripts and languages, then that same handwriting recognition model can be used worldwide to recognize any of those input languages. can provide handwriting input for A first device for users who only want to type in English and Hebrew uses the same handwriting recognition model as a second device for users who only want to type in Chinese and pictograms. functionality can be provided. If the user of the first device uses an English handwriting keyboard (e.g., implemented with an English-specific handwriting recognition model) and a separate Hebrew handwriting keyboard (e.g., implemented with a Hebrew-specific handwriting recognition model) ), the same generic multi-script handwriting recognition model, once installed on the first device, provides handwriting input capabilities for both English and Hebrew, as well as both can be used to provide mixed input in any language. In addition, the second user may use a Chinese handwriting input keyboard (e.g., implemented with a Chinese-specific handwriting recognition model) and a separate pictographic handwriting input keyboard (e.g., implemented with a glyph handwriting recognition model). ), the same generic multi-script handwriting recognition model, once installed on the second device, provides handwriting input capabilities for both Chinese and pictograms, as well as in both scripts. can be used to provide a mixed input of By using the same multiscript handwriting model to handle a broad repertoire across multiple scripts (e.g., a substantial portion or all of the characters encoded in nearly 100 different scripts), device suppliers and users The utility of the recognizer is increased without imposing a substantial burden on the

Training a multi-script handwriting recognition model using a large multi-script training corpus differs from conventional HMM-based handwriting recognition systems in that it does not rely on temporal information associated with individual strokes of characters. . Moreover, the resource and memory requirements for this multi-script recognition system do not scale with the increase in symbols and languages covered by the multi-script recognition system. For example, in a conventional handwriting system, increasing the number of languages means adding another independently trained model, and memory requirements increase to accommodate the increasing power of the handwriting recognition system. will be at least doubled. In contrast, when a multiscript model is trained with a multiscript training corpus, increasing language coverage requires retraining the handwriting recognition model using additional handwriting samples. , increases the size of the output face, but the amount of increase is very modest. The multiscript training corpus contains handwriting samples corresponding to n different languages, and the multiscript handwriting recognition model occupies a memory of size m, increasing the language coverage to N (N>n) languages. , the device retrains the multi-script handwriting recognition model based on the spatially derived features of a second multi-script training corpus, which is trained on N different languages contains a second handwriting sample corresponding to . For variations in N/n between 1 and 100, variations in M/m remain substantially constant within the range of 1-2. After the multi-script handwriting recognition model is retrained, the device can use the retrained multi-script handwriting recognition model to provide real-time handwriting recognition for the user's handwriting input.

8A, 8B show exemplary user interfaces for providing real-time multi-script handwriting recognition and input on a portable user device (eg, device 100). 8A and 8B, a handwriting input interface 802 is displayed on a user device's touch-sensitive display screen (eg, touch screen 112). Handwriting input interface 802 includes handwriting input area 804 , candidate display area 806 , and text input area 808 . In some embodiments, the handwriting input interface 802 further includes a plurality of control elements, each of which can be invoked to cause the handwriting input interface to perform predetermined functions. Included within the handwriting input interface, as shown in FIG. 8A, are a delete button, a space button, a carriage return or Enter button, and a keyboard toggle button. Other control elements are possible and can optionally be provided within the handwriting input interface 802 to suit different applications utilizing the handwriting input interface 802 . The layout of the various components of handwriting input interface 802 is merely exemplary and may vary with different devices and different applications.

」）、幾つかの手書き英語文字（例えば、「Ｈａｐｐｙ」）、及び手描きの絵文字（例えば、笑顔）を含む、幾つもの手書きストロークを提供している。これらの手書き文字は、手書き入力区域８０４内の複数の行（例えば、２つの行）内に分布されている。 In some embodiments, handwriting input area 804 is a touch sensitive area for receiving handwriting input from a user. A sustained contact on the touch screen within the handwriting input area 804 and its associated path of movement are registered as handwriting strokes. In some embodiments, the handwriting strokes registered by the device are visually rendered within the handwriting input area 804 at the same locations traced by the sustained contact. As shown in FIG. 8A, the user enters several handwritten Chinese characters (eg, "

”), some handwritten English characters (eg, “Happy”), and hand-drawn glyphs (eg, a smiley face). These handwritten characters are distributed in multiple lines (eg, two lines) within the handwriting input area 804 .

In some embodiments, candidate display area 806 displays one or more recognition results (eg, 810 and 812 ) for handwriting input currently accumulated in handwriting input area 804 . Generally, the highest ranked recognition result (eg, 810) is displayed in the first position within the candidate display area. This is because the handwriting recognition model described herein is capable of recognizing characters from multiple non-overlapping scripts, including Chinese characters, Latin scripts, and glyphs, as shown in FIG. 8A. The recognition results (eg, 810) provided by the recognition model accurately include Chinese characters, English characters, and glyphs represented by the handwritten input. The user is not required to interrupt mid-writing input to select or switch recognition languages.

」）は、メモアプリケーションに既に提供されたテキスト入力である。一部の実施形態では、カーソル８１３が、テキスト入力区域８０８内での、現在のテキスト入力位置を示す。 In some embodiments, text input area 808 is an area that displays text input provided to respective applications employing this handwriting input interface. As shown in FIG. 8A, the text entry area 808 is used by the Notes application to display the text currently shown in the text entry area 808 (eg, "America

') is the text input already provided in the Notes application. In some embodiments, cursor 813 indicates the current text entry position within text entry area 808 .

In some embodiments, the user may, for example, have an explicit selection input (e.g., a tap gesture on one of the displayed recognition results) or an implicit confirmation input (e.g., an "Enter" button A tap gesture on, or a double tap gesture within the handwriting input area) may select a particular recognition result displayed within the candidate display area 806 . As shown in FIG. 8B, the user explicitly taps the top-ranking recognition result 810 using a tap gesture (as indicated by contact 814 on recognition result 810 in FIG. 8A). have selected. In response to this selection input, the text of recognition result 810 is inserted at the insertion point indicated by cursor 813 within text entry area 808 . As shown in FIG. 8B, once the text of the selected recognition result 810 is entered into the text entry area 808, the handwriting entry area 804 and the candidate display area 806 are both cleared. At this point, handwriting input area 804 is ready to accept new handwriting input, and candidate display area 806 can now be used to display recognition results for that new handwriting input. . In some embodiments, the implicit confirmation entry causes the top-ranking recognition result to be entered in the text entry area 808 without requiring the user to pause and select the top-ranking recognition result. Well-designed implicit confirmation input increases text entry speed and reduces the cognitive burden placed on the user during text composition.

In some embodiments (not shown in FIGS. 8A, 8B), the top-ranking recognition result of the current handwriting input is optionally provisionally displayed within the text entry area 808 . The provisional text entry shown within this text entry area 808 is visually distinguished from other text entries within the text entry area by, for example, a provisional entry box surrounding the provisional text entry. The text shown in this provisional entry box has not yet been finalized, nor has it been provided to the associated application (e.g., a memo application), e.g. , is automatically updated when the top-ranking recognition result is changed by the handwriting input module.

9A, 9B are flow diagrams of an exemplary process 900 for providing multi-script handwriting recognition on user devices. In some embodiments, as shown in diagram 900, a user device receives 902 a multi-script handwriting recognition model, which is a spatially derived feature of a multi-script training corpus. (eg, stroke order and stroke direction independent expression features), this multiscript training corpus contains handwriting samples corresponding to characters of at least three non-overlapping scripts. In some embodiments, the multiscript handwriting recognition model is a single convolutional neural network (906) with a single input surface and a single output surface, represented in the multiscript training corpus. Spatially derived features and respective weights for those spatially derived features for distinguishing characters of at least three non-overlapping scripts. In some embodiments, the multi-script handwriting recognition model is configured to recognize characters based on respective input images of one or more recognition units identified in the handwriting input (908); Each spatially derived feature used for is independent of its respective stroke order, stroke direction, and stroke continuity within the handwritten input.

In some embodiments, the user device receives 908 handwritten input from the user, the handwritten input including one or more handwritten strokes provided on a touch-sensitive surface coupled to the user device. . For example, handwriting input includes data about the location and movement of contact between a finger or stylus and a touch-sensitive surface coupled to the user device, respectively. In response to receiving this handwriting input, the user device generates one or more handwriting recognition results in real-time based on a multiscript handwriting recognition model that has been trained on spatially derived features of the multiscript training corpus. (910) and (912).

In some embodiments, when providing real-time handwriting recognition results to a user, the user device divides 914 the user's handwriting input into one or more recognition units, each recognition unit being processed by the user. Contains one or more of the provided handwriting strokes. In some embodiments, the user device segments the user's handwriting input according to the shape, location, and size of individual strokes produced by contact between the user's finger or stylus and the touch-sensitive surface of the user device. do. In some embodiments, this handwriting segmentation further takes into account the relative order and position of individual strokes produced by contact between the user's finger or stylus and the touch-sensitive surface of the user device. In some embodiments, the user's handwritten input is in cursive typeface, and each successive stroke in the handwritten input may correspond to multiple strokes in the recognized characters in printed form. In some embodiments, the user's handwritten input may include continuous strokes spanning multiple recognized characters in printed form. In some embodiments, segmentation of handwriting input produces one or more input images, one for each recognition unit. In some embodiments, some of those input images optionally contain some overlapping pixels. In some embodiments, those input images do not contain any overlapping pixels. In some embodiments, the user device generates a split bundle, each split chain of the split bundle representing a respective way of splitting the current handwritten input. In some embodiments, each arc in the split chain corresponds to a respective group of strokes in the current handwriting input.

」に関する手書き入力は、類似する場合が多い。同様に、絵文字「

」に関する手書き入力は、ＣＪＫ部首「

」に関する手書き入力と類似する場合がある。一部の実施形態では、マルチスクリプト手書き認識モデルは、多くの場合、ユーザの手書き入力に対応する可能性が高い、複数の候補認識結果を生成するが、これは、手書き入力の視覚的外観が、人間の読み手でさえも、解読することが困難となるためである。一部の実施形態では、第１のスクリプトは、ＣＪＫ基本文字ブロックであり、第２のスクリプトは、Ｕｎｉｃｏｄｅ規格によって符号化されているようなラテンスクリプトである。一部の実施形態では、第１のスクリプトは、ＣＪＫ基本文字ブロックであり、第２のスクリプトは、絵文字のセットである。一部の実施形態では、第１のスクリプトは、ラテンスクリプトであり、第２のスクリプトは、絵文字である。 As shown in diagram 900, a user device provides images corresponding to each of one or more recognition units as input to a multi-script recognition model (914). For at least one of the one or more recognition units, the user device outputs at least a first output character from the first script and a second output character different from the first script from the multi-script handwriting recognition model. At least a second output character from the script is obtained (916). For example, the same input image can cause a multi-script recognition model to output two or more similar-looking output characters from different scripts as recognition results for that same input image. For example, handwritten input for the letter "a" in Latin script and the letter "α" in Greek script are often similar. Furthermore, the letter "J" in Latin script and the Chinese letter "

” are often similar. Similarly, the emoji "

The handwritten input for " is the CJK radical "

” may be similar to handwriting input. In some embodiments, the multi-script handwriting recognition model often generates multiple candidate recognition results that likely correspond to the user's handwriting, but this is because the visual appearance of the handwriting is , making it difficult for even a human reader to decipher. In some embodiments, the first script is CJK basic character blocks and the second script is a Latin script as encoded by the Unicode standard. In some embodiments, the first script is a CJK basic character block and the second script is a set of glyphs. In some embodiments, the first script is a Latin script and the second script is a glyph.

」及びギリシャ文字「λ」を特定していると想定すると、ユーザデバイスは、そのユーザデバイス上に、中国語ソフトキーボード（例えば、ピンイン入力法を使用するキーボード）又はギリシャ語入力キーボードのいずれをユーザがインストールしているかを判定する。ユーザデバイスが、中国語ソフトキーボードのみがインストールされていると判定する場合には、そのユーザデバイスは、任意選択的に、ユーザへの認識結果として中国文字「

」のみを表示し、ギリシャ文字「λ」は表示しない。 In some embodiments, the user device displays both the first output character and the second output character in the candidate display area of the handwriting input interface of the user device (918). In some embodiments, the user device determines which one of the first script and the second script is the corresponding script used with the soft keyboard currently installed on the user device. selectively displaying one of the first output character and the second output character based on (920). For example, if a handwriting recognition model selects the Chinese character "

' and the Greek letter 'λ', the user device has either a Chinese soft keyboard (e.g., a keyboard using Pinyin input method) or a Greek input keyboard on its user device. Determine if is installed. If the user device determines that only the Chinese soft keyboard is installed, the user device optionally provides the recognition result to the user as the Chinese character "

, not the Greek letter λ.

In some embodiments, the user device provides real-time handwriting recognition and input. In some embodiments, the user device responds to the user's ongoing additions or corrections to the handwriting input before the user implements an explicit or implicit selection of recognition results displayed to the user. One or more recognition results for the user's handwritten input are continuously corrected (922). In some embodiments, in response to each correction of one or more recognition results, the user displays the respective corrected one or more recognition results to the user within the candidate display area of the handwriting input user interface. (924).

In some embodiments, the multiscript handwriting recognition model is adapted to recognize all characters of at least three non-overlapping scripts, including Chinese characters, glyphs, and Latin script, encoded according to the Unicode standard. It is trained (926). In some embodiments, the at least three non-overlapping scripts include Chinese script, Arabic script, and Latin script. In some embodiments, the multi-script handwriting recognition model has at least 30,000 output classes representing at least 30 characters across at least 3 non-overlapping scripts (928).

ｉｎ Ｃｈｉｎｅｓｅ．」を書き込むことができ、その場合、中国文字「

」を書き込む前に、英語から中国語に入力言語を切り替えることも、又は英語の語「ｉｎ Ｃｈｉｎｅｓｅ」を書き込む際に、中国語から英語に入力言語を切り替えて戻すことも必要とはしない。 In some embodiments, a user device allows a user to enter multi-script handwriting input, such as a phrase that includes characters from more than one script. For example, a user can continuously write and receive handwriting recognition results that include characters from two or more scripts without pausing in the middle of writing to manually switch recognition languages. For example, a user may enter the multiscript sentence "Hello means

in Chinese. " can be written, in which case the Chinese character "

', or switching back from Chinese to English when writing the English word 'in Chinese'.

As described herein, a multi-script handwriting recognition model is used to provide real-time handwriting recognition on user input. In some embodiments, real-time handwriting recognition is used to provide real-time multi-script handwriting input functionality on a user's device. 10A-10C are flow diagrams of an exemplary process 1000 for providing real-time handwriting recognition and input on user devices. Specifically, this real-time handwriting recognition is stroke-order independent at the character level, phrase level, and sentence level.

」に関しては、ユーザが、水平のストロークを最初に書き込むか、又は垂直のストロークを最初に書き込むかにかかわらず、同じ認識結果「

」が、手書き認識モデルによって与えられる。 In some embodiments, stroke-order-independent handwriting recognition at the character level is such that regardless of the arrangement of individual strokes of a particular character provided by the user, the handwriting recognition model It is required to provide the same recognition results. For example, individual strokes of Chinese characters are typically written in a particular order. Native Chinese speakers are often trained in school to write each character in a particular order, but many users deviate from that traditional stroke order for their personal style and style. An array of strokes was later adopted. Furthermore, cursive typefaces are highly individualized, and multiple strokes in the printed form of Chinese characters are often fused into a single stylized stroke, which is not distorted. , turns around, and may even continue to the next letter. Stroke-order-independent recognition models are trained on images of writing samples that do not contain temporal information associated with individual strokes. Therefore, this recognition is independent of stroke order information. For example, the Chinese character "

, the same recognition result "

” is given by the handwriting recognition model.

」に関する手書き入力は、典型的には、実質的に垂直の手書きストロークと交差する、実質的に水平の手書きストロークを含む。 As shown in FIG. 10A, in process 1000, a user device receives 1002 multiple handwritten strokes from a user, and these multiple handwritten strokes correspond to one handwritten character. For example, the character "

The handwritten input for "" typically includes substantially horizontal handwritten strokes intersecting substantially vertical handwritten strokes.

」）を表示する（１００８）。 In some embodiments, the user device generates 1004 an input image based on the plurality of handwritten strokes. In some embodiments, the user device provides 1006 the input image to a handwriting recognition model to perform real-time handwriting recognition of the handwritten characters, which handwriting recognition model uses stroke order independent handwriting. provide recognition. The user device then receives multiple handwritten strokes (e.g., horizontal strokes and vertical strokes) from the user, regardless of their respective order, in real-time to receive the same number of handwritten strokes. 1 output character (e.g., the character "

”) is displayed (1008).

In some conventional handwriting recognition systems, minor stroke order variations in a small number of characters are tolerated by specifically including such variations in the training of the handwriting recognition system. Although such conventional handwriting recognition systems are not scalable to accommodate arbitrary stroke order variations in large numbers of complex characters, such as Chinese characters, this is useful for characters of moderate complexity. Even so, it already causes a large number of variations in stroke order. Furthermore, by simply including more permutations of the acceptable stroke order for a particular character, conventional recognition systems combine multiple strokes into a single stroke (e.g., superscript handwriting input where a single stroke is broken into multiple sub-strokes (such as in characters captured with a very coarse sampling of the input strokes), but still cannot be handled. Impossible. Therefore, a multiscript handwriting system trained on spatially derived features, as described herein, offers advantages over conventional recognition systems.

In some embodiments, stroke order independent handwriting recognition is performed independently of the temporal information associated with individual strokes within each handwritten character. In some embodiments, stroke order-independent handwriting recognition is performed in conjunction with stroke distribution information that takes into account the spatial distribution of individual strokes before being fused into a flat input image. be. Further details on how this temporally derived stroke distribution information is used to enhance the above-described stroke order independent handwriting recognition are provided later (e.g., in FIGS. 25A-27). related) are provided herein. The techniques described in connection with FIGS. 25A-27 do not compromise the stroke order independence of the handwriting recognition system.

」を書き込んでいる場合には、手書き認識モデルは、ユーザが、水平のストロークを左から右に描いたか、又は右から左に描いたかにかかわらず、同じ認識結果を出力する。同様に、手書き認識モデルは、ユーザが、垂直のストロークを下向きの方向で描いたか、又は上向きの方向で描いたかにかかわらず、同じ認識結果を出力する。別の実施例では、多くの中国文字は、２つ以上の部首で構造的に作成される。一部の中国文字は、左の部首及び右の部首をそれぞれが含み、人々は、慣例的に、左の部首を最初に書き込み、右の部首を２番目に書き込む。一部の実施形態では、手書き認識モデルは、ユーザが手書き文字を完成させた際に、結果的に得られる手書き入力が、右の部首の左側に左の部首を示す限りは、ユーザが左の部首を最初に書き込んだか、又は右の部首を最初に書き込んだかにかかわらず、同じ認識結果を提供する。同様に、一部の中国文字は、上の部首及び下の部首をそれぞれが含み、人々は、慣例的に、上の部首を最初に書き込み、下の部首を最後に書き込む。一部の実施形態では、手書き認識モデルは、結果的に得られる手書き入力が、下の部首の上方に上の部首を示す限りは、ユーザが上の部首を最初に書き込んだか、又は下の部首を最初に書き込んだかにかかわらず、同じ認識結果を提供する。換言すれば、手書き認識モデルは、ユーザが提供する、手書き文字の個別のストロークの方向に依存することなく、手書き文字の同一性を判定する。 In some embodiments, the handwriting recognition model provides stroke direction independent handwriting recognition (1010). In some embodiments, stroke-direction-independent recognition enables a user device, in response to receiving a plurality of handwriting inputs, to recognize each of a plurality of handwriting strokes provided by a user, regardless of the respective stroke direction. , to display the same first output character. For example, if a user enters the Chinese character "

, the handwriting recognition model outputs the same recognition results whether the user draws horizontal strokes from left to right or from right to left. Similarly, the handwriting recognition model outputs the same recognition results whether the user draws vertical strokes in a downward or upward direction. In another example, many Chinese characters are structurally made up of two or more radicals. Some Chinese characters each contain a left radical and a right radical, and people conventionally write the left radical first and the right radical second. In some embodiments, the handwriting recognition model assumes that when the user completes the handwriting, as long as the resulting handwriting input shows the left radical to the left of the right radical, the It provides the same recognition results regardless of whether the left radical is written first or the right radical is written first. Similarly, some Chinese characters each include an upper radical and a lower radical, and people conventionally write the upper radical first and the lower radical last. In some embodiments, the handwriting recognition model determines whether the user wrote the top radical first, or It provides the same recognition results regardless of whether the lower radical is written first. In other words, the handwriting recognition model determines the identity of handwritten characters without depending on the direction of the individual strokes of the handwritten characters provided by the user.

」の形状を構成するために、４つのストローク（例えば、十字形状の文字を構成するための、２つの短い水平のストローク及び２つの短い垂直のストローク）を提供したか、又は２つのストローク（例えば、Ｌ形状のストローク及び７形状のストローク、あるいは水平のストローク及び垂直のストローク）を提供したか、又は任意の他の数のストローク（例えば、数百個の極度に短いストローク又は点）を提供したかにかかわらず、同じ認識結果を出力する。 In some embodiments, the handwriting recognition model provides handwriting recognition based on the image of the recognition unit regardless of the number of substrokes in the recognition unit provided by the user. In other words, in some embodiments, the handwriting recognition model provides stroke-count independent handwriting recognition (1014). In some embodiments, the user device, in response to receiving multiple handwritten strokes, outputs the same stroke regardless of how many handwritten strokes are used to form successive strokes in the input image. Display the first output character. For example, if the user is writing the Chinese character "+" in the handwriting input area, the handwriting recognition model will assume that the user writes the character "

” (e.g., two short horizontal strokes and two short vertical strokes to construct a cross-shaped character), or two strokes (e.g., , L-shaped and 7-shaped strokes, or horizontal and vertical strokes), or any other number of strokes (e.g., hundreds of extremely short strokes or points). The same recognition result is output regardless of whether

In some embodiments, the handwriting recognition model is not only capable of recognizing the same characters regardless of the order, direction, and number of strokes in which each single character was written; is also capable of recognizing characters regardless of the temporal order of strokes of the characters provided by the user.

」及び「

」）を書き込んでいる場合には、ユーザデバイスは、手書き入力区域内に現時点で蓄積されている手書き入力が、文字「

」に関するストロークの左に、文字「

」に関するストロークを示す限りは、ユーザが文字「

」のストロークを最初に書き込んだか、又は文字「

」のストロークを最初に書き込んだかにかかわらず、認識結果「

」を表示する。実際に、ユーザが、文字「

」に関するストロークのうちの一部（例えば、垂直のストローク）の前に、文字「

」に関するストロークのうちの一部（例えば、左に傾斜したストローク）を書き込んでいる場合であっても、手書き入力区域内の手書き入力の結果的に得られる画像が、文字「

」に関する全てのストロークの左に、文字「

」に関する全てのストロークを示す限りは、ユーザデバイスは、それらの２つの手書き文字の空間的配列で、認識結果「

」を示す。 In some embodiments, the user device has not only received a first plurality of handwritten strokes from the user, but has also received a second plurality of handwritten strokes (1016), and those second The plurality of handwritten strokes of corresponds to a second handwritten character. In some embodiments, the user device generates 1018 a second input image based on those second plurality of handwritten strokes. In some embodiments, the user device provides 1020 this second input image to a handwriting recognition model to perform real-time recognition of the second handwriting. In some embodiments, the user device displays 1022 a second output character corresponding to the second plurality of handwritten strokes in real-time as it receives the second plurality of handwritten strokes. In some embodiments, the second output character and the first output character are independent of the respective order of the first plurality of handwritten strokes and the second plurality of handwritten strokes provided by the user. displayed simultaneously in a spatial arrangement. For example, if a user enters two Chinese characters (e.g., "

"as well as"

”), the user device indicates that the currently accumulated handwriting input within the handwriting input area is the character “

To the left of the stroke about ", the character "

as long as the user indicates the stroke for the character "

" stroke first, or the character "

" regardless of whether the stroke of " is written first or not, the recognition result "

” is displayed. In fact, if the user

before some of the strokes (e.g. vertical strokes) with respect to the character "

, the resulting image of the handwriting input within the handwriting area is the character "

to the left of all strokes with respect to the character "

”, the user device can display the recognition result “

” is shown.

In other words, as shown in FIG. 10B, in some embodiments, the spatial arrangement of the first output character and the second output character is the default (e.g., left correspond 1024 to the spatial distribution of the first plurality of handwritten strokes and the second plurality of strokes along the writing direction (from right to left). In some embodiments, the second plurality of handwriting strokes is received (1026) later in time than the first plurality of handwriting strokes, and the second output character is the handwriting input interface of the user device: Precedes the first output character in spatial alignment along the default (eg, left-to-right) writing direction.

」が第１の手書き文内に存在し、手書き文字「

」が第２の手書き文内に存在し、それらの２つの手書き文字が、１つ以上の他の手書き文字及び／又は語によって隔てられている場合であっても、手書き認識モデルは、それらの２つの文字を空間的配列「

」で示す認識結果を、依然として提供する。この認識結果、及び２つの認識文字の空間的配列は、ユーザが手書き入力を完了した際に、それらの２つの文字に関する認識単位が、配列「

」で空間的に配置構成されているならば、ユーザによって提供された、それらの２つの文字のストロークの時間的順序にかかわらず、同じままで維持される。一部の実施形態では、第１の手書き文字（例えば、「

」）は、第１の手書き文（例えば、「

ｉｓ ａ ｎｕｍｂｅｒ．」）の一部として、ユーザによって提供され、第２の手書き文字（例えば、「

」）は、第２の手書き文（例えば、「

ｉｓ ａｎｏｔｈｅｒ ｎｕｍｂｅｒ．」）の一部として、ユーザによって提供され、これらの第１の手書き文及び第２の手書き文は、ユーザデバイスの手書き入力区域内に、同時に表示される。一部の実施形態では、この認識結果（例えば、「

ｉｓ ａ ｎｕｍｂｅｒ．

ｉｓ ａｎｏｔｈｅｒ ｎｕｍｂｅｒ．」）が正しい認識結果であることを、ユーザが確認すると、それらの２つの文は、ユーザデバイスのテキスト入力区域内に入力され、手書き入力区域は、ユーザが別の手書き入力を入力するために、クリアされる。 In some embodiments, the handwriting recognition model provides stroke order independent recognition at the sentence-to-sentence level. For example, in the handwriting input area, the handwritten character "

is present in the first handwritten sentence and the handwritten character "

is in a second handwritten sentence, and the two handwritten characters are separated by one or more other handwritten characters and/or words, the handwriting recognition model will 2 characters into a spatial array "

still provides recognition results denoted by . This recognition result, and the spatial arrangement of the two recognized characters, is such that when the user completes handwriting input, the recognition units for those two characters are arranged in the array "

will remain the same regardless of the user-provided temporal order of the strokes of those two letters. In some embodiments, the first handwritten character (eg, "

”) is the first handwritten sentence (e.g., “

is a number. ), provided by the user as part of a second handwritten character (e.g., "

”) is a second handwritten sentence (e.g., “

is another number. ), and these first and second handwritten sentences are simultaneously displayed within the handwriting input area of the user device. In some embodiments, this recognition result (e.g., "

is a number.

is another number. ) is the correct recognition result, those two sentences are entered into the text entry area of the user device, and the handwriting entry area is opened for the user to enter another handwriting entry. , is cleared.

In some embodiments, the handwriting recognition model is stroke-order independent, not only at the character level, but also at the phrase and sentence level, so that the user can make corrections to previous incomplete characters as well as subsequent characters. It can be executed after writing. For example, if the user forgets to write a particular stroke for a character before transitioning to writing one or more subsequent characters within the handwriting input area, the user may still write that particular stroke. Missing strokes can be filled in later in the correct place in the character to receive correct recognition results.

In conventional stroke-order dependent systems (e.g., HMM-based recognition systems), once a character is written, the character is fixed and the user can no longer make any changes to the character. . If the user wishes to make any changes, the user must delete that letter and all subsequent letters and start over. Some conventional recognition systems require the user to complete a handwritten character within a short predefined time window, and any strokes entered outside that predefined time window It cannot be included within the same recognition unit as other strokes provided between windows. Such conventional systems are difficult to use and cause a great deal of user frustration. Stroke order independent systems do not suffer from these shortcomings and allow users to complete characters in any order and in any time frame as the user sees fit. The user can also make modifications to previously written characters (eg, add one or more strokes) after writing one or more subsequent characters into the handwriting input interface. In some embodiments, the user can also individually delete previously written characters (e.g., using the methods described below in connection with FIGS. 21A, 22B) and write within the handwriting input interface. You can also rewrite the character in the same place in the .

」）を書き込んでいる。第１の複数のストロークは、手書き文字「

」を形成する。ユーザは、実際には文字「

」を書き込むことを意図していたが、１つのストロークを書き落としたことに留意されたい。第２の複数のストロークは、手書き文字「

」を形成する。ユーザが、「

」ではなく「

」を書き込むことを望んでいたことを、その後に理解すると、ユーザは、単純に、文字「

」に関するストロークの下方に、もう１つの垂直のストロークを記入することができ、ユーザデバイスは、その垂直のストロークを、第１の認識単位（例えば、「

」に関する認識単位）に割り当てる。ユーザデバイスは、第１の認識単位に関する新たな出力文字（例えば、「

」）を出力し、この新たな出力文字が、認識結果内の従前の出力文字（例えば、「

」）に置き換わる。図１０Ｃに示されるように、第３の手書きストロークの受信に応じて、ユーザデバイスは、その第３の手書きストロークの、第１の複数の手書きストロークへの相対的な近接性に基づいて、第１の複数の手書きストロークと同じ認識単位に、その第３の手書きストロークを割り当てる（１０３２）。一部の実施形態では、ユーザデバイスは、第１の複数の手書きストローク及び第３の手書きストロークに基づいて、訂正された入力画像を生成する（１０３４）。ユーザデバイスは、訂正された手書き文字のリアルタイム認識を実行するために、この訂正された入力画像を、手書き認識モデルに提供する（１０３６）。一部の実施形態では、ユーザデバイスは、第３の手書き入力の受信に応じて、訂正された入力画像に対応する第３の出力文字を表示し（１０４０）、この第３の出力文字は、第１の出力文字に置き換わり、デフォルトの書き込み方向に沿った空間的配列で、第２の出力文字と同時に表示される。 As shown in FIGS. 10B and 10C, the second plurality of handwriting strokes spatially follows the first plurality of handwriting strokes along the default writing direction of the handwriting input interface of the user device ( 1028), the second output character follows the first output character in spatial alignment along its default writing direction within the candidate display area of the handwriting input interface. The user device receives 1030 a third handwritten stroke from the user to correct the first handwritten character (i.e., the handwritten character formed by the first plurality of handwritten strokes); The handwritten strokes are received later in time than the first plurality of handwritten strokes and the second plurality of handwritten strokes. For example, a user may enter two characters in a left-to-right spatial arrangement (e.g., "

”). The first plurality of strokes is the handwritten character "

” is formed. The user actually sees the character "

, but missed one stroke. The second plurality of strokes is the handwritten character "

” is formed. If the user says "

"not"

”, the user simply writes the characters “

Below the stroke for "

” recognition unit). The user device outputs a new character for the first recognition unit (e.g., "

), and this new output character replaces the previous output character in the recognition result (e.g., "

”). As shown in FIG. 10C, in response to receiving a third handwritten stroke, the user device may determine a third handwritten stroke based on the relative proximity of the third handwritten stroke to the first plurality of handwritten strokes. Assign 1032 the third handwritten stroke to the same recognition unit as one handwritten stroke. In some embodiments, the user device generates a corrected input image based on the first plurality of handwritten strokes and the third handwritten stroke (1034). The user device provides this corrected input image to the handwriting recognition model to perform real-time recognition of the corrected handwriting (1036). In some embodiments, the user device displays (1040) a third output character corresponding to the corrected input image in response to receiving the third handwriting input, the third output character being: It replaces the first output character and is displayed simultaneously with the second output character in spatial alignment along the default writing direction.

In some embodiments, the handwriting recognition module recognizes handwriting input written in a left-to-right default writing direction. For example, a user can write characters from left to right and in one or more lines. In response to this handwriting input, the handwriting input module presents recognition results that include characters in a left-to-right spatial arrangement and in one or more lines as appropriate. If the user selects a recognition result, the selected recognition result is entered into the text entry area of the user device. In some embodiments, the default writing direction is top to bottom. In some embodiments, the default writing direction is right to left. In some embodiments, the user optionally changes the default writing direction to an alternate writing direction after recognition results are selected and the handwriting input area is cleared.

In some embodiments, the handwriting input module allows the user to enter multi-character handwriting input into the handwriting input area, and not all recognition units at once from the handwriting input. Allows deletion of strokes in one recognition unit. In some embodiments, the handwriting input module allows deletion of one stroke at a time from the handwriting input. In some embodiments, deletion of recognition units proceeds one-by-one in a direction opposite to the default writing direction, regardless of the order of recognition units or strokes entered to produce the current handwriting input. do. In some embodiments, the deletion of strokes proceeds one by one in the reverse order in which the strokes were entered in each recognition unit, and when all strokes in one recognition unit have been deleted, the strokes are deleted. Deletion proceeds to the next recognition unit in the direction opposite to the default writing direction.

In some embodiments, while the third output character and the second output character are simultaneously displayed as candidate recognition results in the candidate display area of the handwriting input interface, the user device receives a delete input from the user. to receive In response to this delete input, the user device deletes the second output character from the recognition result while maintaining the third output character within the displayed recognition result within the candidate display area.

」内の欠落した垂直のストロークを提供した後、ユーザが、削除入力を入力した場合には、文字「

」に関する認識単位内のストロークが、手書き入力区域から除去され、文字「

」が、ユーザデバイスの候補表示区域内の認識結果「

」から除去される。この削除の後、文字「

」に関するストロークは、手書き入力区域内に維持され、認識結果は、文字「

」のみを示す。 In some embodiments, as shown in FIG. 10C, the user device generates a first plurality of handwritten strokes, a second plurality of handwritten strokes, and a third handwritten stroke, each of which handwritten strokes is a Render 1042 in real-time as provided by the user. In some embodiments, in response to receiving a delete input from the user, the user device renders each of the second plurality of handwritten inputs (eg, corresponding to the second handwritten character) from the handwritten input area. while deleting a rendering of each of the first plurality of handwriting strokes and the third handwriting stroke (e.g., corresponding together to the corrected first handwriting character) within the handwriting input area Maintain (1044). For example, if the user has a character array "

If the user enters a delete input after providing the missing vertical stroke in

” is removed from the handwriting input area, leaving the character “

" is the recognition result in the candidate display area of the user device "

” is removed from After this deletion, the character "

” is kept within the handwriting input area, and the recognition result is the character “

” only.

In some embodiments, the handwritten characters are multi-stroke Chinese characters. In some embodiments, the first plurality of handwritten inputs is provided in cursive typeface. In some embodiments, the first plurality of handwritten inputs is provided in cursive typeface, and the handwritten characters are multi-stroke Chinese characters. In some embodiments, the handwriting is written in cursive Arabic. In some embodiments, the handwritten characters are written in other cursive scripts.

In some embodiments, the user device establishes respective default constraints for a set of acceptable dimensions for handwriting input and applies the currently accumulated plurality of handwriting strokes to the respective default constraints. Based on this, it is divided into a plurality of recognition units, from which respective input images are generated and provided to the handwriting recognition model for recognition as corresponding output characters.

In some embodiments, the user device receives additional handwriting strokes from the user after dividing the currently accumulated plurality of handwriting strokes. The user device assigns the additional handwritten stroke to each one of the plurality of recognition units based on the spatial location of the additional handwritten stroke with respect to the plurality of recognition units.

Attention is now directed to exemplary user interfaces for providing handwriting recognition and input on user devices. In some embodiments, this exemplary user interface is provided on a user device based on a multi-script handwriting recognition model that provides real-time, stroke order-independent handwriting recognition of a user's handwriting input. . In some embodiments, this exemplary user interface includes an exemplary handwriting input area 804, a candidate display area 804, and a text input area 808 (eg, as shown in FIGS. 8A and 8B). It is a user interface of the input interface 802 . In some embodiments, exemplary handwriting input interface 802 also includes multiple control elements 1102, such as a delete button, a spacebar, an enter button, a keyboard toggle button, and the like. One or more other areas and/or elements may be provided within handwriting input interface 902 to enable additional functionality described below.

As described herein, a multi-script handwriting recognition model can have a very broad repertoire of tens of thousands of characters in many different scripts and languages. As a result, for handwritten input, the recognition model is very likely to identify a large number of output characters, all of which have a reasonably good likelihood of being the characters intended by the user. have On user devices with limited display area, it is advantageous to provide only a subset of recognition results initially, while keeping other results available upon user request.

11A-11G for displaying a subset of the recognition results within the normal view of the candidate display area, along with affordances for invoking an extended view of the candidate display area for displaying the remainder of the recognition results. , shows an exemplary user interface of the . Furthermore, within the expanded view of the candidate display area, the recognition results are grouped into different categories and displayed on different tabbed pages within the expanded view.

FIG. 11A shows an exemplary handwriting input interface 802. FIG. The handwriting input interface includes a handwriting input area 804 , a candidate display area 806 and a text input area 808 . One or more control elements 1102 are also included within handwriting input interface 1002 .

As shown in FIG. 11A, the candidate display area 806 optionally includes an area for displaying one or more recognition results and an affordance 1104 for invoking an enhanced version of the candidate display area 806 (eg, an extended icon) and

11A-11C illustrate that when a user provides one or more handwriting strokes (eg, strokes 1106, 1108, and 1110) within handwriting area 804, the user device currently accumulates within handwriting area 804 indicates to identify and display each set of recognition results corresponding to the strokes being identified. As shown in FIG. 11B, after the user enters a first stroke 1106, the user device outputs three recognition results 1112, 1114, and 1116 (e.g., the characters "/", "1", and ", ) is identified and displayed. In some embodiments, within candidate display area 806, a small number of candidate characters are displayed in order according to the recognition confidence associated with each character.

In some embodiments, the highest ranked candidate result (eg, “/”) is provisionally displayed within text entry area 808 , eg, within box 1118 . The user confirms that the top-ranked candidate is the intended input with a simple confirmation input (e.g., by pressing the "Enter" key or providing a double-tap gesture within the handwriting area). , can optionally be verified.

FIG. 11C shows that if the user enters two more strokes 1108 and 1110 into the handwriting input area 804 before the user selects any of the candidate recognition results, those additional strokes are combined with the initial stroke 1106. Rendered within the handwriting input area 804, the candidate results indicate that the recognition units identified from the currently accumulated handwriting input are updated to reflect those changes. Based on the three strokes, the user device has identified a single recognition unit, as shown in FIG. 11C. Based on this single recognition unit, the user device identifies and displays several recognition results 1118-1124. In some embodiments, one or more of these recognition results (e.g., 1118 and 1122) currently displayed in candidate display area 806 are each a plurality of similar recognition results for the current handwriting input. These are representative candidate characters selected from the candidate characters with the same appearance.

11C and 11D, when a user selects affordance 1104 (eg, using a tap gesture with touch 1126 on affordance 1104), the candidate display areas are displayed (eg, 11C) to an extended view (eg, shown in FIG. 11D). In some embodiments, this expanded view shows all of the recognition results (eg, candidate characters) that have been identified for the current handwriting input.

In some embodiments, the initially displayed normal view of the candidate display area 806 shows only the most commonly used characters used in the respective script or language, whereas extended The view shows all candidate characters, including characters that are infrequently used in scripts or languages. This extended view of the candidate display area can be designed in various ways. 11D-11G show exemplary designs of expanded candidate display areas, according to some embodiments.

As shown in FIG. 11D, in some embodiments, expanded candidate display area 1128 comprises one or more tabbed pages (e.g., pages 1130, 1132, 1134 and 1136). The tabbed design shown in FIG. 11D allows the user to quickly find the desired category of characters and then find the character the user intended to enter within its corresponding tabbed page. Become.

」、「β」、「

」など）を含む。 In FIG. 11D, a first tabbed page 1130 displays all candidate characters identified for the currently accumulated handwriting input, including both commonly used characters as well as infrequent characters. indicate. As shown in FIG. 11D, tabbed page 1130 includes all of the characters that were shown within initial candidate display area 806 in FIG. additional characters (e.g. "

, β, and

”, etc.).

In some embodiments, the characters displayed in the initial candidate display area 806 are characters from a set of commonly used characters associated with a particular script (eg, encoded according to the Unicode standard). all characters in basic blocks of CJK scripts, such as In some embodiments, the characters displayed in the extended candidate display area 1128 are a set of low-frequency characters associated with the script (e.g., an extension of the CJK script, as encoded according to the Unicode standard). all characters in the block). In some embodiments, expanded candidate display area 1128 further includes candidate characters from other scripts not commonly used by users, such as Greek scripts, Arabic scripts, and/or glyph scripts.

In some embodiments, as shown in FIG. 11D, expanded candidate display areas 1128 each display a category of candidate characters (eg, all characters, infrequent characters, characters from Latin script, respectively). , and characters from the glyph script), respectively. Figures 11E-11G illustrate that the user can select each of the various tabbed pages to reveal candidate characters within the corresponding category. FIG. 11E shows only low-frequency characters (eg, characters from the extended block of CJK script) that correspond to the current handwriting input. FIG. 11F shows only Latin and Greek letters, corresponding to the current handwriting input. FIG. 11G shows glyphs only, corresponding to the current handwriting input.

In some embodiments, expanded candidate display area 1128 displays candidate characters within each tabbed page based on respective criteria (e.g., based on phonetic spelling, based on number of strokes, and based on number of strokes). Further includes one or more affordances for sorting (based on radicals, etc.). The ability to sort candidate characters within each category according to criteria other than recognition confidence scores provides users with additional capabilities to quickly find desired candidate characters for text input.

FIGS. 11H-11K show that in some embodiments, similar-looking candidate characters can be grouped, and only representative characters from each group of similar-looking candidate characters are included in the initial It is shown to be presented within the candidate display area 806 . Because the multiscript recognition model described herein can produce a number of nearly equally valid candidate characters for a given handwritten input, the recognition model replaces one candidate with another similar-looking character. Elimination at the expense of candidates is not always possible. On devices with limited display area, displaying many similar-looking candidates all at once does not help the user pick out the correct character, but this facilitates fine distinctions. and even if the desired character is visible to the user, selecting that character from a very crowded display with a finger or stylus is This is because it may be difficult.

」及び１１２２「Ｔ」）は、いずれのグループにも属さない候補文字（例えば、候補文字１１２０「

」及び１１２４「Ｊ」）とは異なる方式で（例えば、太枠ボックス内に）表示される。一部の実施形態では、グループの代表的文字を選択するための基準は、そのグループ内での候補文字の相対的な使用頻度に基づく。一部の実施形態では、他の基準を使用することができる。 In some embodiments, to address the above problem, the user device selects candidate characters that have high similarity to each other (eg, according to a concordance or dictionary of similar looking characters, or some image-based criteria). identify and group those candidate characters into their respective groups. In some embodiments, one or more groups of similar looking characters may be identified from the set of candidate characters for a given handwriting input. In some embodiments, the user device has identified representative candidate characters among multiple similar-looking candidate characters in the same group, and displays only the representative candidates in the initial candidate display area 806 . display in If a commonly used letter does not appear sufficiently similar to any other candidate letter, the letter itself is displayed. In some embodiments, representative candidate characters for each group (eg, candidate characters 1118 "

and 1122 "T") are candidate characters that do not belong to any group (e.g., candidate characters 1120 "

” and 1124 “J”) in a different manner (eg, in a bolded box). In some embodiments, the criteria for selecting representative characters for a group are based on the relative frequency of use of candidate characters within the group. Other criteria may be used in some embodiments.

In some embodiments, once the representative characters are displayed to the user, the user optionally expands the candidate display area 806 to show similar looking candidate characters in the enlarged view. can be done. In some embodiments, selection of a particular representative character can produce an expanded view of only candidate characters within the same group as the selected representative character.

Various designs are possible for providing a magnified view of similar looking candidates. FIGS. 11H-11K illustrate one embodiment in which a predefined gesture (eg, magnifying gesture) detected over a representative candidate character (eg, representative character 1118) invokes a magnified view of the representative candidate character. indicates A default gesture for invoking a magnified view (eg, a magnifying gesture) is different than a default gesture for selecting representative characters for text input (eg, a tap gesture).

」、「

」、及び「

」）が、同じ拡大グループ内には存在しない他の候補文字（例えば、「

」）と比較して、拡大されたビュー内に（例えば、それぞれ、拡大されたボックス１１４２、１１４４、及び１１４６内に）提示される。 As shown in FIGS. 11H and 11I, the user performs an expansion gesture (eg, as indicated by two contacts 1138 and 1140 moving away from each other) over the first representative character 1118. When provided, the area displaying representative characters 1118 is expanded to display three similar looking candidate characters (e.g., "

", "

",as well as"

), but other candidate characters that are not in the same expansion group (e.g., "

”) are presented in enlarged views (eg, in enlarged boxes 1142, 1144, and 1146, respectively).

」、「

」、及び「

」）の微細な区別は、ユーザによって、より容易に視認することができる。これらの３つの候補文字のうちの１つが、意図された文字入力である場合には、ユーザは、例えば、その文字が表示されている区域をタッチすることによって、その候補文字を選択することができる。図１１Ｊ及び図１１Ｋに示されるように、ユーザは、拡大ビュー内のボックス１１４４内に示される第２の文字（例えば、「

」）を、（接触１１４８で）選択している。これに応じて、選択された文字（例えば、「

」）が、テキスト入力区域８０８内に、カーソルによって示される挿入ポイントで入力される。図１１Ｋに示されるように、文字が選択されると、手書き入力区域８０４内の手書き入力、及び候補表示区域８０６（又は、候補表示区域の拡大ビュー）内の候補文字は、後続の手書き入力のために消去される。 As shown in FIG. 11I, three similar-looking candidate characters (e.g., "

", "

",as well as"

”) can be more easily visually recognized by the user. If one of these three candidate characters is the intended character input, the user can select that candidate character, for example, by touching the area where that character is displayed. can. As shown in FIGS. 11J and 11K, the user selects a second character (eg, "

”) is selected (with contact 1148). Depending on this, the selected character (e.g. "

”) is entered into the text entry area 808 at the insertion point indicated by the cursor. As shown in FIG. 11K, once a character is selected, the handwriting input within the handwriting input area 804 and the candidate characters within the candidate display area 806 (or an expanded view of the candidate display area) are used for subsequent handwriting input. erased for

In some embodiments, if the user does not see the desired candidate character within the enlarged view of the first representative candidate character 1142, the user optionally uses the same gesture to Other characters displayed within the candidate display area 806 can be enlarged. In some embodiments, by expanding another representative character in the candidate display area 806, the currently presented expanded view is automatically restored to the normal view. In some embodiments, the user optionally uses the shrink gesture to restore the currently zoomed view to the normal view. In some embodiments, the user can scroll (eg, left or right) the candidate display area 806 to reveal other candidate characters that are not visible within the candidate display area 806 .

12A and 12B illustrate that a first subset of recognition results is presented within an initial candidate display area while a second subset of recognition results is hidden until specifically invoked by the user. 12 is a flow diagram of an exemplary process 1200 presented within an expanded candidate display area that is not visible. In exemplary process 1200, the device identifies a subset of recognition results from a plurality of handwriting recognition results for handwriting input that have a visual similarity level above a predetermined threshold. The user device then selects a representative recognition result from the subset of recognition results and displays the selected representative recognition result within the candidate display area of the display. Process 1200 is illustrated in FIGS. 11A-11K.

As shown in FIG. 12A, in exemplary process 1200, a user device receives handwritten input from a user (1202). This handwriting input is one or more handwriting strokes (e.g., 1106 in FIG. 11C) provided within a handwriting input area (e.g., 806 in FIG. , 1108, 1110). The user device identifies 1204 a plurality of output characters (eg, characters shown in tabbed page 1130 of FIG. 11C) for the handwriting input based on the handwriting recognition model. The user device classifies 1206 the plurality of output characters into two or more categories based on predefined categorization criteria. In some embodiments, this predefined categorization criterion determines (1208) whether each character is a commonly used character or an infrequent character.

In some embodiments, the user device displays a first of the two or more categories (e.g., , commonly used characters), and the initial view of this candidate display area to invoke an extended view of the candidate display area (eg, 1128 in FIG. 11D). (eg, 1104 in FIG. 11C).

In some embodiments, the user device receives user input (1212) selecting an affordance for invoking the enhanced view, eg, as shown in FIG. 11C. In response to this user input, the user device displays each output character in the first category in the expanded view of the candidate display area and the characters in the initial view of the candidate display area, for example, as shown in FIG. 11D. display 1214 each output character in at least a second one of the two or more categories that was not previously displayed in the .

In some embodiments, each character in the first category is a character found in a dictionary of commonly used characters and each character in the second category is an infrequent character is a character found in the dictionary of In some embodiments, the dictionary of commonly used characters and the dictionary of infrequent characters are dynamically adjusted or updated based on usage history associated with the user device.

In some embodiments, the user device, from those multiple output characters, according to a predefined similarity criterion (e.g., based on a dictionary of similar characters, or based on some spatially derived feature): Groups of characters that are visually similar to each other are identified (1216). In some embodiments, the user device selects representative characters from a group of visually similar characters based on predefined selection criteria (eg, based on historical frequency of use). In some embodiments, this default selection criterion is based on the relative frequency of character usage within the group. In some embodiments, this default selection criterion is based on the preferred input language associated with the device. In some embodiments, representative candidates are based on other factors that indicate the likelihood that each candidate is the intended input by the user. These factors include, for example, whether the candidate character belongs to a script for the soft keyboard currently installed on the user's device, or whether the candidate character is associated with the user or user device. whether it is among the most commonly used character sets in any language.

」）を、その視覚的に類似する文字のグループ内の他の文字（例えば、「

」、「

」）の代わりに表示する（１２２０）。一部の実施形態では、各候補文字が、グループの代表的文字であるか、又はいずれのグループ内にも存在しない通常の候補文字であるかを示すために、候補表示区域の初期ビュー内に、視覚的指示（例えば、選択的な視覚的強調表示、特殊な背景）が提供される。一部の実施形態では、ユーザデバイスは、例えば、図１１Ｈに示されるように、候補表示区域の初期ビュー内に表示された代表的文字を対象とする、既定の拡大入力（例えば、拡大ジェスチャ）を、ユーザから受信する（１２２２）。一部の実施形態では、この既定の拡大入力の受信に応じて、ユーザデバイスは、例えば、図１１Ｉに示されるように、代表的文字の拡大されたビューと、視覚的に類似する文字のグループ内の１つ以上の他の文字の、それぞれの拡大されたビューとを、同時に表示する（１２２４）。 In some embodiments, the user device displays representative characters (eg, "

) to other characters within that group of visually similar characters (e.g., "

", "

”) is displayed (1220). In some embodiments, each candidate character has a , visual indications (eg, selective visual highlighting, special backgrounds) are provided. In some embodiments, the user device provides a default magnifying input (eg, a magnifying gesture) directed to representative characters displayed within the initial view of the candidate display area, eg, as shown in FIG. 11H. is received 1222 from the user. In some embodiments, in response to receiving this default magnifying input, the user device displays a magnified view of representative characters and groups of visually similar characters, for example, as shown in FIG. 11I. A respective magnified view of one or more other characters in the character is simultaneously displayed (1224).

In some embodiments, this default magnifying input is a magnifying gesture detected over a representative character displayed within the candidate display area. In some embodiments, this default magnifying input is a touch detected over a representative character displayed within the candidate display area and lasting longer than a predefined threshold time. In some embodiments, this sustained touch to expand the group has a longer threshold duration than the tap gesture that selects the representative character for text input.

In some embodiments, each representative character is displayed simultaneously with a corresponding affordance (eg, a corresponding magnifying button) for invoking a magnified view of that group of similar-looking candidate characters. In some embodiments, the default magnifying input is a selection of corresponding affordances associated with the representative character.

As described herein, in some embodiments, the repertoire of multi-script handwriting recognition models includes glyph scripts. The handwriting recognition module can recognize glyphs based on the user's handwriting input. In some embodiments, the handwriting recognition module presents both glyphs identified directly from handwriting and the characters or words in natural human language that represent the identified glyphs. In some embodiments, the handwriting input module recognizes characters or words in natural human language based on the user's handwriting input, and corresponds to the recognized characters or words and the recognized characters or words. Present both with the emoji you want. In other words, the handwriting input module provides a method for entering glyphs without switching from the handwriting input interface to the glyph keyboard. Furthermore, the handwriting input module also provides a method for entering regular natural language characters by drawing glyphs by hand. Figures 13A-13E provide exemplary user interfaces that illustrate these various methods of entering glyphs and regular natural language characters.

」）を受信している。 FIG. 13A shows an exemplary handwriting input interface 802 invoked under a chat application. Handwriting input interface 802 includes handwriting input area 804 , candidate display area 806 , and text input area 808 . In some embodiments, once the user is satisfied with the text composition in text entry area 808, the user can choose to send the text composition to another participant in the current chat session. The interaction history of the chat session is shown in interaction panel 1302 . In this example, a user may enter a chat message 1304 displayed within interaction panel 1302 (e.g., "Happy Birthday

”) is being received.

As shown in FIG. 13B, the user has provided handwriting input 1306 in handwriting area 804 for the English word "Thanks." In response to handwriting input 1306, the user device has identified several candidate recognition results (eg, recognition results 1308, 1310, and 1312). The top ranked recognition result 1303 has been provisionally entered in box 1314 within text entry area 808 .

As shown in FIG. 13C, after the user enters the handwritten word “Thanks” into handwriting input area 806, the user then enters stroke 1316 (eg, a rounded circle down) into handwriting input area 806. ), which depicts a stylized exclamation point. The user device recognizes that this additional stroke 1316 forms a separate recognition unit from other recognition units previously recognized from accumulated handwriting strokes 1306 in handwriting input area 806 . Based on this newly entered recognition unit (i.e., the recognition unit formed by stroke 1316), the user device uses a handwriting recognition model to identify glyphs (e.g., stylized "!"). do. Based on this recognized glyph, the user device presents a first recognition result 1318 (eg, "Thanks!" with a stylized "!") in the candidate display area 806. Additionally, the user device also identifies the number "8", which is also visually similar to the newly entered recognition unit. Based on this recognized digit, the user device presents a second recognition result 1322 (eg, “Thanks 8”) within candidate display area 806 . Further, based on the identified glyph (eg, stylized "!"), the user device also identifies the canonical character (e.g., canonical character "!") corresponding to the glyph. Based on this indirectly recognized canonical character, the user device presents a third recognition result 1320 (eg, “Thanks!” with a canonical “!”) in candidate display area 806 . At this point, the user may select any one of candidate recognition results 1318 , 1320 , and 1322 to enter that candidate recognition result within text entry area 808 .

」、及び代替的に数字「０」を特定する。新たな認識単位から認識された、これらの新たな候補文字に基づいて、ユーザデバイスは、２つの更新された候補認識結果１３２６及び１３３０（例えば、「Ｔｈａｎｋｓ

」及び「Ｔｈａｎｋｓ ８０」）を提示する。一部の実施形態では、ユーザデバイスは、特定された絵文字（例えば、「

」）に対応する、正規の文字又は語（例えば、「Ｌｏｖｅ」）を更に特定する。認識された絵文字に関する、その特定された正規の文字又は語に基づいて、ユーザデバイスは、認識された絵文字が、その対応する正規の文字又は語で置き換えられた、第３の認識結果１３２８を提示する。図１３Ｄに示されるように、認識結果１３２８では、絵文字「

」が、通常の感嘆符「！」で置き換えられ、絵文字「

」が、正規の文字又は語「Ｌｏｖｅ」で置き換えられている。 The user continues to provide additional handwriting strokes 1324 within the handwriting input area 806, as shown in FIG. 13D. This time, the user has drawn a stylized exclamation mark followed by a heart symbol. In response to this new handwriting stroke 1324, the user device recognizes that the newly provided handwriting stroke 1324 forms yet another new unit of recognition. Based on this new recognition unit, the user device selects the pictogram "

”, and alternatively the digit “0”. Based on these new candidate characters recognized from the new recognition units, the user device generates two updated candidate recognition results 1326 and 1330 (e.g., "Thanks

” and “Thanks 80”). In some embodiments, the user device displays a specified emoji (e.g., "

) that corresponds to the canonical character or word (eg, “Love”). Based on the identified canonical characters or words for the recognized glyph, the user device presents a third recognition result 1328 in which the recognized glyph is replaced with its corresponding canonical character or word. do. As shown in FIG. 13D, the recognition result 1328 includes the glyph "

” is replaced by a normal exclamation point ``!'' and the emoji ``

' is replaced by the regular letter or word 'Love'.

」を示す候補結果１３２６）を選択しており、この選択された認識結果のテキストが、テキスト入力区域８０８内に入力され、その後、チャットセッションの他の参加者に送信される。メッセージの吹き出し１３３２は、対話パネル１３０２内に、そのメッセージのテキストを示す。 As shown in FIG. 13E, the user selects one of the candidate recognition results (eg, the mixed script text "Thanks

1326) and the text of this selected recognition result is entered into the text entry area 808 and then sent to the other participants in the chat session. Message bubble 1332 shows the text of the message in dialog panel 1302 .

FIG. 14 is a flow diagram of an exemplary process 1400 in which a user enters glyphs using handwriting input. 13A-13E show an exemplary process 1400, according to some embodiments.

」又は絵文字「

」）と、自然人間言語のスクリプトからの少なくとも第１の文字（例えば、図１３Ｄでの、語「Ｔｈａｎｋｓ」からの文字）とを含む。一部の実施形態では、ユーザデバイスは、例えば、図１３Ｄに示されるように、それらの第１の絵文字（例えば、図１３Ｄでの、様式化された感嘆符「

」又は絵文字「

」）と自然人間言語のスクリプトからの第１の文字（例えば、図１３Ｄでの、語「Ｔｈａｎｋｓ」からの文字）とを含む認識結果（例えば、図１３Ｄでの結果１３２６）を、手書き入力インタフェースの候補表示区域内に表示する（１４０６）。 In process 1400, a user device receives (1402) handwriting input from a user. The handwriting input includes a plurality of handwriting strokes provided within the handwriting input area of the handwriting input interface. In some embodiments, the user device recognizes (1404) multiple output characters from its handwritten input based on a handwriting recognition model. In some embodiments, those output characters are at least a first glyph (e.g., the stylized exclamation point "

" or Emoji "

”) and at least the first character from the natural human language script (eg, the character from the word “Thanks” in FIG. 13D). In some embodiments, the user device displays their first glyph (e.g., the stylized exclamation mark "

" or Emoji "

”) and the first character from the natural human language script (e.g., the characters from the word “Thanks” in FIG. 13D) are sent to the handwriting input interface. is displayed in the candidate display area (1406).

」及び「

」の絵文字を示す、認識結果）を表示する（１４１２）。一部の実施形態では、第２の認識結果を表示することは、少なくとも第１の意味的単位（例えば、語「Ｔｈａｎｋｓ」）を含む第３の認識結果（例えば、認識結果「Ｔｈａｎｋｓ

」）と同時に、第２の認識結果を表示することを更に含む。 In some embodiments, based on the handwriting recognition model, the user device optionally recognizes (1408) at least a first semantic unit (eg, the word "Thanks") from the handwriting input; This first semantic unit includes each character, word, or phrase capable of conveying each semantic meaning in each human language. In some embodiments, the user device identifies a second glyph (e.g., the "handshake" glyph) associated with the first semantic unit (e.g., the word "Thanks") recognized from the handwritten input. (1410). In some embodiments, the user device, within the candidate display area of the handwriting input interface, selects a second glyph that includes at least a second glyph identified from the first semantic unit (eg, the word “Thanks”). Recognition results (e.g. "handshake" emoji followed by "

"as well as"

, a recognition result) is displayed (1412). In some embodiments, displaying the second recognition result includes a third recognition result (eg, the recognition result "Thanks") that includes at least the first semantic unit (eg, the word "Thanks").

”) and simultaneously displaying the second recognition result.

In some embodiments, the user receives user input selecting a first recognition result displayed within the candidate display area. In some embodiments, in response to this user input, the user device enters text of the selected first recognition result into a text entry area of the handwriting input interface, the text being at least a first 1 glyph and the first character from the natural human language script. In other words, the user can use a single handwriting input (albeit a handwriting input containing multiple strokes) within the handwriting input area to enter mixed script text without switching between a natural language keyboard and an emoji keyboard. Input can be entered.

In some embodiments, the handwriting recognition model is trained on a multi-script training corpus containing written samples corresponding to characters of at least three non-overlapping scripts, the three non-overlapping scripts being Emoji, Chinese Includes a set of characters, and a Latin script.

」の絵文字）に対応する、第２の意味的単位（例えば、語「Ｌｏｖｅ」）を特定する（１４１４）。一部の実施形態では、ユーザデバイスは、手書き入力インタフェースの候補表示区域内に、第１の絵文字（例えば、「

」の絵文字）から特定された第２の意味的単位（例えば、語「Ｌｏｖｅ」）を少なくとも含む、第４の認識結果（例えば、図１３Ｄでの１３２８）を表示する（１４１６）。一部の実施形態では、ユーザデバイスは、図１３Ｄに示されるように、候補表示区域内に、第１の認識結果（例えば、結果「Ｔｈａｎｋｓ

」）と同時に、第４の認識結果（例えば、結果１３２８「Ｔｈａｎｋｓ！Ｌｏｖｅ」）を表示する。 In some embodiments, the user device recognizes the first glyph directly from the handwriting input (e.g., "

A second semantic unit (eg, the word “Love”) is identified 1414 that corresponds to the word “Love”. In some embodiments, the user device displays a first glyph (e.g., "

A fourth recognition result (eg, 1328 in FIG. 13D) is displayed (1416) that includes at least the second semantic unit (eg, the word "Love") identified from the "" glyph). In some embodiments, the user device displays a first recognition result (e.g., the result "Thanks

”) and simultaneously display a fourth recognition result (eg result 1328 “Thanks! Love”).

」を書き込む代わりに、様式化された猫を描くことができる。ユーザデバイスが、このユーザによって提供された手書き入力に基づいて、「猫」に関する絵文字を特定する場合には、ユーザデバイスはまた、任意選択的に、候補表示区域内に、「猫」に関する絵文字と共に、中国語で「猫」を意味する中国文字「

」も提示する。認識された絵文字に関する通常のテキストを提示することによって、ユーザデバイスは、周知の絵文字に一般的に関連付けられる、少ない数の様式化されたストロークを使用して、複雑な文字又は語を入力する、代替的な方法を提供する。一部の実施形態では、ユーザデバイスは、絵文字を、１つ以上の好ましいスクリプト又は言語（例えば、英語又は中国語）での、それらの対応する通常のテキスト（例えば、文字、語、語句、記号など）と結びつける、辞書を記憶する。 In some embodiments, the user device allows the user to enter regular text by drawing glyphs. For example, if the user does not know how to spell the word "elephant," the user optionally draws a stylized glyph for "elephant" within the handwriting input area, and the user device If the handwriting input can be correctly recognized as a glyph for "elephant", the user device optionally also selects normal text as one of the recognition results displayed in the candidate display area. also presents the word "elephant". In another example, the user may type the Chinese character "

, you can draw a stylized cat. If the user device identifies an emoji for "cat" based on the handwriting input provided by this user, then the user device optionally also displays in the candidate display area along with the emoji for "cat" , the Chinese character for "cat" in Chinese "

” is also presented. By presenting ordinary text on recognized pictograms, the user device enters complex characters or words using a small number of stylized strokes commonly associated with well-known pictograms; Provide an alternative method. In some embodiments, the user device converts glyphs into their corresponding regular text (e.g., letters, words, phrases, symbols) in one or more preferred scripts or languages (e.g., English or Chinese). etc.), memorize the dictionary.

In some embodiments, the user device recognizes glyphs based on visual similarity of glyphs to images generated from handwritten input. In some embodiments, to enable recognition of glyphs from handwritten input, the handwriting recognition model used on the user device includes handwriting samples corresponding to characters of a natural human language script, and an artificial It is trained using a training corpus containing both handwriting samples corresponding to the set of designed glyphs. In some embodiments, glyphs associated with the same semantic concept may have different appearances when used within mixed input having text in different natural languages. For example, an emoji for the semantic concept of "Love" can be an emoji for "heart" when presented in plain text in one natural language (e.g., Japanese), and an emoji for "heart" in another natural language (e.g., , English or French), it can be a "kiss" glyph.

As described herein, when performing multi-character handwriting recognition, the handwriting input module performs segmentation of the currently accumulated handwriting input within the handwriting input area and divides the accumulated handwriting input into Separates strokes into one or more recognition units. One of the parameters used to determine how to divide the handwriting is how the strokes are clustered within the handwriting area and the distance between clusters of different strokes. can be a distance of People have different typefaces. Some people tend to write at very low densities with large distances between strokes, or different parts of the same letter, while others write with large distances between strokes, or different parts of the same letter. They tend to write at very high densities with very small distances between characters. Even for the same user, imperfect planning can cause handwriting to deviate from a balanced appearance and become skewed, stretched, or compressed in various ways. As described herein, the multi-script handwriting recognition model provides stroke-order independent recognition, thus allowing a user to write characters or parts of characters out of order. . As a result, achieving spatial uniformity and balance of handwriting input between characters can be difficult.

In some embodiments, the handwriting input model described herein allows the user to have the handwriting input module fuse two adjacent recognition units into a single recognition unit, or A method is provided for signaling whether a recognition unit should be split into two separate recognition units. With this user's help, the handwriting input module can correct the initial split and produce the result intended by the user.

15A-15J illustrate some exemplary user interfaces and processes by which a user provides predefined pinch and expand gestures to modify recognition units identified by a user device.

」、「

」、及び「

」）を提示している。 As shown in FIGS. 15A and 15B, a user has entered a plurality of handwriting strokes 1502 (eg, three strokes) within handwriting input area 806 of handwriting input interface 802 . The user device has identified a single unit of recognition based on the currently accumulated handwriting strokes 1502, and within the candidate display area 806, three candidate characters 1508 (e.g., "

", "

",as well as"

”).

FIG. 15C shows the user entering a few more additional strokes 1510 to the right of the initial handwriting stroke 1502 within the handwriting input area 606 . The user device determines (eg, based on the dimensions and spatial distribution of strokes 1502 and 1510) that stroke 1502 and stroke 1510 should be considered two separate units of recognition. Based on this recognition unit separation, the user devices provide their first recognition unit and second recognition unit input images to a handwriting recognition model to obtain two sets of candidate characters. The user device then generates multiple recognition results (eg, 1512, 1514, 1516, and 1518) based on various combinations of those recognized characters. Each recognition result includes a recognized character for the first recognition unit and a recognized character for the second recognition unit. As shown in FIG. 15C, multiple recognition results 1512, 1514, 1516, and 1518 each include two recognized characters.

」）の左部分（例えば、左の部首「

」）と右部分（例えば、右の部首「

」）との間に、過度に大きい空間を不用意に残したと想定する。候補表示区域８０６内に提示された結果（例えば、１５１２、１５１４、１５１６、及び１５１８）を視認して、ユーザは、ユーザデバイスが、現在の手書き入力を、誤って２つの認識単位へと分割していることを理解する。この分割は、客観的な基準に基づき得るものであるが、ユーザにとっては、現在の手書き入力を削除し、この文字全体を、左部分と右部分との距離をより小さくして再度書き直すことは、望ましいものではない。 In this example, the user actually intended for this handwritten input to be recognized as a single character, but the handwritten character (e.g., "

) (e.g., the left radical "

") and the right part (e.g., the right radical "

), and inadvertently left an excessively large space between them. Upon viewing the results (e.g., 1512, 1514, 1516, and 1518) presented in the candidate display area 806, the user may realize that the user device has erroneously split the current handwriting input into two recognition units. understand that Although this division can be based on objective criteria, it is not for the user to delete the current handwriting input and rewrite the entire character with a smaller distance between the left and right parts. , is not desirable.

Instead, as shown in FIG. 15D, the user uses a pinch gesture on two clusters of handwriting strokes 1502 and 1510 to indicate that the two recognition units identified by their handwriting input module are Indicates to the handwriting input module that it should be merged as a single recognition unit. This pinch gesture is illustrated by two contacts 1520 and 1522 on the touch-sensitive surface moving towards each other.

」、「

」、及び「

」）を取得している。一部の実施形態では、図１５Ｅに示されるように、ユーザデバイスは、任意選択的に、手書き入力区域８０６内の手書き入力のレンダリングを、手書きストロークの左クラスタと右クラスタとの距離が低減されるように調節する。一部の実施形態では、ユーザデバイスは、手書き入力区域６０８内に示される手書き入力のレンダリングを、ピンチジェスチャに応じて変更することがない。一部の実施形態では、ユーザデバイスは、手書き入力区域８０６内で検出される、（単一の接触とは対照的な）２つの同時接触に基づいて、ピンチジェスチャを入力ストロークから識別する。 FIG. 15E illustrates that in response to a user's pinch gesture, the user device corrects the division of the currently accumulated handwriting input (e.g., strokes 1502 and 1510) and divides the handwriting strokes into a single recognition unit. It shows that it is fused with As shown in FIG. 15E, the user device provides an input image based on the corrected recognition unit to the handwriting recognition model, and three new candidate characters 1524, 1526, and 1528 (for example, "

", "

",as well as"

”). In some embodiments, as shown in FIG. 15E, the user device optionally renders the handwriting input within the handwriting input area 806 such that the distance between the left and right clusters of handwriting strokes is reduced. Adjust so that In some embodiments, the user device does not change the rendering of the handwriting shown within the handwriting area 608 in response to the pinch gesture. In some embodiments, the user device identifies pinch gestures from input strokes based on two simultaneous contacts (as opposed to a single contact) detected within handwriting input area 806 .

」に関するストローク）の右に、更に２つのストローク１５３０を入力している。ユーザデバイスは、この新たに入力されたストローク１５３０が、新たな認識単位であることを判定し、その新たに特定された認識単位に関する候補文字（例えば、「

」）を認識する。次いで、ユーザデバイスは、その新たに特定された文字（例えば、「

」）を、以前に特定された認識単位に関する候補文字と組み合わせ、候補表示区域８０６内に、幾つかの異なる認識結果（例えば、結果１５３２及び１５３４）を提示する。 As shown in FIG. 15F, the user has previously entered the handwriting input (i.e., the character "

), two more strokes 1530 are entered. The user device determines that this newly entered stroke 1530 is a new recognition unit, and the candidate character for the newly identified recognition unit (e.g., "

”). The user device then displays the newly identified character (e.g., "

”) with candidate characters for previously identified recognition units to present several different recognition results (eg, results 1532 and 1534) in candidate display area 806 .

Following handwriting stroke 1530, the user continues to write additional strokes 1536 (eg, three additional strokes) to the right of stroke 1530, as shown in FIG. 15G. Because the horizontal distance between strokes 1530 and 1536 is very small, the user device determines that strokes 1530 and 1536 belong to the same recognition unit, and adds them to the handwriting recognition model formed by strokes 1530 and 1536. Provide an input image. The handwriting recognition model identifies three different candidate characters for the corrected recognition unit and produces two corrected recognition results 1538 and 1540 for the currently accumulated handwriting input.

」及び「±」）として意図されていると想定する。ユーザは、ユーザデバイスが、２つのストローク１５３０及び１５３６のセットを、誤って単一の認識単位へと組み合わせていることを視認した後、ユーザは、続けて拡大ジェスチャを提供して、ユーザデバイスに、２つのストローク１５３０及び１５３６のセットが、２つの別個の認識単位へと分離されるべきであることを通知する。図１５Ｈに示されるように、ユーザは、ストローク１５３０及び１５３６の周りで、２つの接触１５４２及び１５４４を実施して、次いで、それらの２つの接触を、概して水平な方向で（すなわち、デフォルトの書き込み方向に沿って）互いから離れる方向に移動させる。 In this example, the last two sets of strokes 1530 and 1536 are actually two separate characters (e.g., "

” and “±”). After the user sees that the user device has erroneously combined the set of two strokes 1530 and 1536 into a single recognition unit, the user continues to provide an enlargement gesture to direct the user device to , signals that the set of two strokes 1530 and 1536 should be separated into two separate recognition units. As shown in FIG. 15H, the user makes two contacts 1542 and 1544 around strokes 1530 and 1536 and then makes those two contacts in a generally horizontal orientation (i.e., the default writing direction) away from each other.

FIG. 15I shows that in response to this user expansion gesture, the user device corrects the previous division of the currently accumulated handwriting input, assigning strokes 1530 and 1536 to two consecutive recognition units. indicates that Based on the input images generated for those two separate recognition units, the user device identifies one or more candidate characters for the first recognition unit based on stroke 1530, and identifies one or more candidate characters for the first recognition unit based on stroke 1536. One or more candidate characters are identified for the two recognition units. The user device then generates two new recognition results 1546 and 1548 based on various combinations of those recognized characters. In some embodiments, the user device optionally modifies the rendering of strokes 1536 and 1536 to reflect previously identified separation of recognition units.

As shown in FIGS. 15J, 15K, the user has selected (as indicated by contact 1550) one of the candidate recognition results displayed in candidate display area 806, and the selected A recognition result (eg, result 1548) has been entered in the text entry area 808 of the user interface. After the selected recognition result has been entered into the text input area 808, both the candidate display area 806 and the handwriting input area 804 are cleared and ready to display subsequent user input.

Figures 16A, 16B inform how the user splits or corrects the existing splits of the current handwriting input using predefined gestures (e.g., pinch and/or zoom gestures). , is a flow diagram of an exemplary process 1600. FIG. Figures 15J and 15K provide diagrams of an exemplary process 1600, according to some embodiments.

In some embodiments, the user device receives (1602) handwriting input from the user. The handwritten input includes multiple handwritten strokes provided within a touch-sensitive surface coupled to the device. In some embodiments, the user device renders 1604 the multiple handwriting strokes in real-time within a handwriting input area of the handwriting input interface (eg, handwriting input area 806 of FIGS. 15A-15K). The user device receives one of a pinch gesture input and a magnifying gesture input over their multiple handwriting strokes, eg, as shown in FIGS. 15D and 15H.

In some embodiments, upon receiving a pinch gesture input, the user device treats multiple handwriting strokes as a single recognition unit, e.g., as shown in FIGS. A first recognition result is generated 1606 based on the plurality of handwritten strokes.

In some embodiments, upon receiving a magnifying gesture input, the user device converts multiple handwriting strokes into two separate units of recognition separated by the magnifying gesture input, eg, as shown in FIGS. 15G-15I. , to generate a second recognition result based on the plurality of handwritten strokes (1608).

In some embodiments, upon generating a respective one of the first recognition result and the second recognition result, the user device outputs the generated recognition result, for example, as shown in FIGS. 15E and 15I. The results are displayed in the candidate display area of the handwriting input interface.

In some embodiments, pinch gesture input includes two simultaneous contacts on the touch-sensitive surface that converge toward each other within an area occupied by multiple handwriting strokes. In some embodiments, the magnifying gesture input includes two simultaneous contacts on the touch-sensitive surface that are separated from each other within an area occupied by multiple handwritten strokes.

In some embodiments, the user device identifies two adjacent units of recognition from these multiple handwritten strokes (eg, 1614). The user device displays an initial recognition result (e.g., result 1512, 1514, 1516, and 1518) are displayed (1616). In some embodiments, when displaying a first recognition result (eg, results 1524, 1526, or 1528 in FIG. 15E) in response to a pinch gesture, the user device displays an initial The recognition result is replaced with its first recognition result (1618). In some embodiments, the user device receives a pinch gesture input (1620) while initial recognition results are being displayed in the candidate display area, as shown in FIG. 15D. In some embodiments, in response to this pinch gesture input, the user device may: The handwritten strokes are re-rendered (1622).

」「

」）を含む、初期の認識結果（例えば、図１５Ｇの結果１５３８又は１５４０）を表示する（１６２６）。一部の実施形態では、拡大ジェスチャに応じて、第２の認識結果（例えば、図１５Ｉでの結果１５４６又は１５４８）を表示する場合、ユーザデバイスは、例えば、図１５Ｈ、図１５Ｉに示されるように、候補表示区域内で、初期の認識結果（例えば、結果１５３８又は１５４０）を、その第２の認識結果（例えば、結果１５４６又は１５４８）で置き換える（１６２８）。一部の実施形態では、ユーザデバイスは、図１５Ｈに示されるように、初期の認識結果が候補表示区域内に表示されている間に、拡大ジェスチャ入力を受信する（１６３０）。一部の実施形態では、この拡大ジェスチャ入力に応じて、ユーザデバイスは、図１５Ｈ及び図１５Ｉに示されるように、手書き入力区域内の、第１の認識単位に割り当てられた、第１のストロークのサブセットと、第２の認識単位に割り当てられた、第２の手書きストロークのサブセットとの距離を増大させるように、複数の手書きストロークを再レンダリングする（１６３２）。 In some embodiments, the user device identifies 1624 a single unit of recognition from these multiple handwriting strokes. The user device displays characters recognized from that single recognition unit (e.g., "

""

), including initial recognition results (eg, results 1538 or 1540 in FIG. 15G) are displayed (1626). In some embodiments, when displaying a second recognition result (eg, result 1546 or 1548 in FIG. 15I) in response to the magnification gesture, the user device may, for example, Then, within the candidate display area, replace 1628 the initial recognition result (eg, result 1538 or 1540) with its second recognition result (eg, result 1546 or 1548). In some embodiments, the user device receives the magnifying gesture input (1630) while the initial recognition results are displayed within the candidate display area, as shown in FIG. 15H. In some embodiments, in response to this magnifying gesture input, the user device moves the first stroke within the handwriting input area, assigned to the first recognition unit, as shown in FIGS. 15H and 15I. and the second subset of handwritten strokes assigned to the second recognition unit (1632).

In some embodiments, immediately after the user provides strokes and realizes that they may be too spread out for correct segmentation based on the standard segmentation process, The user optionally provides a pinch gesture to inform the user device to process those multiple strokes as a single unit of recognition. A user device can distinguish a pinch gesture from a regular stroke based on the two simultaneous contacts present in the pinch gesture. Similarly, in some embodiments, users provide strokes and realize that they may be too dense for correct segmentation based on standard segmentation processes. Shortly thereafter, the user optionally provides a magnifying gesture to inform the user device to treat the multiple strokes as two separate units of recognition. A user device can distinguish an expansion gesture from a regular stroke based on the two simultaneous contacts present in the pinch gesture.

など）内の単一の構成要素を、その複合文字の種々の部分（例えば、上、下、左、又は右の部分）に分離することの手引きも、提供することができる。このことは、複雑な複合文字を手で書き込む場合に、ユーザは正しい比率及びバランスを失う傾向があるため、複雑な複合中国文字を認識するために特に役立つ。手書き入力の完了後に、例えば、ピンチジェスチャ及び拡大ジェスチャによって、手書き入力の比率及びバランスを調節することが可能であることは、ユーザが、正しい比率及びバランスに到達するための幾つかの試みを実施することを必要とせずに、正しい文字を入力するために、特に役立つ。 In some embodiments, the direction of movement of the pinch or expand gesture is optionally used to provide further guidance on how to split the stroke under the gesture. For example, with respect to the handwriting area, if multiple lines of handwriting are in effect, a pinch gesture with two vertically moving contacts will cause the handwriting input module to specify two lines identified in two adjacent lines. Recognition units can be signaled to merge into a single recognition unit (eg, as top and bottom radicals). Similarly, a magnifying gesture with two touches moving vertically can tell the handwriting input module to separate a single recognition unit into two recognition units in two adjacent lines. . In some embodiments, the pinch and magnify gestures also guide the division in sub-parts of character input, e.g., fusing two sub-components, or compound characters (

) into various parts (eg, top, bottom, left, or right parts) of the compound character can also be provided. This is particularly useful for recognizing complex compound Chinese characters, as users tend to lose the correct proportions and balance when writing complex compound characters by hand. The ability to adjust the proportion and balance of the handwriting after completion of the handwriting, for example by pinching and expanding gestures, suggests that the user may make several attempts to reach the correct proportion and balance. It is especially useful for entering the correct characters without having to

As described herein, the handwriting input module allows a user to enter multi-character handwriting input, within a handwriting input area, within a single character, across multiple characters, or even allows out-of-order strokes for their multi-character handwritten input across multiple words, sentences, and/or lines. In some embodiments, the handwriting input module also provides character-by-character deletion within the handwriting input area, where the order of character deletion is in the opposite direction of writing and the stroke for each character is It is irrelevant whether provided in the handwriting input area. In some embodiments, deletion of each recognition unit (e.g., character or radical) within the handwriting input area is optionally performed on a stroke-by-stroke basis, where the strokes are removed within the recognition unit. Deleted in the reverse chronological order provided. 17A-17H illustrate exemplary user interfaces for providing character-by-character deletion within multi-character handwritten input in response to deletion input from a user.

」、「

」）は全て、既定の閾値を下回る認識信頼度を有する。一部の実施形態では、ユーザデバイスは、第１の認識単位に関する候補文字（例えば、「

」）のみを含むが、第２の認識単位に関するいずれの候補文字も含まない、部分的認識結果（例えば、結果１７１２）を、候補表示区域８０６内に提示する。一部の実施形態では、ユーザデバイスは、認識信頼度が既定の閾値に合格しているか否かにかかわらず、双方の認識単位に関する候補文字を含む、完全認識結果（例えば、結果１７１４又は１７１６）を更に表示する。部分的認識結果を提供することは、その手書き入力のいずれの部分に訂正が必要であるかを、ユーザに通知する。更には、ユーザはまた、最初に、その手書き入力の正しく認識された部分を入力することを選択し、次いで、正しく認識されなかった部分を書き直すこともできる。 As shown in FIG. 17A, a user has provided a plurality of handwriting strokes 1702 within the handwriting input area 804 of the handwriting input user interface 802 . Based on these currently accumulated strokes 1702 , the user device presents three recognition results (eg, results 1704 , 1706 and 1708 ) within candidate display area 806 . As shown in FIG. 17B, the user has provided additional strokes 1710 within the handwriting input area 806 . The user device recognizes three new output characters and replaces the three previous recognition results 1704 , 1706 and 1708 with three new recognition results 1712 , 1714 and 1716 . In some embodiments, even if the user device identifies two separate units of recognition from the current handwriting input (e.g., stroke 1702 and stroke 1710), as shown in FIG. 17B , strokes 1710 do not correspond well to any known characters in the handwriting recognition module's repertoire. As a result, the candidate character identified for the recognition unit containing stroke 1710 (e.g., "

", "

') all have recognition confidence below a predefined threshold. In some embodiments, the user device selects candidate characters for the first recognition unit (e.g., "

”) but not any candidate characters for the second recognition unit (eg, result 1712) is presented in the candidate display area 806. FIG. In some embodiments, the user device generates a full recognition result (e.g., result 1714 or 1716) that includes candidate characters for both recognition units, whether or not the recognition confidence passes a predefined threshold. display more. Providing partial recognition results informs the user which parts of their handwritten input need correction. Furthermore, the user may also choose to first enter the correctly recognized portion of their handwriting input and then rewrite the incorrectly recognized portion.

」）が認識され、新たな認識結果１７２０、１７２２、及び１７２４のセットが生成される。この場合も、第１の認識結果１７２０は、ストローク１７１０に関して特定される候補文字のいずれもが、既定の信頼度閾値を満たさないため、部分的認識結果である。 FIG. 17C shows the user continuing to provide additional handwriting strokes 1718 to the left of stroke 1710 . Based on the relative location and distance of stroke 1718 , the user device determines that this newly added stroke belongs to the same recognition unit as the cluster of handwritten strokes 1702 . Based on this corrected recognition unit, a new character for the first recognition unit (e.g., "

”) is recognized and a new set of recognition results 1720, 1722, and 1724 is generated. Again, the first recognition result 1720 is a partial recognition result because none of the candidate characters identified for stroke 1710 meet the predefined confidence threshold.

」）に関する手書きストロークの全ての入力を完了しており、正しい認識結果１７２８が、候補表示区域８０６内に示される。 FIG. 17D shows that the user is now entering a number of new strokes 1726 between strokes 1702 and 1710. FIG. The user device assigns this newly entered stroke 1726 to the same recognition unit as stroke 1710 . At this point, the user has entered two Chinese characters (e.g. "

”) and the correct recognition result 1728 is shown in the candidate display area 806 .

FIG. 27E shows the user entering the initial portion of the delete input, eg, by performing a light touch 1730 on the delete button 1732. FIG. If the user maintains contact with the delete button 1732, the user can delete the current handwritten input character by character (or by recognition unit). This deletion is not performed for all handwritten inputs at the same time.

」に関する認識単位）が、手書き入力区域８０４内で同時に表示されている他の認識単位に対して、視覚的に強調表示される（例えば、境界線１７３４又は明るい背景などで強調表示される）。 In some embodiments, when the user's finger first touches the delete button 1732 on the touch-sensitive screen, the default (eg, left-to-right) writing direction is applied as shown in FIG. 17E. The last recognition unit (e.g. the character "

) is visually highlighted (eg, highlighted with a border 1734 or a light background, etc.) relative to other recognized units displayed simultaneously in the handwriting input area 804 .

In some embodiments, when the user device detects that the user has maintained contact 1730 on delete button 1732 for more than a threshold duration, the user device will , removes the highlighted unit of recognition (eg, in box 1734) from the handwriting input area 806. FIG. Furthermore, the user device also removes the recognition results shown in the candidate display area 608 to remove any output characters that were generated based on the removed recognition units, as shown in FIG. 17F. correct.

」に関する認識単位）が削除された後、ユーザが、削除ボタン１７３２上での接触１７３０を継続して維持する場合には、削除された認識単位に隣接する認識単位（例えば、文字「

」に関する認識単位）が、削除される次の認識単位になることを更に示す。図１７Ｆに示されるように、この残存する認識単位は、（例えば、ボックス１７３６内で）視覚的に強調表示されており、削除される準備が整っている。一部の実施形態では、認識単位の視覚的な強調表示は、ユーザが、削除ボタンとの接触を継続して維持する場合には、削除されることになる認識単位のプレビューを提供する。閾値持続時間に到達する前に、ユーザが、削除ボタンとの接触を中断する場合には、視覚的な強調表示は、最後の認識単位から除去され、その認識単位は削除されない。当業者には認識されるように、接触の持続時間は、認識単位が削除されるたびに、毎回リセットされる。更には、一部の実施形態では、接触の強度（例えば、ユーザが、タッチ感知スクリーンとの接触１７３０に加えている圧力）を任意選択的に使用して、現在強調表示されている認識単位を削除するユーザの意図を確認するための、閾値持続時間を調節する。図１７Ｆ及び図１７Ｇは、閾値持続時間に到達する前に、ユーザが、削除ボタン１７３２との接触１７３０を中断しており、文字「

」に関する認識単位が、手書き入力区域８０６内に保存されていることを示す。ユーザが、この認識単位に関する第１の認識結果（例えば、結果１７３８）を、（例えば、接触１７４０によって示されるように）選択すると、図１７Ｇ及び図１７Ｈに示されるように、その第１の認識結果１７３８内のテキストが、テキスト入力区域８０８内に入力される。 FIG. 17F shows the final recognition unit within the handwriting input area 806 (eg, the character "

) is deleted, if the user continues to maintain contact 1730 on delete button 1732, recognition units adjacent to the deleted recognition unit (e.g., the character "

) will be the next recognition unit to be deleted. As shown in FIG. 17F, this remaining recognition unit is visually highlighted (eg, in box 1736) and ready to be deleted. In some embodiments, the visual highlighting of the recognition unit provides a preview of the recognition unit that will be deleted if the user continues to maintain contact with the delete button. If the user discontinues contact with the delete button before reaching the threshold duration, the visual highlighting is removed from the last recognition unit and the recognition unit is not deleted. As those skilled in the art will recognize, the contact duration is reset each time a recognition unit is deleted. Furthermore, in some embodiments, the strength of contact (e.g., the pressure the user is exerting on contact 1730 with the touch-sensitive screen) is optionally used to determine the currently highlighted unit of recognition. Adjust the threshold duration for checking user intent to delete. Figures 17F and 17G show that the user interrupted contact 1730 with the delete button 1732 before the threshold duration was reached and the characters "

” is stored in the handwriting input area 806 . When the user selects a first recognition result (eg, result 1738) for this recognition unit (eg, as indicated by contact 1740), that first recognition result is displayed as shown in FIGS. 17G and 17H. The text in result 1738 is entered in text entry area 808 .

FIGS. 18A and 18B are a flow diagram of an exemplary process 1800 by which a user device provides character-by-character deletion within multi-character handwritten input. In some embodiments, this deletion of handwriting is performed before recognized characters from the handwriting are verified and entered into the text entry area of the user interface. In some embodiments, deletion of characters within handwritten input proceeds according to the reverse spatial order of the recognition units identified from the handwritten input, independent of the temporal arrangement in which those recognition units are formed. is. 17A-17H show an exemplary process 1800, according to some embodiments.

」の最後に現れる文字「

」を除去する（１８１２）。 As shown in FIG. 18A, in an exemplary process 1800, a user device receives (1802) handwriting input from a user, which is input to a handwriting input area of a handwriting input interface (eg, area 804 in FIG. 17D). ), including multiple handwritten strokes. The user device identifies 1804 a plurality of recognition units from the plurality of handwritten strokes, each recognition unit including a corresponding subset of the plurality of handwritten strokes. For example, a first recognition unit includes strokes 1702 and 1718 and a second recognition unit includes strokes 1710 and 1726, as shown in FIG. 17D. The user device generates (1806) a multi-character recognition result (eg, result 1728 in FIG. 17D) that includes each character recognized from those multiple recognition units. In some embodiments, the user device displays its multi-character recognition results (eg, results 1728 in FIG. 17D) within the candidate display area of the handwriting input interface. In some embodiments, while the multi-character recognition results are displayed in the candidate display area, the user device prompts the user for a delete input (eg, a delete button), for example, as shown in FIG. 17E. Contact 1730 on 1732 is received (1810). In some embodiments, in response to receiving this delete input, the user device displays multiple From the character recognition result (e.g., result 1728), the last character (e.g., the spatial array "

character that appears at the end of "

' is removed (1812).

」）に対応する。 In some embodiments, when multiple handwriting strokes are provided by a user, the user device inserts the plurality of handwriting strokes into the handwriting input area of the handwriting input interface, for example, as shown in FIGS. 17A-17D. is rendered in real time (1814). In some embodiments, in response to receiving a delete input, the user device moves from the handwriting area (e.g., handwriting area 804 in FIG. 17E) to within the handwriting area formed by a plurality of recognition units. Remove 1816 the corresponding subset of the plurality of handwritten strokes that correspond to the last recognition unit (eg, the recognition unit that includes strokes 1726 and 1710) in the resulting spatial array. This last recognition unit is the last character (e.g., the character "

”).

In some embodiments, this last unit of recognition does not include the temporally last handwritten stroke of the plurality of handwritten strokes provided by the user (1818). For example, if the user provides stroke 1718 after providing strokes 1726 and 1710, the last recognition unit containing strokes 1726 and 1710 is still deleted first.

In some embodiments, in response to receiving the initial portion of the delete input, the user device replaces the last recognized unit with other recognized units identified within the handwriting area, for example, as shown in FIG. 17E. , is visually identified (1820). In some embodiments, the initial portion of the delete input is the initial touch detected 1822 on the delete button in the handwriting input interface, wherein the initial touch exceeds a predetermined threshold amount of time. If persisted, delete input is detected.

In some embodiments, the final recognition unit corresponds to handwritten Chinese characters. In some embodiments, the handwritten input is written in cursive typeface. In some embodiments, the handwritten input corresponds to a plurality of Chinese characters written in cursive typeface. In some embodiments, at least one of the handwritten strokes is separated into two adjacent recognition units of the plurality of recognition units. For example, in some cases, a user may use a long stroke that is written over multiple characters, and in such cases, the segmentation module of the handwriting input module optionally divides the long Separate strokes into several recognition units. If handwriting deletion is performed character by character (or recognition unit by recognition unit), only segments of this long stroke (eg, segments within the corresponding recognition unit) are deleted at a time.

In some embodiments, the delete input is a persistent touch 1824 on a delete button provided within the handwriting input interface, and removing the corresponding subset of the plurality of handwriting strokes is the final removing a subset of the handwritten strokes in the recognition unit from the handwritten input area stroke by stroke in the reverse temporal order in which the subset of handwritten strokes was provided by the user.

In some embodiments, the user device generates a partial recognition result (1826) that includes a subset of each recognized character from the plurality of recognition units, eg, as shown in FIGS. 17B and 17C. , each of the respective character subsets satisfies a predetermined confidence threshold. In some embodiments, the user device displays partial recognition results (e.g., result 1712 in FIG. 17B and result 1720 in FIG. 17C) in the candidate display area of the handwriting input interface along with multi-character recognition results. (eg, results 1714 and 1722) are simultaneously displayed (1828).

In some embodiments, the partial recognition result does not include at least the last character in the multi-character recognition result. In some embodiments, the partial recognition result does not include at least the first character in the multi-character recognition result. In some embodiments, the partial recognition result does not include at least the intermediate characters in the multi-character recognition result.

In some embodiments, the smallest unit of deletion is a radical, and the handwriting is written whenever the radical happens to be the last recognized unit in the handwriting that still remains in the handwriting area. One radical is deleted at a time.

As described herein, in some embodiments the user device provides both horizontal and vertical write modes. In some embodiments, the user device allows the user to enter text in one or both of a left-to-right writing direction and a right-to-left direction in a horizontal writing mode. . In some embodiments, the user device allows the user to enter text in one or both of a top-to-bottom writing direction and a bottom-to-top direction in a vertical writing mode. . In some embodiments, the user device displays various affordances (e.g., writing mode or writing direction buttons) on the user interface for invoking respective writing modes and/or writing directions for the current handwriting input. offer. In some embodiments, the text input direction within the text input area is by default the same as the handwriting input direction in the handwriting input direction. In some embodiments, the user device allows the user to manually set the input direction within the text input area and the writing direction within the handwriting input area. In some embodiments, the text display direction within the candidate display area is the same as the handwriting input direction within the handwriting input area by default. In some embodiments, the user device allows the user to manually set the text display direction within the text input area independently of the handwriting input direction within the handwriting input area. In some embodiments, the user device associates a writing mode and/or writing direction of the handwriting input interface with a corresponding device orientation such that changes in orientation of the device cause changes in writing mode and/or writing direction. triggered. In some embodiments, the change in writing direction automatically enters the highest ranking recognition result entry into the text entry area.

19A-19F illustrate exemplary user interfaces for user devices that provide both horizontal and vertical input modes.

FIG. 19A shows the user device in horizontal input mode. In some embodiments, the horizontal input mode is provided when the user device is in landscape orientation, as shown in FIG. 19A. In some embodiments, a horizontal input mode is optionally associated and provided when the device is operated in a portrait orientation. In different applications, the relationship between device orientation and write mode may differ.

In the horizontal input mode, the user can provide handwriting in a horizontal writing direction (eg, a left-to-right default writing direction or a right-to-left default writing direction). In horizontal input mode, the user device performs a division of handwriting input into one or more recognition units along this horizontal writing direction.

In some embodiments, the user device only allows input of a single line within the handwriting input area. In some embodiments, the user device allows multiple lines of input (eg, two lines of input) within the handwriting input area, as shown in FIG. 19A. In FIG. 19A, the user has provided multiple handwriting strokes on several lines within the handwriting input area 806 . Based on the arrangement of the handwritten strokes provided by the user and the relative locations of the handwritten strokes and the distance between the strokes, the user device determines that the user is inputting two lines of characters. do. After dividing this handwritten input into two separate lines, the device determines the recognition units within each line.

As shown in FIG. 19A, the user device has recognized each character for each recognition unit identified in the current handwriting input 1902 and has generated several recognition results 1904 and 1906 . As further shown in FIG. 19A, in some embodiments, if the output character (e.g., the letter "I") for a particular set of recognition units (e.g., recognition units formed by initial strokes) is low, , the user device optionally generates a partial recognition result (eg, result 1906) showing only the output characters with sufficient recognition confidence. It should be understood from this partial recognition result 1906 that, in some embodiments, the user can correct the first stroke or delete it individually in order for the recognition model to produce the correct recognition result. can be done. In this particular example, the first recognition unit 1904 indicates the desired recognition result for the first recognition unit, so editing the first recognition unit is not necessary.

In this example, as shown in Figures 19A, 19B, the user has rotated the device to a portrait orientation (eg, shown in Figure 19B). In response to this change in device orientation, the handwriting input interface changes from horizontal input mode to vertical input mode, as shown in FIG. 19B. In vertical input mode, the layout of handwriting input area 804, candidate display area 806, and text input area 808 may differ from that shown in horizontal input mode. The specific layout of the horizontal and vertical input modes can be varied to suit various device geometries and application needs. In some embodiments, rotation of the device orientation and change of input mode causes the user device to automatically display the top ranked result (eg, result 1904) as text input 1910 in text input area 808. input. The orientation and position of the cursor 1912 also reflect changes in input mode and writing direction.

In some embodiments, input mode changes are optionally triggered by the user touching a special input mode selection affordance 1908 . In some embodiments, this input mode selection affordance is a graphical user interface element that also indicates the current writing mode, current writing direction, and/or current paragraph direction. In some embodiments, the input mode selection affordance may cycle through all available input modes and writing directions provided by handwriting input interface 802 . As shown in FIG. 19A, affordance 1908 indicates that the current input mode is horizontal input mode, with left-to-right writing direction and top-to-bottom paragraph direction. In FIG. 19B, affordance 1908 indicates that the current input mode is vertical input mode, with a top-to-bottom writing direction and a right-to-left paragraph direction. Other combinations of writing direction and paragraph direction are possible according to various embodiments.

」に関する手書きストローク）を入力している。この手書き入力は、垂直の書き込み方向で書き込まれる。ユーザデバイスは、この垂直方向での手書き入力を、２つの認識単位へと分割し、垂直方向でレイアウトされた２つの認識文字をそれぞれが含む、２つの認識結果１９１６及び１９１８を表示する。 As shown in FIG. 19C, in vertical input mode, the user enters multiple new strokes 1914 (e.g., two Chinese characters "

) are being entered. The handwriting input is written in a vertical writing direction. The user device divides this vertical handwriting input into two recognition units and displays two recognition results 1916 and 1918, each containing two recognized characters laid out vertically.

19C and 19D illustrate that when a user selects a displayed recognition result (eg, result 1916), the selected recognition result is entered vertically into text entry area 808. FIG.

Figures 19E and 19F show the user entering additional lines of handwriting input 1920 in a vertical writing direction. These lines proceed from left to right, following the paragraph direction of traditional Chinese writing. In some embodiments, candidate display area 806 also shows recognition results (eg, results 1922 and 1924) in the same writing direction and paragraph direction as for the handwriting input area. In some embodiments, according to the primary language associated with the user device or the language of the soft keyboard installed on the user device (eg, Arabic, Chinese, Japanese, English, etc.), other writing directions and Paragraph direction can be provided by default.

19E and 19F also show that when the user selects a recognition result (eg, result 1922), the text of the selected recognition result is entered within text entry area 808. FIG. As shown in FIG. 19F, the text currently entered within the text entry area 808 is therefore text written in horizontal mode, with a left-to-right writing direction, and text written in a top-to-bottom mode. It contains both text written in vertical mode that has direction. The paragraph direction for horizontal text is top to bottom, while the paragraph direction for vertical text is right to left.

In some embodiments, the user device allows the user to independently establish a preferred writing direction, paragraph direction, for each of the handwriting input area 804, candidate display area 806, and text input area 808. . In some embodiments, the user device, for each of the handwriting input area 804, the candidate display area 806, and the text input area 808, allows the user to establish a preferred writing direction and paragraph direction associated with the respective device orientation. make it possible to

Figures 20A-20C are flow diagrams of an exemplary process 2000 for changing the text input direction and handwriting input direction of a user interface. Figures 19A-19F show a process 2000, according to some embodiments.

In some embodiments, the user device determines 2002 the orientation of the device. Device orientation, and changes in device orientation, can be detected by accelerometers and/or other orientation sensing elements within the user device. In some embodiments, the user device provides a handwriting input interface on the device in a horizontal input mode according to the device being in the first orientation (2004). Each line of handwriting input entered in this horizontal input mode is divided into one or more respective recognition units along the horizontal writing direction. In some embodiments, the device provides a handwriting input interface on the device in a vertical input mode according to the device being in the second orientation (2006). Each line of handwriting input entered in this vertical input mode is divided into one or more respective recognition units along the vertical writing direction.

In some embodiments, while operating in horizontal input mode (2008), the device detects a change in orientation of the device from a first orientation to a second orientation (2010). In some embodiments, in response to this change in orientation of the device, the device switches from horizontal input mode to vertical input mode (2012). This is shown, for example, in Figures 19A and 19B. In some embodiments, while operating (2014) in vertical input mode, the user device detects (2016) a change in orientation of the device from the second orientation to the first orientation. In some embodiments, in response to this change in device orientation, the user device switches 2018 from vertical input mode to horizontal input mode. In some embodiments, the association between device orientation and input mode may be reversed from that described above.

In some embodiments, while operating (2020) in the horizontal input mode, the user device receives (2022) a first multi-word handwritten input from the user. In response to this first multi-word handwritten input, the user device presents (2024) first multi-word recognition results in a candidate display area of the handwriting input interface according to a horizontal writing direction. This is shown, for example, in FIG. 19A. In some embodiments, while operating (2026) in the vertical input mode, the user device receives (2028) a second multi-word handwritten input from the user. In response to this second multi-word handwritten input, the user device presents (2030) second multi-word recognition results according to a vertical writing direction within the candidate display area. This is shown, for example, in Figures 19C and 19E.

In some embodiments, the user device receives (2032) a first user input selecting a first multi-word recognition result, for example as shown in FIGS. is implicitly implemented using input to change input direction (eg, device rotation, or affordance 1908 selection). The user device receives a second user input selecting a second multi-word recognition result (2034), eg, as shown in FIG. 19C or FIG. 19E. The user device concurrently displays (2036) corresponding text of the first multi-word recognition result and the second multi-word recognition result in a text entry area of the handwriting input interface; The corresponding text of the recognition result is displayed according to a horizontal writing direction and the corresponding text of the second multi-word recognition result is displayed according to a vertical writing direction. This is shown, for example, in text entry area 808 in FIG. 19F.

In some embodiments, the handwriting input area accepts multi-line handwriting input in a horizontal writing direction and has a default top-to-bottom paragraph direction. In some embodiments, this horizontal writing direction is left to right. In some embodiments, this horizontal writing direction is right to left. In some embodiments, the handwriting input area accepts multi-line handwriting input in a vertical writing direction and has a default left-to-right paragraph direction. In some embodiments, the handwriting input area accepts multi-line handwriting input in a vertical writing direction and has a default right-to-left paragraph direction. In some embodiments, this vertical writing direction is from top to bottom. In some embodiments, the first orientation defaults to landscape display orientation and the second orientation defaults to portrait display orientation. In some embodiments, the user device provides corresponding affordances within the handwriting input interface to manually switch between horizontal and vertical input modes regardless of device orientation. In some embodiments, the user device provides corresponding affordances within the handwriting input interface to manually switch between two alternative writing directions. In some embodiments, the user device provides corresponding affordances within the handwriting input interface for manually switching between two alternative paragraph orientations. In some embodiments, this affordance is a toggle button that, when invoked one or more times in succession, rotates through each available combination of input direction and paragraph direction.

In some embodiments, the user device receives 2038 handwritten input from the user. The handwriting input includes a plurality of handwriting strokes provided within the handwriting input area of the handwriting input interface. In response to this handwriting input, the user device displays (2040) one or more recognition results in the candidate display area of the handwriting input interface. While the one or more recognition results are displayed in the candidate display area, the user device detects user input to switch from the current handwriting input mode to an alternate handwriting input mode (2042). . In response to this user input (2044), the user device switches (2046) from the current handwriting input mode to an alternate handwriting input mode. In some embodiments, the user device clears (2048) the handwriting input from within the handwriting input area. In some embodiments, the user device automatically enters the top ranked recognition result among the one or more recognition results displayed in the candidate display area into the text input area of the handwriting input interface. (2050). This is shown, for example, in Figures 19A and 19B, where the current handwriting input mode is the horizontal input mode and the alternate handwriting input mode is the vertical input mode. In some embodiments, the current handwriting input mode is vertical input mode and the alternate handwriting input mode is horizontal input mode. In some embodiments, the current handwriting input mode and the alternate handwriting input mode are modes in which any two different handwriting input directions or paragraph directions are provided. In some embodiments, this user input is a rotation of the device from its current orientation to a different orientation (2052). In some embodiments, this user input is a call to an affordance to manually switch the current handwriting input mode to an alternate handwriting input mode.

As described herein, a handwriting input module allows a user to enter handwritten strokes and/or characters in any temporal order. Therefore, deleting individual handwritten characters within a multi-character handwritten input, and rewriting the same or different handwritten characters in the same place as the deleted characters, will allow the user to rewrite the entire handwritten input. This is advantageous because it helps correct long handwritten entries without having to delete the .

FIGS. 20A-20H illustrate an exemplary user interface for visually highlighting and/or deleting recognition units identified within multiple handwriting strokes currently accumulated within a handwriting input area. Indicates an interface. Allowing the user to individually select, view, and delete any one of the recognition units identified within the inputs is a multi-character handwriting input, and also It is particularly useful where multi-line handwritten input is allowed by the user device. By allowing the user to delete certain recognition units at the beginning or middle of the handwritten input, without requiring the user to delete all recognition units that are located after the undesirable recognition units. , allowing the user to implement corrections to long entries.

As shown in FIGS. 21A-21C, a user has provided multiple handwriting strokes (eg, strokes 2102, 2104, and 2106) within handwriting input area 804 of handwriting input user interface 802. FIG. As the user continues to provide additional strokes in the handwriting area 804, the user device updates the recognition units identified from the handwriting input currently accumulated in the handwriting area 804, and updates the recognition units. Correct the recognition result according to the output character recognized from the recognized recognition unit. As shown in FIG. 20C, the user device identifies two recognition units from the current handwriting input and presents three recognition results (eg, 2108, 2010, and 2112) each containing two Chinese characters. is doing.

In this example, after the user writes two handwritten characters, the user writes the first recognition unit incorrectly, resulting in the desired recognition result being specified by the user device within the candidate display area. Understand that you are not presenting.

In some embodiments, when the user provides a tap gesture on the touch-sensitive display (e.g., a touch followed by an immediate lift-off in place), the user device applies that tap gesture within the handwriting area. Interpret as input to trigger visual highlighting of the currently identified individual recognition unit. In some embodiments, another predefined gesture (e.g., a multi-finger wipe gesture over the handwriting area) is used to direct the user device to individual recognition units within the handwriting area 804. to highlight. A tap gesture may be preferred because it is relatively easy to distinguish from handwriting strokes (those that typically involve longer duration persistent contact and have movement of the contact within the handwriting input area 804). Multi-tap gestures may be preferred because they are relatively easy to distinguish from handwriting strokes (which typically involve a single touch within the handwriting input area 804). In some embodiments, the user device is controlled by the user (eg, via contact 2114) to visually highlight individual recognition units (eg, as indicated by boxes 2108 and 2110). An invokeable affordance 2112 is provided in the user interface. In some embodiments, this affordance is preferred when there is sufficient screen space to accommodate such affordance. In some embodiments, this affordance can be invoked multiple times in succession by the user, which causes the user device to visually highlight recognition units identified according to different split chains within the split bundle. Make it visible and, for that highlight, turn it off when all split strands are shown.

As shown in FIG. 21D, when the user is providing the necessary gesture to highlight individual recognition units within the handwriting input area 804, the user device Above, the respective delete affordances (eg, small delete buttons 2116 and 2118) are also displayed. 21E and 21F illustrate that when the user touches (eg, via contact 2120) the delete affordance of the respective recognition unit (eg, delete button 2116 for the first recognition unit in box 2118), the respective Indicates that the recognition unit (eg, in box 2118 ) is removed from the handwriting area 804 . In this particular embodiment, the recognition unit that is deleted is neither the temporally last input recognition unit nor the spatially last recognition unit along the writing direction. In other words, the user can delete any recognition unit regardless of when and where that recognition unit was provided within the handwriting input area. FIG. 21F shows that in response to deleting this first recognition unit within the handwriting input area, the user device also updates the recognition results displayed within the candidate display area 806 . As shown in FIG. 21F, the user device has also deleted candidate characters corresponding to the deleted recognition units from the recognition result. As a result, a new recognition result 2120 is shown in candidate display area 806 .

As shown in FIGS. 21G and 21H, after the first recognition unit has been removed from the handwriting input interface 804, the user may place multiple lines in the area previously occupied by this removed recognition unit. A new handwriting stroke 2122 is provided. The user device is subdividing the currently accumulated handwriting input within the handwriting input area 804 . Based on the recognition units identified from this handwriting input, the user device has regenerated recognition results (eg, results 2124 and 2126) in candidate display area 806. FIG. 21G and 21H illustrate that the user has selected (eg, via contact 2128) one of the recognition results (eg, result 2124) and the text of the selected recognition result is , indicates that it is entered in the text entry area 808 .

Figures 22A and 22B show that individual recognition units identified within the current handwriting input are visually presented and can be individually deleted regardless of the temporal order in which they are formed. 22 is a flow diagram for an exemplary process 2200. FIG. Figures 21A-21H illustrate a process 2200, according to some embodiments.

In exemplary process 2200, a user device receives 2202 handwritten input from a user. This handwritten input includes a plurality of handwritten strokes provided on a touch-sensitive surface coupled to the device. In some embodiments, the user device renders (2204) the plurality of handwriting strokes within a handwriting input area (eg, handwriting input area 804) of the handwriting input interface. In some embodiments, the user device divides (2206) the plurality of handwritten strokes into two or more recognition units, each recognition unit being a corresponding subset of the plurality of handwritten strokes. including.

In some embodiments, the user device receives (2208) an edit request from the user. In some embodiments, this edit request is a touch detected (2210) over a default affordance (eg, affordance 2112 in FIG. 21D) provided within the handwriting input interface. In some embodiments, this edit request is a tap gesture detected 2212 over a predefined area within the handwriting input interface. In some embodiments, this default zone exists within the handwriting input zone of the handwriting input interface. In some embodiments, this default area exists outside the handwriting input area of the handwriting input interface. In some embodiments, another predefined gesture (eg, cross gesture, horizontal swipe gesture, vertical swipe gesture, tilt swipe gesture) outside the handwriting input area can be used as an edit request. Gestures outside the handwriting area are provided outside the handwriting area and can be easily distinguished from the handwriting strokes.

In some embodiments, in response to this edit request, the user device visually identifies two or more recognition units within the handwriting input area using, for example, boxes 2108 and 2110 in FIG. 21D. (2214). In some embodiments, visually identifying the two or more recognition units further includes highlighting respective boundaries between the two or more recognition units within the handwriting input area (2216 ). Various embodiments may use various methods of visually identifying identified recognition units within the current handwriting input.

In some embodiments, the user device provides means for individually deleting each of the two or more recognition units from the handwriting input area (2218). In some embodiments, the means for individually deleting each of the two or more recognition units is located proximal to each recognition unit, eg, as indicated by delete buttons 2116 and 2118 in FIG. 21D. It is the respective delete button that is displayed. In some embodiments, the means for individually deleting each of the two or more recognition units is means for detecting a predefined delete gesture input over each recognition unit. In some embodiments, the user device does not visually display individual deletion affordances over the highlighted units of recognition. Instead, some embodiments allow the user to use a delete gesture to delete the respective recognition unit under the delete gesture. In some embodiments, when the user device is displaying recognition units in a visually highlighted manner, the user device does not accept additional handwriting strokes within the handwriting input area. Instead, a default gesture or any gesture detected over a visually highlighted recognition unit causes the user device to remove that recognition unit from the handwriting area and display it in the candidate display area. to correct the recognition result obtained. In some embodiments, the tap gesture causes the user device to visually highlight individual recognition units identified within the handwriting recognition area, and then the user uses the delete button to In the opposite direction, individual recognition units can be deleted one by one.

In some embodiments, the user device provides a delete input to individually delete the first of the two or more recognition units from the handwriting input area, eg, as shown in FIG. 21E. , from the user through the means provided (2224). In response to this delete input, the user device removes (2226) the corresponding subset of handwritten strokes in the first recognition unit from the handwritten input area, eg, as shown in FIG. 21F. In some embodiments, the first recognition unit is the spatially first recognition unit within the two or more recognition units. In some embodiments, the first recognition unit is a spatially intermediate recognition unit among two or more recognition units, eg, as shown in FIGS. 21E and 21F. In some embodiments, the first recognition unit is the spatially last recognition unit among the two or more recognition units.

」及び「

」に関する）認識単位が強調表示され、第２のタップジェスチャに応じて、３つの（例えば、それぞれ、文字「

」、「

」、及び「

」に関する）認識単位が強調表示される。一部の実施形態では、第３のタップジェスチャに応じて、視覚的な強調表示が、任意選択的に、全ての認識単位から除去され、手書き入力区域は、通常状態に戻り、更なるストロークを受け入れる準備が整う。一部の実施形態では、ユーザデバイスは、手書き入力区域内に現在表されている、対応の認識単位のセットのそれぞれを、個別に削除する手段を提供する（２２３４）。一部の実施形態では、この手段は、強調表示された各認識単位に関する、個別の削除ボタンである。一部の実施形態では、この手段は、強調表示された各認識単位の上での、既定の削除ジェスチャを検出し、及び、その既定の削除ジェスチャの下の強調表示された認識単位を、削除する機能を呼び出す手段である。 In some embodiments, the user device generates 2228 a segmentation bundle from the plurality of handwritten strokes, the segmentation bundle representing a set of corresponding recognition units each identified from the plurality of handwriting strokes. contains a plurality of alternative split strands. For example, FIG. 21G shows recognition results 2024 and 2026, with recognition result 2024 generated from one split strand having two recognition units and recognition result 2026 generated from another split strand having three recognition units. be done. In some embodiments, the user device receives (2230) two or more sequential edit requests from the user. For example, those two or more consecutive edit requests can be several consecutive taps on affordance 2112 in FIG. 21G. In some embodiments, in response to each of the two or more sequential edit requests, the user device recognizes a corresponding recognition from a different one of the plurality of alternative split chains within the handwriting input area. A set of units is visually identified (2232). For example, in response to the first tap gesture, within handwriting input area 804, two (eg, the characters "

"as well as"

) is highlighted and, in response to a second tap gesture, three (eg, each of the letters "

", "

",as well as"

) is highlighted. In some embodiments, in response to the third tap gesture, the visual highlighting is optionally removed from all recognition units, the handwriting area returns to normal, and further strokes are performed. ready to receive. In some embodiments, the user device provides means for individually deleting each set of corresponding recognition units currently represented in the handwriting input area (2234). In some embodiments, this means is a separate delete button for each highlighted recognition unit. In some embodiments, the means detects a default delete gesture over each highlighted recognition unit and deletes the highlighted recognition unit below the default delete gesture. It is a means of calling functions that

As described herein, in some embodiments the user device provides a continuous input mode within the handwriting input area. Since the area of the handwriting input area is limited on portable user devices, user-provided handwriting may be cached to allow the user to free up screen space without committing to previously provided handwriting. It may be desirable to provide a method for enabling reuse. In some embodiments, the user device gradually transitions the input area a certain amount (e.g., one recognition unit at a time) when the user is getting close enough to the end of the handwritten input area. , provides a scrolling handwriting area. In some embodiments, migrating existing recognition units within the handwriting area can interfere with the user's writing process and possibly prevent correct segmentation of recognition units, so the recognition units can be moved dynamically. It may be advantageous to reclaim the previously used area of the input area without migration. In some embodiments, if the user reuses an area occupied by handwriting that has not yet been entered within the text input area, the top recognition result for that handwriting input area will be within the text input area. Automatically entered, the user can continue to provide new handwriting input without explicitly selecting its top-ranked recognition result.

Some conventional systems allow the user to write over existing handwriting that is still shown within the handwriting area. Such systems use temporal information to determine whether a new stroke is part of a previous recognition unit or a new recognition unit. Systems that rely on such temporal information impose strict requirements on the speed and urgency with which a user provides handwritten input, which many users find difficult to meet. . Furthermore, the visual rendering of the handwritten input can be cluttered and difficult for the user to decipher. Therefore, the writing process can be frustrating and confusing for the user, resulting in a poor user experience.

To indicate when a user can reuse an area occupied by a previously written recognition unit to continue writing within that handwriting input area, as described herein , a fading process is used. In some embodiments, this fading process gradually reduces the visibility of each recognition unit that has been presented within the handwriting input area for a threshold amount of time over which new strokes are written. Also, existing text does not visually conflict with the new stroke. In some embodiments, writing over a faded recognition unit allows the user to pause writing and rewrite that recognition unit without needing to explicitly provide a selection input regarding the top-ranking recognition result. automatically enter the top-ranked recognition result for the text input area. This implicit and automatic confirmation of top-ranking recognition results increases the efficiency and speed of input in handwriting interfaces and imposes a cognitive burden on the user to maintain the current text composition train of thought. reduce In some embodiments, writing over a faded recognition unit does not cause automatic selection of the highest ranked search result. Instead, the faded recognition unit is cached in the handwriting stack and combined with the new handwriting as the current handwriting. The user can view the recognition results generated based on all of those recognition units accumulated in the handwriting stack before making a selection.

FIGS. 23A-23J illustrate that recognition units provided within different regions of a handwriting input area gradually fade out from their corresponding regions after, for example, a predetermined amount of time, and this fade-out occurs within a particular region. After it occurs, the user is shown an exemplary user interface and process by which he can provide new handwriting strokes within that region.

As shown in FIG. 23A, the user has provided a plurality of handwritten strokes 2302 (eg, three handwritten strokes for the capital letter "I") within the handwriting input area 804 . A handwritten stroke 2302 is identified by the user device as one recognition unit. In some embodiments, the handwriting currently shown in the handwriting area 804 is cached in a first layer in the user device's handwriting stack. Several recognition results generated based on this identified recognition unit are provided in candidate display area 804 .

FIG. 23B shows that as the user continues to write one or more strokes 2302 to the right of stroke 2304, the handwritten strokes 2302 within the first recognition unit gradually begin to fade out within the handwritten input area 804. indicate that In some embodiments, an animation is displayed that mimics the gradual fading or disappearance of the visual rendering of this first recognition unit. For example, this animation can create the visual effect of ink evaporating from a whiteboard. In some embodiments, the fading of recognition units is not uniform across recognition units. In some embodiments, the fading of a recognition unit increases over time until the recognition unit becomes completely invisible within the handwritten area. However, even if a recognition unit is no longer visible within the handwriting area 804, in some embodiments, the invisible recognition unit is maintained at the top of the handwriting stack and a The generated recognition results are continuously displayed in the candidate display area. In some embodiments, faded recognition units are not completely removed from view until new handwriting is written over them.

In some embodiments, the user device allows new handwriting input to be provided over the area occupied by the faded recognition units as soon as the fading animation begins. In some embodiments, the user device will only post new handwriting after the fading has progressed to a certain stage (e.g., to the fadingest level, or until recognition is completely invisible in that area). Allows input to be provided over the area occupied by the faded recognition unit.

FIG. 23C shows that the first recognition unit (i.e., stroke 2302) has completed its fading process (e.g., ink color has stabilized at a very faint level or has become invisible). ). The user device identifies additional recognition units (e.g., recognition units for handwritten characters "a" and "m") from additional handwritten strokes provided by the user and presents recognition results in candidate display area 804. is updated.

22D-22F show that over time, the user provides multiple additional handwriting strokes (eg, 2304 and 2306) within handwriting input area 804. FIG. At the same time, the previously identified recognition unit gradually fades out of the handwriting input area 804 . In some embodiments, each recognition unit takes a predetermined amount of time to initiate its own fading process after the recognition unit is identified. In some embodiments, the fading process for each recognition unit does not begin until the user begins entering a second recognition unit downstream of that recognition unit. As shown in FIGS. 23B-23F , when handwriting input is provided in cursive, a single stroke (eg, stroke 2304 or stroke 2306) can be divided into multiple recognition units (eg, recognition unit for each handwritten character in the word "am" or "back").

FIG. 22G shows that even after the recognition unit has begun its fading process, the user is still able to follow the default playback input, e.g. indicates that the recognition unit can be restored to its non-faded state by a tap gesture. When the recognition unit is played, its appearance returns to normal visibility level. In some embodiments, the playback of faded recognition units is performed character-by-character in the opposite direction of writing within the handwriting input area 804 . In some embodiments, the playback of faded recognition units is performed word by word within the handwriting input area 804 . As shown in FIG. 23G, the recognition unit corresponding to the word "back" has been reproduced from a fully faded state to a fully unfaded state. In some embodiments, the clock for starting the fading process is reset for each recognition unit as it is played to a non-faded state.

FIG. 22H shows that a sustained touch on the delete button causes the last recognition unit in the default writing direction (e.g., the recognition unit for the letter 'k' in the word 'back') to be deleted from the handwriting input area 804. indicates that If this deletion input is continuously maintained, more recognition units (e.g. recognition units for the letters 'c', 'a', 'b' in the word 'back') will appear in the opposite direction of writing, deleted one by one. In some embodiments, the deletion of recognition units is word-by-word, and all characters of the handwritten word "back" that are deleted from the handwritten input area 804 are removed at the same time. Figure 22H also shows that when contact 2308 is maintained on the delete button 2310 after deletion of the recognition unit for the letter "b" in the handwritten word "back", the previously faded recognition unit "m" It also shows that it is reproduced in the same way.

FIG. 23I shows that if the deletion input is stopped before the deletion of the reproduced recognition unit 'm' in the handwritten word 'am' is performed, the reproduced recognition unit gradually reappears. Indicates to fade. In some embodiments, the state of each recognition unit (eg, selected from a set of one or more fade and non-fade states) is maintained and updated in a handwriting stack.

FIG. 23J shows the user providing one or more strokes 2312 over the area occupied by faded recognition units (eg, recognition units for the letter “I”) within the handwriting input area. , in some embodiments, the text of the highest-ranking recognition result (eg, result 2314) for the handwriting input performed before stroke 2312 is displayed in text input area 808, as shown in FIGS. indicates that it will be entered automatically. As shown in FIG. 23J, the text “I am” is no longer shown as being provisional, but instead has been fixed within the text entry area 808 . In some embodiments, once text input is performed for fully faded handwriting or partially faded handwriting, the handwriting is removed from the handwriting stack. The newly entered stroke (eg, stroke 2312) becomes the current entry in the handwriting stack.

As shown in FIG. 23J, the text “I am” is no longer shown as being provisional, but instead has been fixed within the text entry area 808 . In some embodiments, once text input is performed for fully faded handwriting or partially faded handwriting, the handwriting is removed from the handwriting stack. The newly entered stroke (eg, stroke 2312) becomes the current entry in the handwriting stack.

In some embodiments, if stroke 2312 is provided over an area in the handwriting input area that was occupied by faded recognition units (eg, recognition units for the letter "I"), then stroke 2312 is preceded by The text of the highest-ranking recognition result (eg, result 2314) for the handwriting input performed in . Instead, the current handwriting (both faded and non-faded) within the handwriting area 804 is erased and cached in the handwriting stack. The new stroke 2312 is appended to the cached handwriting in its handwriting stack. The user device determines recognition results based on the totality of handwriting input currently accumulated in the handwriting input stack. Those recognition results are displayed in the candidate display area. In other words, even though only a portion of the currently accumulated handwriting is shown in the handwriting area 804, the recognition results are cached in the handwriting stack (the visible and no longer generated based on the entire handwriting input (both non-visible portions).

FIG. 23K shows that the user has entered a further stroke 2316 within the handwriting input area 804 and that stroke is fading over time. FIG. 23L shows text of top recognition result 2320 for faded strokes 2312 and 2316 entered into text entry area 808 by new stroke 2318 written over faded strokes 2312 and 2316. indicates that

In some embodiments, the user optionally provides handwriting input on multiple lines. In some embodiments, the same fading process can be used to clear the handwriting area for new handwriting input when multiple lines of input are enabled.

Figures 24A and 24B are a flow diagram of an exemplary process 2400 for providing a fading process within a handwriting input area of a handwriting input interface. Figures 23A-23K show a process 2400, according to some embodiments.

In some embodiments, the device receives (2402) a first handwritten input from a user. The first handwriting input includes a plurality of handwriting strokes, the plurality of handwriting strokes forming a plurality of recognition units distributed along respective writing directions associated with a handwriting input area of the handwriting input interface. Form. In some embodiments, the user device renders (2404) each of the plurality of handwriting strokes within the handwriting input area as those handwriting strokes are provided by the user.

In some embodiments, the user device initiates a respective fading process for each of the multiple recognition units after the recognition unit has been completely rendered (2406). In some embodiments, renderings of recognition units in the first handwritten input fade away during each fading process. This is illustrated in Figures 23A-23F in some embodiments.

In some embodiments, the user device is occupied by faded recognition units of the plurality of recognition units, for example, as shown in FIGS. 23I, 23J and 23K, 23L. A second handwritten input is received 2408 from the user over the area of the zone. In some embodiments, in response to receiving (2410) this second handwriting input, the user device renders (2412) the second handwriting input within the handwriting area, fading all The recognition unit is erased from the handwriting area (2414). In some embodiments, all recognition units that have been entered within the handwriting input area prior to the second handwriting input, regardless of whether or not the recognition unit has initiated its fading process. , is erased from the handwriting area. This is shown, for example, in Figures 23I, 23J and 23K, 23L.

In some embodiments, the user device generates one or more recognition results for the first handwritten input (2416). In some embodiments, the user device displays (2418) those one or more recognition results in the candidate display area of the handwriting input interface. In some embodiments, in response to receiving the second handwriting input, the user device places the top-ranking recognition results displayed in the candidate display area, without user selection, in the text input area of the handwriting input interface. is automatically entered (2420). This is shown, for example, in Figures 23I, 23J and 23K, 23L.

In some embodiments, the user device stores (2422) an input stack that includes the first handwriting input and the second handwriting input. In some embodiments, the user device comprises one or more multi-character sequences each including a corresponding spatial character array recognized from the concatenation of the first handwritten input and the second handwritten input. A recognition result is generated (2424). In some embodiments, the user device, while the rendering of the second handwriting replaces the rendering of the first handwriting within the handwriting input area, within the candidate display area of the handwriting input interface: The one or more multi-character recognition results are displayed (2426).

In some embodiments, the respective fading process for each recognition unit is initiated when a predetermined period of time has elapsed after that recognition unit has been completed by the user.

In some embodiments, the fading process for each recognition unit is initiated when the user begins entering strokes for the next recognition unit after that recognition unit.

In some embodiments, the end state of the respective fading process for each recognition unit is a default minimum visibility state for that recognition unit.

In some embodiments, the end state of the respective fading process for each recognition unit is a state with zero visibility for that recognition unit.

In some embodiments, after the last unit of recognition in the first handwritten input fades, the user device receives (2428) default playback input from the user. In response to receiving this default playback input, the user device restores (2430) its last recognition unit from the faded state to the non-fade state. This is shown, for example, in Figures 23F-23H. In some embodiments, this default playback input is the initial touch detected on the delete button provided within the handwriting input interface. In some embodiments, a sustained touch detected on the delete button deletes the last recognition unit from the handwriting input area and plays the penultimate recognition unit from faded to non-faded. This is shown, for example, in Figures 23G and 23H.

As described herein, the multiscript handwriting recognition model performs stroke order independent and stroke direction independent recognition of handwriting. In some embodiments, the recognition model is trained only on spatially derived features contained within flat images of written samples that correspond to different characters within the vocabulary of the handwriting recognition model. Since the images of the writing samples do not contain any temporal information related to the individual strokes contained within those images, the resulting recognition model is stroke order independent and stroke direction independent. is.

As explained above, stroke-order and stroke direction-independent handwriting recognition traditionally relies on information related to the temporal generation of characters (e.g., the temporal arrangement of strokes within those characters). offers a number of advantages over the recognition system of However, in real-time handwriting recognition scenarios, temporal information associated with individual strokes is available, and it may be beneficial to utilize this information to improve the recognition accuracy of handwriting recognition systems. Techniques for integrating temporally derived stroke distribution information into the extraction of spatial features of a handwriting recognition model are described below, but the use of this temporally derived stroke distribution information can be applied to handwriting recognition systems. does not compromise the stroke order and/or stroke direction independence of . Based on stroke distribution information associated with various characters, disambiguation between similar looking characters produced by distinct sets of strokes is enabled.

」に関しては、８つの（例えば、図２７で＃１～＃８と標識される）ストロークを使用して、この文字を書き上げることができる。この文字に関するストロークの配列及び方向は、その文字に関連付けられる、幾つかの独自の特徴を提供する。認識システムのストローク順序及びストローク方向非依存性を損なうことなく、ストローク順序及びストローク方向の情報を捕捉するための素朴な方法は、訓練サンプル内に、ストローク順序及びストローク方向の全ての可能な順列を、明示的に列挙することである。しかしながら、単に適度の複雑性の文字に関してさえ、この順列は、１０億を超える可能性に及び、実際に実装することは、不可能ではないにせよ、非現実的なものとなる。本明細書で説明されるように、ストローク生成の時系列態様（すなわち、時間的情報）を抽象化する、ストローク分布プロファイルが、各書き込みサンプルに関して生成される。それらの書き込みサンプルのストローク分布プロファイルは、時間的に導出される特徴のセットを抽出するように訓練され、その時間的に導出される特徴のセットが、その後、（例えば、入力ビットマップ画像からの）空間的に導出される特徴と組み合わされて、手書き認識システムのストローク順序及びストローク方向非依存性に影響を及ぼすことなく、認識精度を向上させる。 In some embodiments, when handwriting input is converted to an input image (eg, an input bitmap image) for a handwriting recognition model (eg, CNN), temporal information associated with individual strokes is lost. For example, the Chinese character "

, eight strokes (eg, labeled #1 through #8 in FIG. 27) can be used to write up this character. The arrangement and direction of strokes for this character provide some unique characteristics associated with that character. A naive method for capturing stroke order and stroke direction information without compromising the stroke order and stroke direction independence of the recognition system is to include all possible permutations of stroke order and direction in the training samples. , is to enumerate explicitly. However, even for characters of only modest complexity, this permutation runs into over a billion possibilities, making it impractical, if not impossible, to implement in practice. As described herein, a stroke distribution profile is generated for each writing sample that abstracts temporal aspects of stroke generation (ie, temporal information). The stroke distribution profile of those writing samples is trained to extract a set of temporally derived features, which are then transformed (e.g., ) in combination with spatially derived features to improve recognition accuracy without affecting the stroke order and stroke direction independence of the handwriting recognition system.

As described herein, temporal information associated with a single character is extracted by computing various pixel distributions that characterize each handwritten stroke. Every handwritten stroke of a character produces a deterministic pattern (or profile) when projected in a given direction. This pattern by itself may be insufficient to clearly recognize that stroke, but when combined with other similar patterns it is adequate to capture the specific characteristics inherent in this particular stroke. can be. This kind of stroke representation can also be integrated with spatial feature extraction (e.g., input image-based feature extraction in CNNs) to reduce the ambiguity between similar-looking characters in the repertoire of handwriting recognition models. Orthogonality information is provided that can be useful for removing gender.

25A and 25B are flow diagrams of an exemplary process 2500 for integrating temporally derived and spatially derived features of handwriting samples during training of a handwriting recognition model. , in which case the resulting recognition model remains stroke order and stroke direction independent. In some embodiments, example process 2500 runs on a server device that provides trained recognition models to user devices (eg, portable device 100). In some embodiments, the server device includes one or more processors and memory that includes instructions that, when executed by the one or more processors, perform process 2500.

In the exemplary process 2500, the device separately trains 2502 a set of spatially derived features and a set of temporally derived features of a handwriting recognition model, and the spatially derived A set of features is trained on a corpus of training images, each image of a handwriting sample for each character in the output character set, and a set of temporally derived features is trained on a corpus of stroke distribution profiles. , each stroke distribution profile numerically characterizes the spatial distribution of a plurality of strokes within the handwriting sample for each character of the output character set.

In some embodiments, separately training the set of spatially derived features further comprises training a convolutional neural network having an input layer, an output layer, and a plurality of convolutional layers (2504 ), the plurality of convolutional layers includes a first convolutional layer, a final convolutional layer, 0 or more intermediate convolutional layers between the first and last convolutional layers, and a hidden layer between the last convolutional layer and the output layer. Including layers. An exemplary convolutional network 2602 is shown in FIG. This exemplary convolutional network 2602 can be implemented in substantially the same manner as the convolutional network 602 shown in FIG. Convolutional network 2602 includes an input layer 2606, an output layer 2608, and a plurality of convolutional layers, including an initial convolutional layer 2610a, zero or more intermediate convolutional layers, and a final convolutional layer 2610n, and It includes a hidden layer 2614 between the final convolutional layer and the output layer 2608 . Convolutional network 2602 also includes a kernel layer 2616 and a subsampling layer 2612 according to the arrangement shown in FIG. The training of this convolutional network is based on images 2614 of written samples in the training corpus 2604 . Spatially derived features are obtained and respective weights associated with the various features are determined by minimizing the recognition error for training samples in the training corpus. Those same features and weights, after being trained, are used for recognition of new handwriting samples not present in the training corpus.

In some embodiments, separately training a set of temporally derived features provides multiple stroke distribution profiles to the statistical model for classifying each character of the output character set. Further comprising determining a plurality of temporally derived parameters and respective weights for the plurality of temporally derived parameters (2506). In some embodiments, a stroke distribution profile 2620 is derived from each writing sample in a training corpus 2622, as shown in FIG. Training corpus 2622 optionally includes the same writing samples as corpus 2604, but also includes temporal information associated with stroke production within each writing sample. The stroke distribution profile 2622 is provided to a statistical modeling process 2624 during which temporally derived features are extracted and respective weights for those various features are applied to a statistical modeling method (e.g. CNN, K-Nearest Neighbors, etc.) by minimizing the recognition or classification error. As shown in FIG. 26, the set of temporally derived features and their respective weights are transformed into a set of feature vectors (e.g., feature vector 2626 or feature vector 2628) and the respective Introduced within a layer. The resulting network thus contains spatially derived parameters and temporally derived parameters that are orthogonal to each other and collectively contribute to character recognition. .

In some embodiments, the device combines the set of spatially derived features and the set of temporally derived features in the handwriting recognition model (2508). In some embodiments, combining the set of spatially derived features and the set of temporally derived features in the handwriting recognition model is one of the convolutional layers of the convolutional neural network. or introducing 2510 a plurality of spatially derived parameters and a plurality of temporally derived parameters in the hidden layer. In some embodiments, multiple temporally derived parameters and respective weights for those multiple temporally derived parameters are used in a final convolutional layer of a convolutional neural network for handwriting recognition (e.g., FIG. 26). is introduced into the final convolutional layer 2610n) at . In some embodiments, multiple temporally derived parameters and respective weights for those multiple temporally derived parameters are used in a hidden layer of a convolutional neural network for handwriting recognition (e.g., is introduced into the hidden layer 2614) of the

In some embodiments, the device uses this handwriting recognition model to provide real-time handwriting recognition for the user's handwriting input (2512).

In some embodiments, the device generates (2514) a corpus of stroke distribution profiles from the plurality of writing samples. In some embodiments, each of the plurality of handwriting samples corresponds 2516 to a character in the output character set, and for each constituent stroke of the handwriting sample, a respective spatial representation when that stroke was written. Store information separately. In some embodiments, to generate a corpus of stroke distribution profiles, the device performs the following steps (2518).

For each of the plurality of handwriting samples (2520), the device identifies (2522) constituent strokes within the handwriting sample, and for each of the identified strokes of the handwriting sample, the device identifies each of the plurality of predetermined orientations. , where the occupancy is the ratio of the projected span in each stroke direction to the maximum projected span of the written sample, and the occupancy of the specified stroke of the handwritten sample is calculated 2524 . For each, the device also calculates (2526) a respective saturation factor for that stroke based on the ratio of the respective number of pixels in that stroke and the total number of pixels in that written sample. The user device then generates (2528) a feature vector for the handwriting sample as the stroke distribution profile of the writing sample, the feature vector representing the respective occupancies and Including the saturation of each, N is a predefined natural number. In some embodiments, N is less than the maximum number of strokes observed in any single written sample in the plurality of written samples.

In some embodiments, for each of the plurality of handwriting samples, the device sorts the occupancy of each of the identified strokes in each of the predetermined directions in descending order, and the N top ranked occupancies and only write samples with saturation are included in the feature vector for that write sample.

In some embodiments, the plurality of predetermined directions includes a horizontal direction, a vertical direction, a positive 45 degree direction, and a negative 45 degree direction of the written samples.

In some embodiments, using this handwriting recognition model, the device receives the user's handwriting input and, in response to receiving this user's handwriting input, provides real-time handwriting recognition for the user's handwriting input. to provide handwriting recognition output to the user substantially concurrently with receiving the handwriting input.

」を使用して、説明の目的のために、例示的実施形態が本明細書で説明される。一部の実施形態では、手書き文字の各入力画像は、任意選択的に、正方形へと正規化される。この正方形の、水平、垂直、斜め＋４５度、及び斜め－４５度上に投影される場合の、個別の各手書きストローク（例えば、ストローク＃１、＃２．．．及び＃３）のスパンが測定される。各ストロークＳｉのスパンは、４つの投影方向に関して、それぞれ、ｘスパン（ｉ）、ｙスパン（ｉ）、ｃスパン（ｉ）、及びｄスパン（ｉ）として記録される。更には、画像全体にわたって観測された最大スパンもまた、記録される。この文字の最大スパンは、４つの投影方向に関して、それぞれ、ｘスパン、ｙスパン、ｃスパン、及びｄスパンとして記録される。説明の目的のために、本明細書では、４つの投影方向が任意選択的に考察されているが、原理的には、任意の恣意的な投影のセットを、様々な実施形態で使用することができる。最大スパン（例えば、ｘスパン、ｙスパン、ｃスパン、及びｄスパンとして示される）、及び４つの投影方向での、文字「

」内のストロークのうちの１つ（例えば、ストローク＃４）のスパン（例えば、ｘスパン（４）、ｙスパン（４）、ｃスパン（４）、及びｄスパン（４）として示される）が、図２７に示される。 The character "

' is used to describe exemplary embodiments herein for purposes of explanation. In some embodiments, each input image of handwritten characters is optionally normalized to a square. Measure the span of each individual handwriting stroke (e.g., strokes #1, #2... and #3) when projected onto this square horizontally, vertically, diagonally +45 degrees, and diagonally -45 degrees. be done. The span of each stroke Si is recorded as x-span(i), y-span(i), c-span(i), and d-span(i) for the four projection directions, respectively. Additionally, the maximum span observed across the image is also recorded. The maximum span of this character is recorded as x-span, y-span, c-span, and d-span for the four projection directions, respectively. For purposes of explanation, four projection directions are arbitrarily considered herein, but in principle any arbitrary set of projections can be used in various embodiments. can be done. The letter "

' (e.g. stroke #4) (denoted as x-span(4), y-span(4), c-span(4), and d-span(4)) is , as shown in FIG.

In some embodiments, when the span is measured for all strokes 1-5 (where 5 is the number of distinct handwritten strokes associated with this input image), the span along each projection direction is , the respective occupancies are calculated. For example, the corresponding occupancy R _x (i) along the x-direction for stroke S _i is calculated as R _x (i)=x span(i)/x span. Similarly, the corresponding occupancies along other projection directions can be calculated, R _y (i)=y span(i)/y span, R _c (i)=c span(i)/c span, R _d (i)=dspan(i)/dspan.

In some embodiments, the occupancies of all strokes in each direction are sorted separately in descending order, so that a respective ranking of all strokes in that input image is given with respect to each projection direction , obtained in terms of their stroke occupancy in that direction. The ranking of strokes in each projection direction reflects the relative importance of each stroke along the associated projection direction. This relative importance is independent of the order and direction in which the strokes were produced within the writing sample. Therefore, this occupancy-based ranking is stroke order and stroke direction independent, temporally derived information.

In some embodiments, each stroke is given a relative weight that indicates the stroke's importance to the character as a whole. In some embodiments, this weight is measured by the ratio of the number of pixels in each stroke to the total number of pixels in the character. This ratio is called saturation, which is associated with each stroke.

In some embodiments, a feature vector can be created for each stroke based on the occupancy and saturation of each stroke. For each character, a set of feature vectors is created containing 5S number of features. This set of features is called the stroke distribution profile of the character.

In some embodiments, only a predetermined number of high-ranking strokes are used in building the stroke distribution profile for each character. In some embodiments, this default number of strokes is ten. Based on those top 10 strokes, 50 stroke-derived features are generated for each character. In some embodiments, these features are introduced in the final convolutional layers of the convolutional neural network or in subsequent hidden layers.

In some embodiments, during real-time recognition, input images for recognition units are provided to a handwriting recognition mode that has been trained on both spatially derived features and temporally derived features. The input image is processed through the layers of the handwriting recognition model shown in FIG. When processing this input image reaches a layer where a stroke distribution profile input is required (eg, the final convolutional layer or hidden layer), the stroke distribution profile for that recognition unit is introduced into that layer. This input image and stroke distribution profile processing continues until an output classification (eg, one or more candidate characters) is provided at the output layer 2608 . In some embodiments, stroke distribution profiles for all recognition units are computed and provided as input to the handwriting recognition model along with the input images for those recognition units. In some embodiments, the input image for a recognition unit is first passed through a handwriting recognition model (without the benefit of temporally trained features). If two or more similar looking candidate characters are identified with close recognition confidence values, then the stroke distribution profile of that recognition unit is trained on temporally derived features (e.g. , final convolutional layer, or hidden layer) into the handwriting recognition model. When the input image and stroke distribution profile of the recognition unit pass through the final layers of the handwriting recognition model, two or more similar looking candidate characters can be better distinguished by their stroke distribution profile differences. can. Therefore, using temporally derived information about how individual handwriting strokes form recognition units, recognition can be performed without compromising the stroke order and stroke direction independence of the handwriting recognition system. Improves accuracy.

The foregoing description, for purposes of explanation, has been described with reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching. These embodiments are provided in order to best explain the principles of the invention and its practical application, and thereby enable others skilled in the art to make various modifications as appropriate for the particular uses envisioned. They have been chosen and described in order to enable them to be used to best utilize the invention and various embodiments.

Claims (162)

A method of providing multi-script handwriting recognition, comprising:
A device having one or more processors and memory,
training a multi-script handwriting recognition model based on spatially derived features of a multi-script training corpus, each of the multi-script training corpora corresponding to characters of at least three non-overlapping scripts; training a multi-script handwriting recognition model including handwriting samples;
providing real-time handwriting recognition for a user's handwriting input using the multiscript handwriting recognition model trained on the spatially derived features of the multiscript training corpus;
A method, comprising: 2. The method of claim 1, wherein the spatially derived features of the multiscript training corpus are stroke order independent and stroke direction independent. 2. The method of claim 1, wherein the training of the multiscript handwriting recognition model is independent of temporal information associated with each stroke in the handwriting sample. 2. The method of claim 1, wherein the at least three non-overlapping scripts include Chinese characters, glyphs, and Latin scripts. 2. The method of claim 1, wherein the at least three non-overlapping scripts include Chinese script, Arabic script, and Latin script. 2. The method of claim 1, wherein the at least three non-overlapping scripts comprise non-overlapping scripts defined by the Unicode standard. training the multi-script handwriting recognition model;
providing the handwritten samples of the multiscript training corpus to a single convolutional neural network having a single input surface and a single output surface;
each of the spatially derived features of the handwriting sample and the spatially derived features for distinguishing characters of the at least three non-overlapping scripts represented in the multiscript training corpus; using the convolutional neural network to determine the weights of
2. The method of claim 1, further comprising: 2. The method of claim 1, wherein the multiscript handwriting recognition model has at least 30,000 output classes representing at least 30,000 characters across the at least three non-overlapping scripts. providing real-time handwriting recognition for a user's handwriting input;
further comprising providing the multi-script handwriting recognition model to a user device, wherein the user device receives a plurality of handwritten strokes from the user; based on the received multi-script handwriting recognition model, the plurality of 2. The method of claim 1, wherein handwriting recognition is performed locally for one or more recognition units identified from handwritten strokes. providing real-time handwriting recognition for a user's handwriting input;
continuously correcting one or more recognition results for the user's handwritten input in response to the user's continued additions or corrections to the handwritten input;
responsive to each correction of the one or more recognition results, displaying the respective corrected one or more recognition results to the user within a candidate display area of a handwriting input user interface;
2. The method of claim 1, further comprising: further comprising providing the multi-script handwriting recognition model to a plurality of devices where there is no overlap in input language, wherein the multi-script handwriting recognition model is associated with each of the user devices for handwriting recognition of different input languages. , in each of said plurality of devices. A method, characterized in that it comprises any combination of the features of claims 1-11. A non-transitory computer-readable medium having instructions stored thereon, the instructions being executed by one or more processors, causing the processors to:
training a multi-script handwriting recognition model based on spatially derived features of a multi-script training corpus, each of the multi-script training corpora corresponding to characters of at least three non-overlapping scripts; training a multi-script handwriting recognition model including handwriting samples;
providing real-time handwriting recognition for a user's handwriting input using the multiscript handwriting recognition model trained on the spatially derived features of the multiscript training corpus;
A non-transitory computer-readable medium characterized by causing it to perform operations including: A non-transitory computer readable medium having instructions stored thereon which, when executed by one or more processors, causes the processor to perform any of the methods of claims 1 to 11. A non-transitory computer-readable medium, characterized by: a system,
one or more processors;
a memory having stored instructions;
and when the instructions are executed by the one or more processors, causing the processors to:
training a multi-script handwriting recognition model based on spatially derived features of a multi-script training corpus, each of the multi-script training corpora corresponding to characters of at least three non-overlapping scripts; training a multi-script handwriting recognition model including handwriting samples;
providing real-time handwriting recognition for a user's handwriting input using the multiscript handwriting recognition model trained on the spatially derived features of the multiscript training corpus;
A system characterized by causing an operation to be performed comprising: a system,
one or more processors;
a memory having stored instructions;
and said instructions, when executed by said one or more processors, cause said processors to perform any of the methods in claims 1 to 11. A method of providing multi-script handwriting recognition, comprising:
A user device having one or more processors and memory,
receiving a multiscript handwriting recognition model, the multiscript recognition model trained on spatially derived features of a multiscript training corpus, the multiscript training corpus comprising at least three non-overlapping receiving a multi-script handwriting recognition model including respective handwriting samples corresponding to characters of the script;
receiving handwritten input from a user, said handwritten input comprising one or more handwritten strokes provided on a touch-sensitive surface coupled to said user device;
In response to receiving the handwriting input, the user outputs one or more handwriting recognition results based on the multiscript handwriting recognition model that has been trained on the spatially derived features of the multiscript training corpus. provided in real time to
A method, comprising: providing real-time handwriting recognition results to the user;
dividing the user's handwritten input into one or more recognition units, each recognition unit including one or more of the handwritten strokes provided by the user; dividing into units;
providing an image corresponding to each of the one or more recognition units as input to the multiscript handwriting recognition model;
at least a first output character from a first script and a second script different from the first script from the multi-script handwriting recognition model for at least one of the one or more recognition units; obtaining at least a second output character from
18. The method of claim 17, further comprising: providing real-time handwriting recognition results to the user;
19. The method of claim 18, further comprising displaying both the first output character and the second output character within a candidate display area of a handwriting input interface of the user device. providing real-time handwriting recognition results to the user;
the first output based on which one of the first script or the second script is a corresponding script used with a soft keyboard currently installed on the user device; 19. The method of claim 18, further comprising selectively displaying one of a character and said second output character. providing real-time handwriting recognition for a user's handwriting input;
continuously correcting one or more recognition results for the user's handwritten input in response to the user's continued additions or corrections to the handwritten input;
responsive to each correction of the one or more recognition results, displaying the respective corrected one or more recognition results to the user within a candidate display area of the handwriting input user interface;
18. The method of claim 17, further comprising: 18. The method of claim 17, wherein the at least three non-overlapping scripts include Chinese characters, glyphs, and Latin scripts. 18. The method of claim 17, wherein the at least three non-overlapping scripts include Chinese, Arabic and Latin scripts. wherein the multiscript handwriting recognition model is a single convolutional neural network having a single input surface and a single output surface, and of the at least three non-overlapping scripts represented in the multiscript training corpus; 18. A method according to claim 17, comprising spatially derived features and respective weights for said spatially derived features for distinguishing characters. 18. The method of claim 17, wherein the multiscript handwriting recognition model has at least 30,000 output classes representing at least 30,000 characters across at least 3 non-overlapping scripts. wherein the multiscript handwriting recognition model is configured to recognize characters based on respective input images of one or more recognition units identified in the handwriting input and used for recognition, respectively 18. The method of claim 17, wherein the spatially derived features of are independent of respective stroke order, stroke direction, and stroke continuity within the handwritten input. A method, characterized in that it comprises any combination of the features of claims 17-26. A non-transitory computer-readable medium having instructions stored thereon, the instructions being executed by one or more processors, causing the processors to:
receiving a multiscript handwriting recognition model, the multiscript recognition model trained on spatially derived features of a multiscript training corpus, the multiscript training corpus comprising at least three non-overlapping receiving a multi-script handwriting recognition model including respective handwriting samples corresponding to characters of the script;
receiving handwritten input from a user, said handwritten input comprising one or more handwritten strokes provided on a touch-sensitive surface coupled to said user device;
In response to receiving the handwriting input, the user outputs one or more handwriting recognition results based on the multiscript handwriting recognition model that has been trained on the spatially derived features of the multiscript training corpus. provided in real time to
A non-transitory computer-readable medium characterized by causing it to perform operations including: A non-transitory computer-readable medium having instructions stored thereon which, when executed by one or more processors, cause said processors to perform any of the methods of claims 17-26. A non-transitory computer-readable medium characterized by: a system,
one or more processors;
a memory having stored instructions;
and when the instructions are executed by the one or more processors, causing the processors to:
receiving a multiscript handwriting recognition model, the multiscript recognition model trained on spatially derived features of a multiscript training corpus, the multiscript training corpus comprising at least three non-overlapping receiving a multi-script handwriting recognition model including respective handwriting samples corresponding to characters of the script;
receiving handwritten input from a user, said handwritten input comprising one or more handwritten strokes provided on a touch-sensitive surface coupled to said user device;
In response to receiving the handwriting input, the user outputs one or more handwriting recognition results based on the multiscript handwriting recognition model that has been trained on the spatially derived features of the multiscript training corpus. provided in real time to
A system characterized by causing an action to be performed comprising: a system,
one or more processors;
a memory having stored instructions;
and said instructions, when executed by said one or more processors, cause said processors to perform any of the methods of claims 17-26. A method of providing real-time handwriting recognition, comprising:
A device having one or more processors and memory,
receiving a plurality of handwritten strokes from a user, the plurality of handwritten strokes corresponding to a single handwritten character;
generating an input image based on the plurality of handwritten strokes;
providing the input image to a handwriting recognition model to perform real-time recognition of the handwriting, wherein the handwriting recognition model provides stroke order independent handwriting recognition. and
displaying the same first output character in real-time as the plurality of handwriting strokes are received, regardless of the respective order of the plurality of handwriting strokes received from the user;
A method, comprising: wherein the handwriting recognition model provides stroke direction independent handwriting recognition and displaying the same first output character;
Further comprising, in response to receiving the plurality of handwriting strokes, displaying the same first output character for each of the plurality of handwriting strokes provided by the user, regardless of the respective stroke direction. 33. The method of claim 32, characterized by: the handwriting recognition model providing stroke-count independent handwriting recognition and displaying the same first output character;
In response to receiving the plurality of handwritten strokes, displaying the same first output character regardless of how many handwritten strokes are used to form successive strokes in the input image. 33. The method of claim 32, further comprising: 33. The method of claim 32, wherein stroke order independent handwriting recognition is performed independently of temporal information associated with individual strokes within the handwritten characters. receiving a second plurality of handwritten strokes from the user, the second plurality of handwritten strokes corresponding to a second handwritten character;
generating a second input image based on the second plurality of handwritten strokes; and directing the handwriting recognition model to the second input image to perform real-time recognition of the second handwritten character. and
displaying a second output character corresponding to the second plurality of handwritten strokes in real-time upon receiving the second plurality of handwritten strokes, wherein the first output character and the second output character; characters are simultaneously displayed in a spatial arrangement independent of the respective order of the first plurality of handwritten inputs and the second plurality of handwritten inputs provided by the user. displaying characters;
33. The method of claim 32, further comprising: wherein the spatial arrangement of the first output character and the second output character is aligned with the first plurality of handwriting strokes and the second plurality of handwriting strokes along a default writing direction of the handwriting input interface of the user device; 37. The method of claim 36, corresponding to a spatial distribution of strokes. The first handwritten character is provided by the user as part of a first handwritten sentence, the second handwritten character is provided by the user as part of a second handwritten sentence, and the second handwritten character is provided by the user as part of a second handwritten sentence. 37. The method of claim 36, wherein one handwritten sentence and said second handwritten sentence are simultaneously displayed within a handwriting input area of said user device. The second plurality of handwritten strokes is received later in time than the first plurality of handwritten strokes, and the second output character is along a default writing direction of a handwriting input interface of the user device. 37. The method of claim 36, preceding the first output character in a uniform spatial arrangement. wherein the second plurality of handwritten strokes spatially follow the first plurality of handwritten strokes along a default writing direction of a handwriting input interface of the user device, the second output character comprising: following the first output character in a spatial arrangement along the default writing direction, the method comprising:
receiving a third handwritten stroke from the user to correct the handwritten character, the third handwritten stroke comprising the first plurality of handwritten strokes and the second plurality of handwritten strokes; receiving a third handwriting stroke received later in time than
In response to receiving the third handwritten stroke, recognizing the same as the first plurality of handwritten strokes based on the relative proximity of the third handwritten stroke to the first plurality of handwritten strokes. assigning the third handwriting stroke to a unit;
generating a corrected input image based on the first plurality of handwritten strokes and the third handwritten stroke;
providing the corrected input image to the handwriting recognition model to perform real-time recognition of the corrected handwriting;
displaying a third output character corresponding to the corrected input image in response to receiving the third handwriting input, the third output character replacing the first output character. , displaying a third output character displayed concurrently with the second output character in the spatial arrangement along the default writing direction;
37. The method of claim 36, further comprising: receiving a delete input from the user while the third output character and the second output character are simultaneously displayed as recognition results in a candidate display area of the handwriting input interface;
responsive to the deletion input, deleting the second output character from the recognition result while maintaining the third output character within the recognition result;
41. The method of claim 40, further comprising: within the handwriting input area of the handwriting input interface, the first plurality of handwriting strokes, the second plurality of handwriting strokes, and the third handwriting strokes, each of the handwriting strokes being provided by the user; rendering in real-time when
deleting respective renderings of the second plurality of handwritten strokes from the handwritten input area while deleting respective renderings of the first plurality of handwritten strokes and the third handwritten strokes from the handwritten input area in response to receiving the delete input. , each rendering within the handwriting area;
42. The method of claim 41, further comprising: 33. The method of claim 32, wherein the handwritten characters are multi-stroke Chinese characters. 33. The method of claim 32, wherein the first plurality of handwritten strokes are provided in cursive typeface. 33. The method of claim 32, wherein the first plurality of handwritten strokes are provided in cursive typeface, and wherein the handwritten characters are multi-stroke Chinese characters. establishing respective default constraints on the set of acceptable dimensions for handwriting input;
dividing a plurality of currently accumulated handwritten strokes into a plurality of recognition units based on the respective predefined constraints, each of the recognition units generating a respective input image; dividing into a plurality of recognition units that are provided to the handwriting recognition model and recognized as corresponding output characters;
41. The method of claim 40, further comprising: After dividing the currently accumulated plurality of handwritten strokes into the plurality of recognition units, receiving further handwritten strokes from the user;
assigning the further handwritten stroke to each one of the plurality of recognition units based on the spatial position of the further handwritten stroke with respect to the plurality of recognition units;
47. The method of claim 46, further comprising: A method comprising any combination of the features of claims 32-47. A non-transitory computer-readable medium having instructions stored thereon, the instructions being executed by one or more processors, causing the processors to:
receiving a plurality of handwritten strokes from a user, the plurality of handwritten strokes corresponding to a single handwritten character;
generating an input image based on the plurality of handwritten strokes;
providing the input image to a handwriting recognition model to perform real-time recognition of the handwriting, wherein the handwriting recognition model provides stroke order independent handwriting recognition. and
displaying the same first output character in real-time as the plurality of handwriting strokes are received, regardless of the respective order of the plurality of handwriting strokes received from the user;
A non-transitory computer-readable medium characterized by causing it to perform operations including: A non-transitory computer-readable medium having instructions stored thereon which, when executed by one or more processors, causes the processor to perform any of the methods of claims 32-47. A non-transitory computer-readable medium characterized by: a system,
one or more processors;
and a memory having instructions stored thereon, wherein when the instructions are executed by the one or more processors, the processors:
receiving a plurality of handwritten strokes from a user, the plurality of handwritten strokes corresponding to a single handwritten character;
generating an input image based on the plurality of handwritten strokes; and providing the input image to a handwriting recognition model to perform real-time recognition of the handwritten characters, wherein the handwriting recognition model: providing the input image to provide stroke order independent handwriting recognition;
displaying the same first output character in real-time as the plurality of handwriting strokes are received, regardless of the respective order of the plurality of handwriting strokes received from the user;
A system characterized by causing an action to be performed comprising: a system,
one or more processors;
a memory having stored instructions;
and said instructions, when executed by said one or more processors, cause said processors to perform any of the methods of claims 32 to 47. 1. A method of providing real-time handwriting recognition, in a device having one or more processors and memory, comprising:
receiving handwriting input from a user, said handwriting input comprising one or more handwriting strokes provided within a handwriting input area of a handwriting input interface;
identifying a plurality of output characters for the handwritten input based on a handwriting recognition model;
classifying the plurality of output characters into two or more categories based on predefined categorization criteria;
displaying each output character in a first of the two or more categories in an initial view of a candidate display area of the handwriting input interface, wherein the initial view of the candidate display area. displays respective output characters provided concurrently with affordances for invoking an expanded view of the candidate display area;
receiving user input selecting the affordance for invoking the enhanced view;
responsive to the user input, in the expanded view of the candidate display area, each of the output characters in the first category and not previously displayed in the initial view of the candidate display area; and displaying each output character in at least a second one of the two or more categories;
A method, comprising: 54. The method of claim 53, wherein the predefined categorization criteria determine whether each character is a commonly used character or an infrequent character. each character in the first category is a character found in a dictionary of commonly used characters, and each character in the second category is a character found in a dictionary of infrequent characters 54. A method according to claim 53, characterized in that it is a character found in . 56. The method of claim 55, wherein the dictionary of commonly used characters and the dictionary of infrequent characters are dynamically adjusted based on usage history associated with the device. Method. identifying groups of characters from the plurality of output characters that are visually similar to each other according to a predefined similarity criterion;
selecting representative characters from the group of visually similar characters based on predefined selection criteria;
displaying the representative character in place of other characters in the group of visually similar characters in the initial view of the candidate display area;
54. The method of claim 53, further comprising: receiving a predefined zoom input from the user directed to the representative characters displayed in the initial view of the candidate display area;
an magnified view of the representative character and each magnified view of one or more other characters in the group of visually similar characters in response to receiving the predefined magnifying input; , displaying at the same time;
58. The method of claim 57, further comprising: 59. The method of claim 58, wherein the predefined magnifying input comprises a magnifying gesture detected over the representative character displayed within the candidate display area. 59. The method according to claim 58, wherein said predefined magnification input comprises a touch detected over said representative character displayed within said candidate display area and lasting longer than a predefined threshold time. described method. 58. The method of claim 57, wherein said predetermined selection criteria are based on the relative frequency of use of said characters within said group. 58. The method of claim 57, wherein the predefined selection criteria are based on preferred input languages associated with the device. A method comprising any combination of the features of claims 53-62. 1. A method of providing real-time handwriting recognition, in a device having one or more processors and memory, comprising:
receiving handwriting input from a user, said handwriting input comprising a plurality of handwriting strokes provided within a handwriting input area of a handwriting input interface;
recognizing a plurality of output characters from the handwritten input based on a handwriting recognition model, the output characters including at least a first glyph and at least a first character from a natural human language script; recognizing multiple output characters;
displaying a recognition result including the first glyph and the first character from the script in the natural human language in a candidate display area of the handwriting input interface;
A method, comprising: recognizing at least a first semantic unit from the handwritten input based on the handwriting recognition model, the first semantic unit having a respective semantic meaning in a respective human language; recognizing at least a first semantic unit comprising each letter, word, or phrase capable of conveying
identifying a second glyph associated with the first semantic unit recognized from the handwritten input;
displaying in the candidate display area of the handwriting input interface a second recognition result comprising at least the second glyph identified from the first semantic unit;
65. The method of claim 64, further comprising: displaying the second recognition result;
66. The method of claim 65, further comprising displaying the second recognition result concurrently with a third recognition result including at least the first semantic unit. receiving user input to select the first recognition result displayed in the candidate display area;
inputting text of the selected first recognition result into a text input area of the handwriting input interface in response to the user input, wherein the text comprises at least the first glyph and inputting a first recognition result text including the first character from the script of the natural human language;
65. The method of claim 64, further comprising: wherein the handwriting recognition model is trained on a multiscript training corpus containing writing samples corresponding to characters of at least three non-overlapping scripts, the three non-overlapping scripts being a set of glyphs, Chinese characters, and Latin scripts; 65. The method of claim 64, comprising: identifying a second semantic unit corresponding to the first glyph recognized from the handwritten input;
displaying in the candidate display area of the handwriting input interface a fourth recognition result comprising at least the second semantic unit identified from the first glyph;
65. The method of claim 64, further comprising: displaying the fourth recognition result,
70. The method of Claim 69, further comprising displaying the fourth recognition result concurrently with the first recognition result in the candidate display area. A method comprising any combination of the features of claims 64-70. A method of providing handwriting recognition, comprising:
A device having one or more processors and memory,
receiving handwriting input from a user including a plurality of handwriting strokes provided within a touch sensitive surface coupled to the device;
Rendering the plurality of handwriting strokes in real-time within a handwriting input area of a handwriting input interface;
receiving one of a pinch gesture input and a magnifying gesture input over the plurality of handwriting strokes;
Upon receiving pinch gesture input, generating a first recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as a single recognition unit;
Upon receiving a magnifying gesture input, generating a second recognition result based on the plurality of handwriting strokes by processing the plurality of handwriting strokes as two separate recognition units separated by the magnifying gesture input. When,
Upon generating one of each of the first recognition result and the second recognition result, displaying the generated recognition result within a candidate display area of the handwriting input interface;
A method, comprising: 73. The method of claim 72, wherein the pinch gesture input comprises two contacts on the touch-sensitive surface that converge toward each other within an area occupied by the plurality of handwritten strokes. 73. The method of claim 72, wherein the magnifying gesture input comprises two contacts on the touch-sensitive surface that are separated from each other within an area occupied by the plurality of handwritten strokes. identifying two adjacent recognition units from the plurality of handwritten strokes;
displaying in the candidate display area an initial recognition result including each recognized character from the two adjacent recognition units;
receiving the pinch gesture input while the initial recognition results are displayed in the candidate display area;
73. The method of claim 72, further comprising: 76. The method of claim 75, wherein displaying the first recognition result further comprises replacing the initial recognition result with the first recognition result within the candidate display area. . responsive to the pinch gesture input, further comprising re-rendering the plurality of handwriting strokes to reduce a distance between the two adjacent recognition units within the handwriting input area. 76. The method of Paragraph 75. identifying a single recognition unit from the plurality of handwritten strokes;
displaying in the candidate display area an initial recognition result including characters recognized from the single recognition unit;
receiving the magnifying gesture input while the initial recognition results are displayed within the candidate display area;
73. The method of claim 72, further comprising: 79. The method of claim 78, wherein displaying the second recognition result further comprises replacing the initial recognition result with the second recognition result within the candidate display area. . a first subset of strokes assigned to a first recognition unit and a second subset of handwritten strokes assigned to a second recognition unit within the handwriting input area in response to the magnifying gesture input; 80. The method of claim 79, further comprising re-rendering the plurality of handwritten strokes to increase distance. A method comprising any combination of the features of claims 72-80. A method of providing handwriting recognition, comprising:
receiving handwriting input from a user, said handwriting input comprising a plurality of handwriting strokes provided within a handwriting input area of a handwriting input interface;
identifying a plurality of recognition units from the plurality of handwritten strokes, each recognition unit comprising a corresponding subset of the plurality of handwritten strokes;
generating a multi-character recognition result including each character recognized from the plurality of recognition units;
displaying the multi-character recognition result in a candidate display area of the handwriting input interface;
receiving a delete input from the user while the multi-character recognition results are displayed in the candidate display area;
removing the last character from the multi-character recognition results displayed in the candidate display area in response to receiving the delete input;
A method, comprising: rendering the plurality of handwriting strokes in real-time within the handwriting input area of the handwriting input interface as the plurality of handwriting strokes are provided by the user;
The plurality of handwritten strokes corresponding to a last recognition unit in a spatial array formed by the plurality of recognition units from the handwriting input area in response to receiving the delete input. wherein the last recognition unit corresponds to the last character in the multi-character recognition result;
83. The method of claim 82, further comprising: 84. The method of claim 83, wherein the last recognition unit does not include the temporally last handwritten stroke of the plurality of handwritten strokes provided by the user. responsive to receiving an initial portion of the deletion input, visually distinguishing the last recognition unit from other recognition units identified within the handwritten input area; 84. The method of Paragraph 83. if the initial portion of the delete input is an initial touch detected on a delete button in the handwriting input interface, and the initial touch is sustained for more than a predetermined threshold amount of time; 86. The method of claim 85, wherein a delete input is detected. 84. The method of claim 83, wherein said last recognition unit corresponds to handwritten Chinese characters. 84. The method of claim 83, wherein the handwritten input is written in cursive typeface. 84. The method of claim 83, wherein the handwritten input corresponds to a plurality of Chinese characters written in cursive typeface. 84. The method of claim 83, wherein at least one of said handwritten strokes is separated into two adjacent recognition units of said plurality of recognition units. wherein the delete input is a persistent touch on a delete button provided within the handwriting input interface, and removing the corresponding subset of the plurality of handwriting strokes;
removing a subset of the handwritten strokes in the last recognition unit from the handwritten input area in reverse temporal order in which the subset of handwritten strokes were provided by the user, stroke by stroke. 84. The method of claim 83, characterized in that: generating a partial recognition result comprising subsets of the respective characters recognized from the plurality of recognition units, each of the subsets of the respective characters meeting a predetermined confidence threshold; generating a target recognition result;
displaying the partial recognition results simultaneously in the candidate display area of the handwriting input interface;
83. The method of claim 82, further comprising: 93. The method of claim 92, wherein the partial recognition result does not include at least the last character in a multi-character recognition result. 93. The method of claim 92, wherein the partial recognition result does not include at least the first character in a multi-character recognition result. 93. The method of claim 92, wherein the partial recognition result does not include at least intermediate characters in a multi-character recognition result. A method comprising any combination of the features of claims 82-95. A method of providing real-time handwriting recognition, comprising:
A device having one or more processors and memory,
determining an orientation of the device;
providing a handwriting input interface on the device in a horizontal input mode according to the device being in a first orientation, wherein each line of handwriting input entered in the horizontal input mode providing a horizontal input mode that is divided into one or more respective recognition units along the writing direction of the
providing the handwriting input interface on the device in a vertical input mode according to the device being in a second orientation, each line of handwriting input entered in the vertical input mode comprising: providing in a vertical input mode divided into one or more respective recognition units along the vertical writing direction;
A method, comprising: While operating in said horizontal input mode,
detecting a change in device orientation from the first orientation to the second orientation;
switching from the horizontal input mode to the vertical input mode in response to a change in orientation of the device;
98. The method of claim 97, further comprising: While operating in said vertical input mode,
detecting a change in device orientation from the second orientation to the first orientation;
switching from the vertical input mode to the horizontal input mode in response to a change in orientation of the device;
98. The method of claim 97, further comprising: While operating in said horizontal input mode,
receiving a first multi-word handwritten input from the user;
presenting recognition results of a first plurality of words according to the horizontal writing direction within a candidate display area of the handwriting input interface in response to handwriting input of the first plurality of words;
While operating in said vertical input mode,
receiving a second multi-word handwritten input from the user;
presenting recognition results of a second plurality of words according to the vertical writing direction within the candidate display area in response to handwritten input of the second plurality of words;
98. The method of claim 97, further comprising: receiving a first user input that selects a recognition result of the first plurality of words;
receiving a second user input that selects the second plurality of word recognition results;
simultaneously displaying corresponding text of the first plurality of word recognition results and the second plurality of word recognition results in a text entry area of the handwriting input interface; the corresponding text of the recognition result of is displayed according to the horizontal writing direction, and the corresponding text of the recognition result of the second plurality of words is displayed according to the vertical writing direction. displaying;
101. The method of claim 100, further comprising: 98. The method of claim 97, wherein the handwriting input area accepts multiple lines of handwriting input in the horizontal writing direction and has a default top-to-bottom paragraph direction. 98. The method of claim 97, wherein the horizontal writing direction is left to right. 98. The method of claim 97, wherein the horizontal writing direction is right to left. 98. The method of claim 97, wherein the handwriting input area accepts multiple lines of handwriting input in the vertical writing direction and has a default left-to-right paragraph direction. 98. The method of claim 97, wherein the handwriting input area accepts multiple lines of handwriting input in the vertical writing direction and has a default right-to-left paragraph direction. 98. The method of claim 97, wherein the vertical writing direction is top to bottom. 98. The method of claim 97, wherein the first orientation defaults to a landscape orientation and the second orientation defaults to a portrait orientation. 12. The claim further comprising providing a corresponding affordance within the handwriting input interface for manually switching between the horizontal input mode and the vertical input mode regardless of orientation of the device. 97. 98. The method of claim 97, further comprising providing a corresponding affordance within the handwriting input interface for manually switching between two alternative writing directions. 98. The method of claim 97, further comprising providing corresponding affordances within the handwriting input interface for manually switching between two alternative paragraph orientations. receiving handwriting input from a user, said handwriting input comprising a plurality of handwriting strokes provided within said handwriting input area of said handwriting input interface;
displaying one or more recognition results in a candidate display area of the handwriting input interface in response to the handwriting input;
detecting user input to switch from a current handwriting input mode to an alternate handwriting input mode while the one or more recognition results are displayed in the candidate display area;
In response to said user input,
switching from the current handwriting input mode to the alternate handwriting input mode;
erasing the handwriting input from within the handwriting input area;
automatically inputting, into a text input area of the handwriting input interface, a top ranked recognition result among the one or more recognition results displayed in the candidate display area;
98. The method of claim 97, further comprising: 113. The method of claim 112, wherein the user input is rotation of the device from its current orientation to a different orientation. 113. The method of claim 112, wherein the user input is an invocation of an affordance to manually switch the current handwriting input mode to the alternate handwriting input mode. A method comprising any combination of the features of claims 97-114. A method of providing real-time handwriting recognition, comprising:
A device having one or more processors and memory,
receiving handwriting input from a user, the handwriting input comprising a plurality of handwriting strokes provided on a touch-sensitive surface coupled to the device;
rendering the plurality of handwriting strokes within a handwriting input area of a handwriting input interface;
dividing the plurality of handwritten strokes into two or more recognition units, each recognition unit including a corresponding subset of the plurality of handwritten strokes. and
receiving an edit request from the user;
visually identifying the two or more recognition units within the handwriting input area in response to the edit request;
providing means for individually deleting each of the two or more recognition units from the handwriting input area;
A method, comprising: 116. Claim 116, wherein said means for individually deleting each of said two or more recognition units is a respective delete button displayed proximally of said each recognition unit. The method described in . The claim, wherein said means for individually deleting each of said two or more recognition units is means for detecting a predefined delete gesture input over said each recognition unit. 117. The method of Paragraph 116. wherein visually identifying the two or more recognition units further comprises highlighting respective boundaries between the two or more recognition units within the handwriting input area; 117. The method of claim 116. 117. The method of claim 116, wherein the edit request is a touch detected over a default affordance provided within the handwriting input interface. 117. The method of claim 116, wherein the edit request is a tap gesture detected over a predefined area within the handwriting input interface. 122. The method of claim 121, wherein the predefined zone exists within the handwriting input zone of the handwriting input interface. 122. The method of claim 121, wherein the predefined area exists outside the handwriting input area of the handwriting input interface. receiving a deletion input from the user through the provided means for individually deleting a first recognition unit of the two or more recognition units from the handwriting input area;
removing a corresponding subset of the handwritten strokes in the first recognition unit from the handwritten input area in response to the deletion input;
117. The method of claim 116, further comprising: 125. The method of claim 124, wherein said first recognition unit is the spatially first recognition unit within said two or more recognition units. 125. The method of claim 124, wherein said first recognition unit is a spatially intermediate recognition unit among said two or more recognition units. generating a segmentation bundle from the plurality of handwritten strokes, the segmentation bundle forming a plurality of alternative fragmentation chains each representing a set of corresponding recognition units identified from the plurality of handwritten strokes; generating a split lattice, including
receiving two or more sequential edit requests from the user;
Visually displaying the corresponding set of recognition units from a different one of the plurality of alternative split chains within the handwriting input area in response to each of the two or more sequential editing requests. identifying to
providing means for individually deleting each of said set of corresponding recognition units currently represented in said handwriting input area;
125. The method of claim 124, further comprising: A method comprising any combination of the features of claims 116 to 127. A method of providing real-time handwriting recognition, comprising:
A device having one or more processors and memory,
Receiving a first handwriting input from a user, the first handwriting input comprising a plurality of handwriting strokes, each of the plurality of handwriting strokes being associated with a handwriting input area of a handwriting input interface. receiving a first handwriting input forming a plurality of recognition units distributed along a writing direction of
rendering each of the plurality of handwriting strokes within the handwriting input area as the handwriting strokes are provided by the user;
initiating a respective fading process for each of the plurality of recognition units after the recognition unit has been completely rendered, wherein during the respective fading process, within the first handwritten input initiating a respective fading process in which the rendering of the recognition units of gradually fades;
receiving a second handwriting input from the user over an area of the handwriting area occupied by faded recognition units of the plurality of recognition units;
In response to receiving the second handwritten input,
rendering the second handwriting within the handwriting area;
erasing all faded recognition units from the handwriting input area;
A method, comprising: generating one or more recognition results for the first handwritten input;
displaying the one or more recognition results in a candidate display area of the handwriting input interface;
automatically inputting, without user selection, a top ranked recognition result displayed in the candidate display area into a text input area of the handwriting input interface in response to receiving the second handwriting input. When,
130. The method of claim 129, further comprising: storing an input stack comprising the first handwriting input and the second handwriting input;
generating one or more multi-character recognition results, each including a corresponding spatial character sequence recognized from the concatenation of the first handwritten input and the second handwritten input;
The one or more displaying recognition results of multiple characters;
130. The method of claim 129, further comprising: 130. The method of claim 129, wherein the respective fading process for each recognition unit is initiated when a predetermined period of time has elapsed after the recognition unit has been completed by the user. . 130. The method of claim 129, wherein the respective fading process for each recognition unit is initiated when, after the recognition unit, the user begins inputting strokes for the next recognition unit. the method of. 130. The method of claim 129, wherein the end state of said respective fading process for each recognition unit is a state with a predetermined minimum visibility for said recognition unit. 130. The method of claim 129, wherein the end state of the respective fading process for each recognition unit is a state with zero visibility for the recognition unit. receiving predefined playback input from the user after the last recognized unit in the first handwritten input fades;
reverting the last recognition unit from the faded state to a non-faded state in response to receiving the predefined playback input;
130. The method of claim 129, further comprising: 137. The method of claim 136, wherein the default playback input is an initial touch detected on a delete button provided within the handwriting input interface. A detected sustained touch on the delete button deletes the last recognition unit from the handwriting input area and reproduces the penultimate recognition unit from the faded state to the non-faded state. 137. The method of claim 136, wherein . A method comprising any combination of the features of claims 129-138. A method of providing handwriting recognition, comprising:
A device having one or more processors and memory,
separately training a set of spatially derived features and a set of temporally derived features of a handwriting recognition model, comprising:
the set of spatially derived features is trained on a corpus of training images, each image of a handwriting sample for each character of an output character set;
The set of temporally derived features is trained on a corpus of stroke distribution profiles, each stroke distribution profile representing the spatial distribution of a plurality of strokes within a handwriting sample for each character of the output character set. , characterize numerically,
training separately;
combining the set of spatially derived features and the set of temporally derived features in the handwriting recognition model;
using the handwriting recognition model to provide real-time handwriting recognition for a user's handwriting input;
A method, comprising: separately training the set of spatially derived features;
further comprising training a convolutional neural network having an input layer, an output layer, and a plurality of convolutional layers, wherein the plurality of convolutional layers are a first convolutional layer, a final convolutional layer, the first convolutional layer and the final convolutional layer; 141. The method of claim 140, comprising zero or more intermediate convolutional layers between layers and a hidden layer between said final convolutional layer and said output layer. separately training the set of temporally derived features;
a plurality of temporally derived parameters for providing the plurality of stroke distribution profiles to a statistical model to classify the respective characters of the output character set; and the plurality of temporally derived parameters. 142. The method of claim 141, further comprising determining respective weights for . combining the set of spatially derived features and the set of temporally derived features in the handwriting recognition model;
introducing the plurality of spatially derived parameters and the plurality of temporally derived parameters into one of the convolutional layers or the hidden layer of the convolutional neural network. 143. The method of claim 142, characterized by: wherein said plurality of temporally derived parameters and respective weights for said plurality of temporally derived parameters are introduced in said final convolutional layer of said convolutional neural network for handwriting recognition. 144. The method of claim 143. 143. CHARACTERIZED IN THAT said plurality of temporally derived parameters and respective weights for said plurality of temporally derived parameters are introduced within said hidden layer of said convolutional handwriting recognition. The method described in . further comprising generating a corpus of the stroke distribution profile from a plurality of writing samples;
each of the plurality of handwriting samples corresponding to a character in the output character set, and separately storing respective spatial information for each constituent stroke of the handwriting sample when the stroke was written;
generating a corpus of stroke distribution profiles;
For each of the plurality of handwriting samples,
identifying constituent strokes in the handwriting sample;
calculating, for each of the identified strokes of the handwriting sample, a respective occupancy along each of a plurality of predetermined directions, the occupancies comprising the projected span of each stroke direction and the be a ratio of the maximum projected span of the written samples, and
For each of the identified strokes of the handwritten sample, calculating a respective saturation ratio for each stroke based on a ratio of the respective number of pixels in each stroke to the total number of pixels in the written sample. and
generating a feature vector for the handwritten sample as the stroke distribution profile of the written sample, wherein the feature vector includes the respective occupancy of at least N strokes in the handwritten sample and the respective generating a feature vector containing the saturation ratio, where N is a predefined natural number;
141. The method of claim 140, further comprising: 147. The method of claim 146, wherein N is less than the maximum number of strokes observed in any single written sample in said plurality of written samples. For each of the plurality of handwriting samples,
sorting the respective occupancies of the identified strokes in each of the predetermined directions in descending order;
including only N top-ranked occupancy and saturation written samples in the feature vector of the written samples;
148. The method of claim 147, further comprising: 147. The method of claim 146, wherein the plurality of predetermined orientations include horizontal, vertical, positive 45 degree orientations, and negative 45 degree orientations of the written samples. using the handwriting recognition model to provide real-time handwriting recognition for a user's handwriting input;
receiving handwriting input of the user;
responsive to receiving the user's handwriting input, providing handwriting recognition output to the user substantially concurrently with the receiving of the handwriting input;
141. The method of claim 140, further comprising: 151. A method comprising any combination of the features of claims 140-150. A non-transitory computer-readable medium having instructions stored thereon, the instructions being executed by one or more processors, causing the processors to:
separately training a set of spatially derived features and a set of temporally derived features of a handwriting recognition model, comprising:
the set of spatially derived features is trained on a corpus of training images, each image of a handwriting sample for each character of an output character set;
The set of temporally derived features is trained on a corpus of stroke distribution profiles, each stroke distribution profile representing the spatial distribution of a plurality of strokes within a handwriting sample for each character of the output character set. , characterize numerically,
training separately;
combining the set of spatially derived features and the set of temporally derived features in the handwriting recognition model;
using the handwriting recognition model to provide real-time handwriting recognition for a user's handwriting input;
A non-transitory computer-readable medium characterized by causing it to perform operations including: 151. A non-transitory computer-readable medium having instructions stored thereon which, when executed by one or more processors, causes the processor to perform any of the methods of claims 140-150. A non-transitory computer-readable medium characterized by: a system,
one or more processors;
a memory having stored instructions;
and when the instructions are executed by the one or more processors, causing the processors to:
separately training a set of spatially derived features and a set of temporally derived features of a handwriting recognition model, comprising:
the set of spatially derived features is trained on a corpus of training images, each image of a handwriting sample for each character of an output character set;
The set of temporally derived features is trained on a corpus of stroke distribution profiles, each stroke distribution profile representing the spatial distribution of a plurality of strokes within a handwriting sample for each character of the output character set. , characterize numerically,
training separately;
combining the set of spatially derived features and the set of temporally derived features in the handwriting recognition model;
using the handwriting recognition model to provide real-time handwriting recognition for a user's handwriting input;
A system characterized by causing an action to be performed comprising: a system,
one or more processors;
a memory having stored instructions;
and said instructions, when executed by said one or more processors, cause said processors to perform any of the methods recited in claims 140-150. A method comprising any combination of the features of claims 1-150. 151. A non-transitory computer-readable medium having instructions stored thereon which, when executed by one or more processors, causes the processor to perform any of the methods of claims 1-150. A non-transitory computer-readable medium characterized by: a system,
one or more processors;
a memory having stored instructions;
and said instructions, when executed by said one or more processors, cause said processors to perform any of the methods recited in claims 1-150. an electronic device,
a display;
one or more processors;
memory;
one or more programs;
wherein said one or more programs are configured to be stored in said memory and executed by said one or more processors, said one or more programs comprising: An electronic device comprising instructions for performing any of the methods of . 151. A graphical user interface on an electronic device comprising a display, a memory and one or more processors for executing one or more programs stored in said memory, as claimed in any of claims 1 to 150. A graphical user interface, comprising a user interface displayed according to any of the methods of . an electronic device,
a display;
means for performing any of the methods of claims 1-150;
An electronic device comprising: An information processing apparatus for use within an electronic device having a display, comprising:
Information processing apparatus, characterized in that it comprises means for performing any of the methods of claims 1-150.

Images