JP7361156B2 - Managing real-time handwriting recognition - Google Patents

Description

This specification relates to providing handwriting input functionality on a computing device, and more particularly, to providing real-time, multi-script, stroke order independent, handwriting recognition and input on a computing device. Relating to providing functionality.

Handwriting input methods are an important alternative input method for computing devices equipped with touch-sensitive surfaces (eg, touch-sensitive display screens, or touch pads). Many users, especially those in some Asian or Arab countries, are accustomed to writing in cursive and may feel comfortable writing in plain handwriting as opposed to typing on a keyboard. can.

For certain logographic writing systems, such as Chinese Hanji and Japanese Kanji (also referred to as Chinese characters), alternative syllabic input methods (e.g., Pinyin or Kana) can be used to write the corresponding logographic characters. Even if a writing system is available for inputting characters, if the user does not know how to phonetically spell the logogram and the incorrect phonetic form of that logogram. When using spelling, such syllable input methods are inappropriate. Therefore, the ability to use handwritten input on a computing device becomes extremely important for users who are unable to pronounce words well or at all with respect to the associated logographic writing system.

Although handwriting input functionality has become somewhat popular within certain areas of the world, improvements are still needed. Specifically, human handwriting is highly variable (e.g., in terms of stroke order, size, typeface, etc.), and high-quality handwriting recognition software is complex and requires extensive training. Therefore, providing efficient real-time handwriting recognition on mobile devices with limited memory and computing resources has become a difficult challenge.

Furthermore, in today's multicultural world, users from many countries use multiple languages and may write in more than one script (e.g. mention a movie title in English, write a message in Chinese, etc.) ) may be required frequently. However, manually switching the recognition system to the desired script or language during writing is cumbersome and inefficient. Furthermore, the usefulness of conventional multi-script handwriting recognition techniques is severely limited, since expanding the recognition capabilities of a device to handle multiple scripts simultaneously increases the complexity of the recognition system. This is because the demands on performance and computer resources are significantly increased.

Moreover, traditional handwriting techniques rely heavily on language or script-specific specificities to achieve recognition accuracy. Such specificities are not easily portable to other languages or scripts. Therefore, adding handwriting input capabilities for new languages or scripts is a difficult task that is not easily undertaken by software and device suppliers. As a result, multilingual users are not provided with significant alternative input methods for their electronic devices.

A conventional user interface for providing handwriting input includes an area for accepting handwriting input from a user and an area for displaying handwriting recognition results. Significant improvements in user interfaces are still needed on portable devices with small form factors to improve efficiency, accuracy, and overall user experience.

This specification describes techniques for providing multi-script handwriting recognition using a general purpose recognizer. This general purpose recognizer is trained using a large multi-script corpus of writing samples for characters in various languages or scripts. The general recognition unit is trained in a language-independent manner, a script-independent manner, a stroke order-independent manner, and a stroke direction-independent manner. Therefore, the same recognition unit can recognize mixed language, mixed script handwritten input without the need for manual switching between input languages during use. Furthermore, this universal recognizer is lightweight enough to be deployed as a standalone module on a mobile device, allowing it to handle handwritten input in a variety of languages and scripts used in different areas around the world. It becomes possible.

Furthermore, this general-purpose recognizer is trained on stroke order-independent and stroke direction-independent spatially derived features and does not require temporal or sequence information at the stroke level; This universal recognizer provides a number of additional features and advantages over traditional time-based recognition methods, such as Hidden Markov Method (HMM)-based recognition methods. For example, a user can enter one or more letters, phrases, and sentence strokes in any order and still obtain the same recognition result. Therefore, out-of-order multiple character input and out-of-order modification (eg, addition or rewriting) of previously input characters are possible in this case.

Furthermore, this general-purpose recognizer is also used for real-time handwriting recognition, where temporal information about each stroke is available and the ambiguity of the handwriting input is determined before character recognition is performed by the general-purpose recognizer. Optionally used to remove or split handwritten input. The real-time, stroke order-independent recognition described herein is different from and better than traditional offline recognition methods (e.g., optical character recognition (OCR)). Can provide good performance. Furthermore, the general-purpose recognizer described herein allows the salient features of various variations (e.g., variations in speed, slowness, stroke order, stroke direction, stroke continuity, etc.) to be explicitly incorporated into the recognition system. The ability to handle high variability in individual writing habits (e.g., variability in speed, slowness, stroke order, stroke direction, stroke continuity, etc.) without incorporating into the recognition system's overall complexity is reduced.

As described herein, in some embodiments, temporally derived stroke distribution information is optionally reintroduced within the general purpose recognizer to improve recognition accuracy and Disambiguation between similar-looking recognition outputs for images is removed. The reintroduction of temporally derived stroke distribution information does not impair the stroke order and stroke direction independence of the general-purpose recognizer, but this This is because the features are obtained through a separate training process and are combined within the handwriting recognition model only after the separate training process is completed. Furthermore, temporally derived stroke distribution information captures salient temporal characteristics of similar-looking characters without relying on explicit knowledge of differences in stroke order of similar-looking characters. carefully designed to do so.

A user interface for providing handwriting input functionality is also described herein.

In some embodiments, a method of providing multi-script handwriting recognition is to train a multi-script handwriting recognition model based on spatially derived features of a multi-script training corpus, the method comprising: training, wherein the corpus includes respective handwriting samples corresponding to characters of at least three non-overlapping scripts; and the multi-script handwriting recognition is trained on spatially derived features of the multi-script training corpus. using the model to provide real-time handwriting recognition for the user's handwriting input.

In some embodiments, a method of providing multi-script handwriting recognition includes receiving a multi-script handwriting recognition model, the multi-script recognition model comprising spatially derived features of a multi-script training corpus. receiving a multi-script handwriting recognition model trained, the multi-script training corpus including respective handwriting samples corresponding to characters of at least three non-overlapping scripts; and receiving handwriting input from a user. receiving handwriting input, the handwriting input comprising one or more handwriting strokes provided on a touch-sensitive surface coupled to a user device; and in response to receiving the handwriting input; providing one or more handwriting recognition results to a user in real-time based on a multi-script handwriting recognition model that is trained on spatially derived features of a multi-script training corpus.

In some embodiments, a method of providing real-time handwriting recognition includes receiving a plurality of handwritten strokes from a user, the plurality of handwritten strokes corresponding to one handwritten character. generating an input image based on the plurality of handwritten strokes; and providing the input image to a handwriting recognition model to perform real-time recognition of the handwritten character; The handwriting recognition model provides stroke order independent handwriting recognition by providing an input image and receiving a plurality of handwritten strokes received from a user, regardless of their respective order. displaying the same first output character in real time.

In some embodiments, the method comprises receiving a second plurality of handwritten strokes from a user, the second plurality of handwritten strokes corresponding to a second handwritten character. generating a second input image based on the second plurality of handwritten strokes; and applying the second input image to the handwriting recognition model to perform real-time recognition of the second handwritten character. and displaying, in real time upon receiving the second plurality of handwritten strokes, a second output character corresponding to the second plurality of handwritten strokes, the first output character and the second and a second output character simultaneously displayed in a spatial arrangement independent of the respective order of the first plurality of handwritten inputs and the second plurality of handwritten inputs provided by the user. and displaying.

In some embodiments, the second plurality of handwritten strokes spatially follows the first plurality of handwritten strokes along a default writing direction of a handwriting input interface of the user device, and the second plurality of handwritten strokes spatially follows the first plurality of handwritten strokes along a default writing direction of a handwriting input interface of the user device; The characters follow the first output character in a spatial arrangement along its default writing direction, and the method includes receiving a third handwritten stroke from the user to correct the handwritten character. receiving a third handwritten stroke, the third handwritten stroke received later in time than the first plurality of handwritten strokes and the second plurality of handwritten strokes; In response to receiving a handwritten stroke, the third handwritten stroke is placed into the same recognition unit as the first plurality of handwritten strokes based on the relative proximity of the third handwritten stroke to the first plurality of handwritten strokes. generating a corrected input image based on the first plurality of handwritten strokes and the third handwritten stroke; and performing real-time recognition of the corrected handwritten character. providing a corrected input image to a handwriting recognition model; and in response to receiving a third handwriting input, displaying a third output character corresponding to the corrected input image, the third output character corresponding to the corrected input image; The output character further includes displaying a third output character that replaces the first output character and is displayed simultaneously with the second output character in a spatial arrangement along a default writing direction.

In some embodiments, the method receives a delete input from the user while the third output character and the second output character are simultaneously displayed as a recognition result in the candidate display area of the handwriting input interface. and, responsive to the deletion input, deleting the second output character from the recognition result while maintaining the third output character in the recognition result.

In some embodiments, a first plurality of handwritten strokes, a second plurality of handwritten strokes, and a third handwritten stroke are provided within a handwriting input area of the handwriting input interface, each of the handwritten strokes being input by the user. rendering in real time as provided, and in response to receiving the delete input, removing respective renderings of the handwritten strokes of the second plurality from the handwriting input area; and maintaining respective renderings of the third handwritten strokes within the handwriting input area.

In some embodiments, a method of providing real-time handwriting recognition includes receiving handwriting input from a user, the handwriting input being provided within one or more handwriting input areas of a handwriting input interface. receiving handwritten input, including handwritten strokes; identifying a plurality of output characters for the handwriting input based on a handwriting recognition model; and determining the plurality of output characters based on predefined categorization criteria. , by classifying into two or more categories and displaying each output character in a first category of the two or more categories in an initial view of a candidate display area of a handwriting input interface. and the initial view of this candidate display area is provided at the same time as an affordance for invoking an expanded view of the candidate display area, and the user selects an affordance for displaying output characters and for invoking this expanded view. receiving an input and, in response to the user input, displaying each output character in the first category in the expanded view of the candidate display area and that previously displayed in the initial view of the candidate display area; and displaying each output character in at least a second category of the two or more categories.

In some embodiments, a method of providing real-time handwriting recognition includes receiving handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided within a handwriting input area of a handwriting input interface. and recognizing a plurality of output characters from the handwriting input based on the handwriting recognition model, the output characters comprising at least a first pictogram and a natural human language. recognizing a plurality of output characters including at least a first character from a script of a script of a handwriting input interface; and displaying the candidate display area within the candidate display area.

In some embodiments, a method of providing handwriting recognition includes receiving handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided in a touch-sensitive surface coupled to a device. and rendering the plurality of handwritten strokes in real time within the handwriting input area of the handwriting input interface, and pinch gesture input and input on the plurality of handwritten strokes. a first one based on the plurality of handwritten strokes by receiving one of the enlarge gesture inputs and processing the plurality of handwritten strokes as a single recognition unit upon receiving the pinch gesture input. based on the plurality of handwritten strokes by generating a recognition result and, upon receiving the enlarged gesture input, processing the plurality of handwritten strokes as two separate recognition units separated by the enlarged gesture input. generating a second recognition result, and displaying the generated recognition result in a candidate display area of the handwriting input interface when generating each one of the first recognition result and the second recognition result; including doing.

In some embodiments, a method of providing handwriting recognition includes receiving handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided within a handwriting input area of a handwriting input interface. and identifying a plurality of recognition units from the plurality of handwritten strokes, each recognition unit comprising a corresponding subset of the plurality of handwritten strokes. generating recognition results for multiple characters including each character recognized from the multiple recognition units; and displaying the recognition results for the multiple characters within a candidate display area of a handwriting input interface. , while the recognition results of the multiple characters are displayed within the candidate display area, a deletion input is received from the user, and in response to the reception of this deletion input, the recognition results of the multiple characters displayed within the candidate display area are received. and removing the last character from the recognition result.

In some embodiments, a method for providing real-time handwriting recognition includes determining an orientation of a device and providing a handwriting input interface on the device in a horizontal input mode according to the device being in the first orientation. providing a horizontal input mode, wherein each line of handwritten input input in the horizontal input mode is divided into one or more respective recognition units along a horizontal writing direction; and providing a handwriting input interface on the device in a vertical input mode according to the second orientation of the device, wherein each line of handwritten input entered in the vertical input mode comprises: and providing in a vertical input mode divided into one or more respective recognition units along a vertical writing direction.

In some embodiments, a method of providing real-time handwriting recognition includes receiving handwriting input from a user, the handwriting input comprising a plurality of handwriting inputs provided on a touch-sensitive surface coupled to a device. receiving handwritten input including strokes; rendering the plurality of handwritten strokes within a handwriting input area of a handwriting input interface; and converting the plurality of handwritten strokes into two or more recognition units. dividing into two or more recognition units, each recognition unit including a corresponding subset of the plurality of handwritten strokes; and receiving an edit request from a user; In response to the edit request, means for visually identifying two or more recognition units within the handwriting input area and individually deleting each of the two or more recognition units from the handwriting input area. including providing.

In some embodiments, a method of providing real-time handwriting recognition includes receiving a first handwriting input from a user, the first handwriting input including a plurality of handwritten strokes, the first handwriting input including a plurality of handwriting strokes, receiving a first handwriting input, the handwriting strokes forming a plurality of recognition units distributed along respective writing directions, the handwriting strokes being associated with a handwriting input area of the handwriting input interface; rendering each of the plurality of handwritten strokes within the handwriting input area as provided by the user, and respective fading for each of the plurality of recognition units after the recognition unit is completely rendered; initiating a process, wherein during each fading process, a rendering of a recognition unit in the first handwritten input gradually fades; receiving a second handwriting input from the user over the area of the handwriting input area that was occupied by the faded recognition unit of the handwriting input area; the second handwriting input, and erasing all faded recognition units from the handwriting input area.

In some embodiments, a method of providing handwriting recognition comprises separately training a set of spatially derived features and a set of temporally derived features of a handwriting recognition model. , the set of spatially derived features is trained on a corpus of training images, each of which is an image of a handwritten sample for a respective character in the output character set, and the set of temporally derived features is trained on a corpus of training images, each of which is an image of a handwritten sample for a respective character in the output character set, and the set of temporally derived features is within the handwriting recognition model, trained on a corpus of distribution profiles, each stroke distribution profile numerically characterizing the spatial distribution of the plurality of strokes within the handwriting sample for each character of the output character set; Combining a set of spatially derived features and a set of temporally derived features, trained separately, and using the handwriting recognition model to provide real-time handwriting recognition of a user's handwriting input. Including things.

The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will be apparent from the description, drawings, and claims.

1 is a block diagram illustrating a portable multifunction device with a touch-sensitive display, according to some embodiments. FIG.

2 illustrates a portable multifunction device with a touch-sensitive display, according to some embodiments.

1 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface, according to some embodiments. FIG.

2 illustrates an example user interface for a multifunction device with a touch-sensitive surface separate from a display, according to some embodiments.

1 is a block diagram of an operating environment for a handwriting input system, according to some implementations. FIG.

FIG. 2 is a block diagram of a multi-script handwriting recognition model, according to some embodiments.

1 is a flowchart of an example process for training a multi-script handwriting recognition model, according to some embodiments.

2 illustrates an example user interface illustrating real-time multi-script handwriting recognition and input on a portable multifunction device, according to some embodiments. 2 illustrates an example user interface illustrating real-time multi-script handwriting recognition and input on a portable multifunction device, according to some embodiments.

1 is a flowchart of an example process for providing real-time multi-script handwriting recognition and input on a portable multifunction device. 1 is a flowchart of an example process for providing real-time multi-script handwriting recognition and input on a portable multifunction device.

1 is a flowchart of an example process for providing real-time, stroke order independent handwriting recognition and input on a portable multifunction device, according to some embodiments. 1 is a flowchart of an example process for providing real-time, stroke order independent handwriting recognition and input on a portable multifunction device, according to some embodiments. is a flowchart of an example process for providing real-time, stroke order independent handwriting recognition and input on a portable multifunction device, according to some embodiments.

selectively displaying one category of recognition results within a normal view of the candidate display area and selectively displaying another category of recognition results within an expanded view of the candidate display area, according to some embodiments; 2 shows an example user interface for. selectively displaying one category of recognition results within a normal view of the candidate display area and selectively displaying another category of recognition results within an expanded view of the candidate display area, according to some embodiments; 2 shows an example user interface for. selectively displaying one category of recognition results within a normal view of the candidate display area and selectively displaying another category of recognition results within an expanded view of the candidate display area, according to some embodiments; 2 shows an example user interface for. selectively displaying one category of recognition results within a normal view of the candidate display area and selectively displaying another category of recognition results within an expanded view of the candidate display area, according to some embodiments; 2 shows an example user interface for. selectively displaying one category of recognition results within a normal view of the candidate display area and selectively displaying another category of recognition results within an expanded view of the candidate display area, according to some embodiments; 2 shows an example user interface for. selectively displaying one category of recognition results within a normal view of the candidate display area and selectively displaying another category of recognition results within an expanded view of the candidate display area, according to some embodiments; 2 shows an example user interface for. selectively displaying one category of recognition results within a normal view of the candidate display area and selectively displaying another category of recognition results within an expanded view of the candidate display area, according to some embodiments; 2 shows an example user interface for. selectively displaying one category of recognition results within a normal view of the candidate display area and selectively displaying another category of recognition results within an expanded view of the candidate display area, according to some embodiments; 2 shows an example user interface for. selectively displaying one category of recognition results within a normal view of the candidate display area and selectively displaying another category of recognition results within an expanded view of the candidate display area, according to some embodiments; 2 shows an example user interface for. selectively displaying one category of recognition results within a normal view of the candidate display area and selectively displaying another category of recognition results within an expanded view of the candidate display area, according to some embodiments; 2 shows an example user interface for. selectively displaying one category of recognition results within a normal view of the candidate display area and selectively displaying another category of recognition results within an expanded view of the candidate display area, according to some embodiments; 2 shows an example user interface for.

selectively displaying one category of recognition results within a normal view of the candidate display area and selectively displaying another category of recognition results within an expanded view of the candidate display area, according to some embodiments; 1 is a flowchart of an exemplary process for doing so. selectively displaying one category of recognition results within a normal view of the candidate display area and selectively displaying another category of recognition results within an expanded view of the candidate display area, according to some embodiments; 1 is a flowchart of an exemplary process for doing so.

4 illustrates an example user interface for inputting emojis through handwritten input, according to some embodiments. 4 illustrates an example user interface for inputting emojis through handwritten input, according to some embodiments. 4 illustrates an example user interface for inputting emojis through handwritten input, according to some embodiments. 4 illustrates an example user interface for inputting emojis through handwritten input, according to some embodiments. 4 illustrates an example user interface for inputting emojis through handwritten input, according to some embodiments.

1 is a flowchart of an example process for inputting emojis through handwritten input, according to some embodiments.

Informing the handwriting input module how to separate the currently accumulated handwriting input into one or more recognition units using a pinch or zoom gesture, according to some embodiments. 1 shows an example user interface for the. Informing the handwriting input module how to separate the currently accumulated handwriting input into one or more recognition units using a pinch or zoom gesture, according to some embodiments. 1 shows an example user interface for the. Informing the handwriting input module how to separate the currently accumulated handwriting input into one or more recognition units using a pinch or zoom gesture, according to some embodiments. 1 shows an example user interface for the. Informing the handwriting input module how to separate the currently accumulated handwriting input into one or more recognition units using a pinch or zoom gesture, according to some embodiments. 1 shows an example user interface for the. Informing the handwriting input module how to separate the currently accumulated handwriting input into one or more recognition units using a pinch or zoom gesture, according to some embodiments. 1 shows an example user interface for the. Informing the handwriting input module how to separate the currently accumulated handwriting input into one or more recognition units using a pinch or zoom gesture, according to some embodiments. 1 shows an example user interface for the. Informing the handwriting input module how to separate the currently accumulated handwriting input into one or more recognition units using a pinch or zoom gesture, according to some embodiments. 1 shows an example user interface for the. Informing the handwriting input module how to separate the currently accumulated handwriting input into one or more recognition units using a pinch or zoom gesture, according to some embodiments. 1 shows an example user interface for the. Informing the handwriting input module how to separate the currently accumulated handwriting input into one or more recognition units using a pinch or zoom gesture, according to some embodiments. 1 shows an example user interface for the. Informing the handwriting input module how to separate the currently accumulated handwriting input into one or more recognition units using a pinch or zoom gesture, according to some embodiments. 1 shows an example user interface for the. Informing the handwriting input module how to separate the currently accumulated handwriting input into one or more recognition units using a pinch or zoom gesture, according to some embodiments. 1 shows an example user interface for the.

Informing the handwriting input module how to separate the currently accumulated handwriting input into one or more recognition units using a pinch or zoom gesture, according to some embodiments. 1 is a flowchart of an example process for. Informing the handwriting input module how to separate the currently accumulated handwriting input into one or more recognition units using a pinch or zoom gesture, according to some embodiments. 1 is a flowchart of an example process for.

2 illustrates an example user interface for providing character-by-character deletion of a user's handwritten input, according to some embodiments. 2 illustrates an example user interface for providing character-by-character deletion of a user's handwritten input, according to some embodiments. 2 illustrates an example user interface for providing character-by-character deletion of a user's handwritten input, according to some embodiments. 2 illustrates an example user interface for providing character-by-character deletion of a user's handwritten input, according to some embodiments. 2 illustrates an example user interface for providing character-by-character deletion of a user's handwritten input, according to some embodiments. 2 illustrates an example user interface for providing character-by-character deletion of a user's handwritten input, according to some embodiments. 2 illustrates an example user interface for providing character-by-character deletion of a user's handwritten input, according to some embodiments. 2 illustrates an example user interface for providing character-by-character deletion of a user's handwritten input, according to some embodiments.

2 is a flowchart of an example process for providing character-by-character deletion of a user's handwritten input, according to some embodiments. 2 is a flowchart of an example process for providing character-by-character deletion of a user's handwritten input, according to some embodiments.

4 illustrates an example user interface for switching between vertical and horizontal writing modes, according to some embodiments. 4 illustrates an example user interface for switching between vertical and horizontal writing modes, according to some embodiments. 4 illustrates an example user interface for switching between vertical and horizontal writing modes, according to some embodiments. 4 illustrates an example user interface for switching between vertical and horizontal writing modes, according to some embodiments. 4 illustrates an example user interface for switching between vertical and horizontal writing modes, according to some embodiments. 4 illustrates an example user interface for switching between vertical and horizontal writing modes, according to some embodiments.

3 is a flowchart of an example process for switching between vertical and horizontal writing modes, according to some embodiments. 3 is a flowchart of an example process for switching between vertical and horizontal writing modes, according to some embodiments. 3 is a flowchart of an example process for switching between vertical and horizontal writing modes, according to some embodiments.

2 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments; FIG. 2 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments; FIG. 2 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments; FIG. 2 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments; FIG. 2 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments; FIG. 2 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments; FIG. 2 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments; FIG. 2 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments; FIG.

2 is a flowchart of an example process for providing a means for displaying and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments. 2 is a flowchart of an example process for providing a means for displaying and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments.

Implicit confirmation input for inputting a new handwriting input provided on top of an existing handwriting input within a handwriting input area with a displayed recognition result for the existing handwriting input, according to some embodiments. 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for inputting a new handwriting input provided on top of an existing handwriting input within a handwriting input area with a displayed recognition result for the existing handwriting input, according to some embodiments. 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for inputting a new handwriting input provided on top of an existing handwriting input within a handwriting input area with a displayed recognition result for the existing handwriting input, according to some embodiments. 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for inputting a new handwriting input provided on top of an existing handwriting input within a handwriting input area with a displayed recognition result for the existing handwriting input, according to some embodiments. 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for inputting a new handwriting input provided on top of an existing handwriting input within a handwriting input area with a displayed recognition result for the existing handwriting input, according to some embodiments. 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for inputting a new handwriting input provided on top of an existing handwriting input within a handwriting input area with a displayed recognition result for the existing handwriting input, according to some embodiments. 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for inputting a new handwriting input provided on top of an existing handwriting input within a handwriting input area with a displayed recognition result for the existing handwriting input, according to some embodiments. 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for inputting a new handwriting input provided on top of an existing handwriting input within a handwriting input area with a displayed recognition result for the existing handwriting input, according to some embodiments. 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for inputting a new handwriting input provided on top of an existing handwriting input within a handwriting input area with a displayed recognition result for the existing handwriting input, according to some embodiments. 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for inputting a new handwriting input provided on top of an existing handwriting input within a handwriting input area with a displayed recognition result for the existing handwriting input, according to some embodiments. 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for inputting a new handwriting input provided on top of an existing handwriting input within a handwriting input area with a displayed recognition result for the existing handwriting input, according to some embodiments. 1 illustrates an exemplary user interface for use as a . Implicit confirmation input for inputting a new handwriting input provided on top of an existing handwriting input within a handwriting input area with a displayed recognition result for the existing handwriting input, according to some embodiments. 1 illustrates an exemplary user interface for use as a .

Implicit confirmation input for inputting a new handwriting input provided on top of an existing handwriting input within a handwriting input area with a displayed recognition result for the existing handwriting input, according to some embodiments. 1 is a flowchart of an exemplary process for use as a computer. Implicit confirmation input for inputting a new handwriting input provided on top of an existing handwriting input within a handwriting input area with a displayed recognition result for the existing handwriting input, according to some embodiments. 1 is a flowchart of an exemplary process for use as a computer.

Integrating temporally derived stroke distribution information into a spatially derived feature-based handwriting recognition model without compromising the stroke order and stroke direction dependence of the handwriting recognition model, according to some embodiments 1 is a flowchart of an exemplary process for doing so. Integrating temporally derived stroke distribution information into a spatially derived feature-based handwriting recognition model without compromising the stroke order and stroke direction dependence of the handwriting recognition model, according to some embodiments 1 is a flowchart of an example process for doing so.

FIG. 2 is a block diagram illustrating separate training and subsequent integration of spatially and temporally derived features of an example handwriting recognition system, according to some embodiments.

FIG. 2 is a block diagram illustrating an example method for calculating a stroke distribution profile of a character.

Like reference numbers refer to corresponding parts throughout these figures.

Many electronic devices have graphical user interfaces with soft keyboards for character input. On some electronic devices, a user may also be able to install or enable a handwriting input interface that allows the user to use a touch-sensitive display coupled to the device. Characters can be entered via handwriting on a screen or touch-sensitive surface. Traditional handwriting recognition input methods and user interfaces have several problems and drawbacks. for example,
- Typically, traditional handwriting input functionality is enabled on a language-by-language or script-by-script basis. Each added input language requires the installation of a separate handwriting recognition model, consuming separate storage space and memory. The synergy provided by combining handwriting recognition models for different languages is negligible, and handwriting recognition for mixed languages or scripts traditionally takes a very long time due to the complex disambiguation process. It was something.
●Furthermore, conventional handwriting recognition systems rely heavily on language-specific or script-specific characteristics for character recognition. Recognition of mixed language handwritten input had low accuracy. Moreover, the available combinations of recognized languages are extremely limited. Most systems required the user to manually specify the desired language-specific handwriting recognizer before providing handwriting input in each non-default language or script.
Many existing real-time handwriting recognition models require temporal or sequence information at the stroke-by-stroke level, which can lead to high variability in how characters can be written (e.g., typeface or personal (high variability in stroke shape, length, slowness, division, order, and direction) can lead to inaccurate recognition results. Some systems also require users to adhere to strict spatial and temporal criteria (e.g., with built-in assumptions about the size, arrangement, and time frame of each character input) when providing handwritten input. It also requires compliance. Any deviation from these standards would result in inaccurate recognition results that were difficult to correct.
●Currently, most real-time handwriting input interfaces only allow users to input a few characters at a time. Long phrase or sentence inputs are broken down into short segments and input separately. This formal input not only imposes a cognitive burden on the user to maintain the flow of the composition, but also makes it difficult for the user to modify or correct previously entered characters or phrases.

The embodiments described below address these and related issues.

1-4 below provide a description of example devices. 5 and 6 and 26 and 27 illustrate example handwriting recognition and input systems. 8A, 8B, 11A to 11K, 13A to 13E, 15A to 15K, 17A to 17F, 19A to 19F, 21A to 21H, and 23A to 12L are handwritten. 2 shows an example user interface for recognition and input. 7, 9A, 9B, 10A to 10C, 12A, 12B, 14, 16A, 16B, 18A, 18B, 20A to 20C, 22A, 22B, 24A , 24B, and 25 illustrate training a handwriting recognition model, providing real-time handwriting recognition results, providing a means for inputting and correcting handwriting input, and inputting the recognition results as text input. 1 is a flowchart illustrating a method for enabling handwriting recognition and input, including providing a means. Users in Figures 8A, 8B, 11A to 11K, 13A to 13E, 15A to 15K, 17A to 17F, 19A to 19F, 21A to 21H, and 23A to 12L The interfaces are shown in FIGS. 7, 9A, 9B, 10A to 10C, 12A, 12B, 14, 16A, 16B, 18A, 18B, 20A to 20C, 22A, 22B. , used to illustrate the process in FIGS. 24A, 24B, and 25.
Exemplary device

Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

It is also understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. will be done. These terms are only used to distinguish one element from another. For example, a first contact can be referred to as a second contact, and similarly a second contact can be referred to as a first contact, without departing from the scope of the invention. Although the first contact and the second contact are both contacts, they are not the same contact.

The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in the description of this invention and the appended claims, the singular forms "a," "an," and "the" also refer to the plural forms unless the context clearly dictates otherwise. intended to include. As used herein, the term "and/or" refers to, and also includes, any and all possible combinations of one or more of the associated listed items. It will be understood. The terms "includes," "including," "comprises," and/or "comprising" as used herein refer to the described features, integers, steps. , which specifies the presence of an action, element, and/or component, but excludes the presence or addition of one or more other features, integers, steps, actions, elements, components, and/or groups thereof. It will be further understood that this is not the case.

As used herein, the term "in case" means "at" or "upon" or "in response to a determination that" or "in response to the detection of..." depending on the context. It can be interpreted as meaning "depending on". Similarly, the phrases "if it is determined that..." or "if (the described condition or event) is detected" can be used as "if it is determined that..." or "if (the described condition or event) is detected," depending on the context. or “upon detection of (the described condition or event)” or “in response to the detection of (the described condition or event).” .

Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device such as a mobile phone that also includes other functionality such as PDA functionality and/or music player functionality. Exemplary embodiments of portable multifunction devices include Apple Inc. (Cupertino, California), iPod Touch®, and iPad® devices. Other portable electronic devices can also be used, such as laptop computers or tablet computers with touch-sensitive surfaces (eg, touch screen displays and/or touch pads). It should also be understood that in some embodiments, the device is a desktop computer with a touch-sensitive surface (eg, a touch screen display and/or touch pad) rather than a portable communication device.

The following discussion describes electronic devices that include displays and touch-sensitive surfaces. However, it should be understood that this electronic device may include one or more other physical user interface devices, such as a physical keyboard, mouse, and/or joystick.

This device typically includes drawing applications, presentation applications, word processing applications, website creation applications, disc authoring applications, spreadsheet applications, gaming applications, telephone applications, video conferencing applications, email applications, instant messaging applications, Various applications are supported, such as one or more of a training support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application.

The various applications that can run on the device can use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface, and corresponding information displayed on the device, may be adjusted and/or changed for each application and/or within each application. In this way, the device's common physical architecture (such as a touch-sensitive surface) can support a variety of applications with a user interface that is intuitive and transparent to the user.

Attention is now directed to embodiments of portable devices with touch-sensitive displays. FIG. 1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display 112, according to some embodiments. Touch-sensitive display 112 may be conveniently referred to as a "touch screen" and may also be known or referred to as a touch-sensitive display system. Device 100 includes memory 102 (which may include one or more computer-readable storage media), memory controller 122, one or more processing units (CPUs) 120, peripheral interface 118, RF circuitry 108, audio circuitry 110, It may include a speaker 111, a microphone 113, an input/output (I/O) subsystem 106, other input or control devices 116, and an external port 124. Device 100 may include one or more optical sensors 164. These components may communicate via one or more communication buses or signal lines 103.

Device 100 is only one example of a portable multifunction device, and device 100 may have more or fewer components than shown, and may include more than one component. It is to be understood that they may be combined or have different configurations or arrangements of the components. The various components shown in FIG. 1 can be implemented as hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and/or application-specific integrated circuits. .

Memory 102 may include high speed random access memory and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile semiconductor memory devices. Access to memory 102 by other components of device 100, such as CPU 120 and peripheral interface 118, may be controlled by memory controller 122.

A peripheral interface 118 may be used to couple the device's input and output peripherals to the CPU 120 and memory 102. One or more processors 120 operate or execute various software programs and/or instruction sets stored within memory 102 to perform various functions and process data for device 100.

In some embodiments, peripheral interface 118, CPU 120, and memory controller 122 may be implemented on a single chip, such as chip 104. In some other embodiments, they can be implemented on separate chips.

RF (radio frequency) circuitry 108 sends and receives RF signals, also referred to as electromagnetic signals. RF circuitry 108 converts electrical signals to electromagnetic signals and communicates with communication networks and other communication devices via the electromagnetic signals.

Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between the user and device 100. Audio circuitry 110 receives audio data from peripheral interface 118 , converts the audio data to an electrical signal, and transmits the electrical signal to speaker 111 . Speaker 111 converts the electrical signal into human audible sound waves. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113. Audio circuitry 110 converts the electrical signal to audio data and sends the audio data to peripheral interface 118 for processing. Audio data may be retrieved from and/or transmitted to memory 102 and/or RF circuitry 108 by peripheral interface 118 . In some embodiments, audio circuitry 110 also includes a headset jack (eg, 212 in FIG. 2).

I/O subsystem 106 couples input and output peripherals on device 100, such as touch screen 112 and other input control devices 116, to peripheral interface 118. I/O subsystem 106 may include a display controller 156 and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive/transmit electrical signals to/from other input or control devices 116 . Other input control devices 116 may include physical buttons (eg, push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and the like. In some alternative embodiments, input controller 160 may be coupled to any (or may not be coupled to) a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. ). The one or more buttons (eg, 208 in FIG. 2) may include up/down buttons for volume adjustment of speaker 111 and/or microphone 113. The one or more buttons may include a push button (eg, 206 in FIG. 2).

Touch-sensitive display 112 provides an input and output interface between the device and the user. Display controller 156 receives and/or transmits electrical signals to and from touch screen 112 . Touch screen 112 displays visual output to the user. This visual output may include graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of these visual outputs may correspond to user interface objects.

Touchscreen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on tactile and/or tactile contact. Touch screen 112 and display controller 156 (along with any associated modules and/or instruction sets in memory 102) detect a touch (and any movement or interruption of that touch) on touch screen 112 and The touch is translated into an interaction with a user interface object (eg, one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In the exemplary embodiment, the point of contact between touch screen 112 and the user corresponds to the user's finger.

The touch screen 112 may use LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, although in other embodiments other display technologies may be used. can. Touch screen 112 and display controller 156 may detect contact and any movement or interruption of that contact using any of a number of touch-sensing technologies now known or hereafter developed; These technologies include capacitive, resistive, infrared, and surface ultrasound technologies, as well as other proximity sensor arrays or other sensors for determining one or more points of contact with the touch screen 112. These elements include, but are not limited to: In an exemplary embodiment, Apple Inc. Projected mutual capacitance sensing technology is used, such as that found in the iPhone®, iPod Touch®, and iPad® by Co., Ltd. (Cupertino, California).

Touch screen 112 may have a video resolution of greater than 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi. A user may contact touch screen 112 using any suitable object or appendage, such as a stylus, finger, etc. In some embodiments, the user interface is designed to work primarily with finger-based touches and gestures, but these can be overridden by stylus-based input due to the larger contact area of the finger on the touch screen. The accuracy may be lower than that of In some embodiments, the device converts coarse finger-based input into precise pointer/cursor positions or commands to perform actions desired by the user. Handwriting input may be provided on the touch screen 112 through the location and movement of finger-based or stylus-based contacts. In some embodiments, the touch screen 112 provides a writing surface for rendering finger-based or stylus-based input as instant visual feedback for that current handwriting input and for using a writing instrument (e.g., a pen). Provides a visual effect of actual writing on (eg, a piece of paper).

In some embodiments, in addition to a touch screen, device 100 may include a touch pad (not shown) for activating or deactivating certain features. In some embodiments, the touchpad is a touch-sensitive area of the device that, unlike a touchscreen, does not display visual output. The touchpad may be a touch-sensitive surface separate from touchscreen 112 or an extension of the touch-sensitive surface formed by the touchscreen.

Device 100 also includes a power system 162 for powering various components. Power system 162 includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power outage detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)). , and any other components associated with the generation, management, and distribution of power within a portable device.

Device 100 may also include one or more optical sensors 164. FIG. 1 shows a light sensor coupled to a light sensor controller 158 within I/O subsystem 106. Optical sensor 164 may include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. Light sensor 164 receives light from the environment that is projected through one or more lenses and converts the light into data representing an image. In conjunction with imaging module 143 (also referred to as a camera module), optical sensor 164 can capture still images or video.

Device 100 may also include one or more proximity sensors 166. FIG. 1 shows a proximity sensor 166 coupled to peripheral interface 118. Alternatively, proximity sensor 166 may be coupled to input controller 160 within I/O subsystem 106. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call), the proximity sensor is disabled by turning off the touch screen 112. do.

Device 100 may also include one or more accelerometers 168. FIG. 1 shows accelerometer 168 coupled to peripheral interface 118. Alternatively, accelerometer 168 may be coupled to input controller 160 within I/O subsystem 106. In some embodiments, information is displayed on a touch screen display in portrait or landscape view based on analysis of data received from one or more accelerometers. In addition to the accelerometer 168, the device 100 optionally includes a magnetometer (not shown) and a GPS (or , GLONASS or other global navigation system) receiver (not shown).

In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a touch/motion module (or instruction set) 130, a graphics module (or instruction set) 130, and a graphics module (or instruction set) 130. ) 132 , a text input module (or instruction set) 134 , a global positioning system (GPS) module (or instruction set) 135 , and an application (or instruction set) 136 . Additionally, in some embodiments, as shown in FIGS. 1 and 3, memory 102 stores a handwriting input module 157. Handwriting input module 157 includes a handwriting recognition model and provides handwriting recognition and input functionality to the user of device 100 (or device 300). Further details of the handwriting input module 157 are provided in connection with FIGS. 5-27 and their accompanying descriptions.

Operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or VxWorks) Embedded operating systems (embedded operating systems) include various software components and/or drivers to control and manage common system tasks (e.g., memory management, storage device control, power management, etc.) and Facilitates communication between components and software components.

Communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software for processing data received by RF circuitry 108 and/or external ports 124. Also includes constituent elements. External ports 124 (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) can be coupled directly to other devices or indirectly via a network (e.g., the Internet, WLAN, etc.). adapted to do so.

Contact/motion module 130 (in conjunction with display controller 156) can detect contact with touch screen 112 and contact with other touch-sensitive devices (eg, a touch pad or physical click wheel). The touch/movement module 130 determines whether a touch has occurred (e.g., detecting a finger down event), determines whether there is movement of the touch, and detects movement across the touch-sensitive surface. (e.g., detecting one or more finger dragging events) and determining whether contact has ceased (e.g., detecting a finger raising event or a break in contact). includes various software components for performing various operations related to contact detection, such as. Touch/motion module 130 receives touch data from a touch-sensitive surface. Determining the movement of a point of contact, as represented by a set of contact data, involves determining the speed (magnitude), velocity (magnitude and direction), and/or acceleration (change in magnitude and/or direction) of the contact point. ). These operations can be applied to a single touch (eg, one finger touch) or multiple simultaneous contacts (eg, "multi-touch"/multiple finger contacts). In some embodiments, touch/movement module 130 and display controller 156 detect a touch on the touchpad.

Touch/motion module 130 can detect gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns. Therefore, gestures can be detected by detecting specific contact patterns. For example, detecting a finger tap gesture involves detecting a finger down event, followed by a finger tap at the same location (or substantially the same location) as that finger down event (e.g., at the location of an icon). including detecting lift-off events. As another example, detecting a finger swipe gesture on a touch-sensitive surface may include detecting a finger down event, followed by one or more finger dragging events, and then subsequently detecting a finger swipe gesture on a touch-sensitive surface. including detecting lift-off events.

The touch/movement module 130 is capable of detecting touch signals within the handwriting input area of the handwriting input interface displayed on the touch-sensitive display screen 112 (or corresponding to the handwriting input area displayed on the display 340 in FIG. 3). (within the area of pad 355) is optionally utilized by handwriting input module 157 to register the input of handwritten strokes. In some embodiments, the location, path of movement, and intensity associated with the first finger down event, the last finger up event, and the contact during any time between them are recorded as a handwritten stroke. Ru. Based on such information, handwritten strokes can be rendered on the display as feedback regarding that user input. Additionally, one or more input images may be generated based on the handwritten strokes registered by the touch/movement module 130.

Graphics module 132 includes various known software components for rendering and displaying graphics on touch screen 112 or other display, including components for changing the brightness of the displayed graphics. . As used herein, the term "graphic" includes any object that can be displayed to a user, including text, web pages, icons (user interface objects including softkeys) etc.), digital images, videos, animations, etc., but are not limited to these.

In some embodiments, graphics module 132 stores data representing the graphics used. Each graphic can be assigned a corresponding code. Graphics module 132 receives one or more codes specifying graphics to be displayed, optionally along with coordinate data and other graphical characteristic data, such as from an application, and then receives one or more codes for output to display controller 156 . Generate screen image data.

Text input module 134 can be a component of graphics module 132 and can be used for various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other applications that require text input. ) provides a soft keyboard for entering text. In some embodiments, handwriting input module 157 is optionally invoked through the user interface of text input module 134, such as through a keyboard selection affordance. In some embodiments, the same or similar keyboard selection affordances are also provided within the handwriting input interface to invoke the text input module 134.

The GPS module 135 determines the location of the device and transmits this information to the phone 138 for use in various applications (e.g., to the phone 138 for use in location-based dialing calls, to the camera as photo/video metadata). 143 and to applications that provide location-based services, such as weather widgets, local yellow pages widgets, and map/navigation widgets).

The application 136 may include the following modules (or sets of instructions), or a subset or superset thereof: a contacts module 137 (sometimes referred to as an address book or contact list); a telephony module 138; a video conferencing module. 139; email client module 140; instant messaging (IM) module 141; training support module 142; camera module 143 for still and/or video images; image management module 144; browser module 147; calendar module 148; weather widget 149 -1, one of a stock price widget 149-2, a calculator widget 149-3, an alarm clock widget 149-4, a dictionary widget 149-5, and other widgets obtained by the user, and a user-created widget 149-6. Widget module 149, which may include: a widget creation module 150 for creating user-created widgets 149-6; a search module 151; a video and music player module 152, which may be comprised of a video player module and a music player module; notes module 153; map module 154; and/or online video module 155.

Examples of other applications 136 that may be stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital writing, etc. rights management, voice recognition, and voice replication.

In conjunction with the touch screen 112, the display controller 156, the contact module 130, the graphics module 132, the handwriting input module 157, and the text input module 134, the contacts module 137 (e.g., contacts module in memory 102 or memory 370) 137 application internal state 192) can be used to manage an address book or contact list, including adding names to the address book, deleting names from the address book. , associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and sorting names; telephone calls 138; video conferencing 139; email 140; or providing a telephone number or email address to initiate and/or facilitate communication, such as via IM 141.

Telephone module 138 in conjunction with RF circuitry 108 , audio circuitry 110 , speaker 111 , microphone 113 , touch screen 112 , display controller 156 , contact module 130 , graphics module 132 , handwriting input module 157 , and text input module 134 enters a sequence of characters corresponding to a telephone number, accesses one or more telephone numbers in the address book 137, modifies the entered telephone number, dials each telephone number, and conducts a conversation. and can be used to disconnect or hang up when the conversation is complete. As mentioned above, this wireless communication may use any of a number of communication standards, protocols, and technologies.

RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, light sensor 164, light sensor controller 158, contact module 130, graphics module 132, handwriting input module 157, text input module 134 , a contact list 137, and a telephony module 138, a video conferencing module 139 initiates, conducts, and terminates a video conference between the user and one or more other participants according to the user's instructions. Contains executable instructions to do so.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, email client module 140, in response to user instructions, Contains executable instructions for creating, sending, receiving, and managing email. In conjunction with the image management module 144, the email client module 140 greatly facilitates creating and sending emails with still or video images taken with the camera module 143.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, instant messaging module 141 generates a series of characters corresponding to an instant message. and modify previously entered characters (e.g., using Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for telephone-based instant messaging, or Internet-based instant messaging). includes executable instructions for sending instant messages (using XMPP, SIMPLE, or IMPS), receiving instant messages, and viewing received instant messages. In some embodiments, instant messages sent and/or received include graphics, photos, audio files, video files, and/or other May contain attachments. As used herein, "instant messaging" refers to telephone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., using XMPP, SIMPLE, or IMPS). (messages sent).

In conjunction with the RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, GPS module 135, map module 154, and music player module 146, the training Support module 142 is used to create workouts (e.g., with time, distance, and/or calorie expenditure goals), communicate with training sensors (sport devices), receive training sensor data, and monitor training. Executable instructions include executable instructions for calibrating sensors to be used, selecting and playing training music, and displaying, storing, and transmitting training data.

In conjunction with touch screen 112, display controller 156, light sensor 164, light sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 captures still images or video (including video streams). The image processing apparatus includes executable instructions for capturing images, storing them in memory 102, modifying characteristics of the still images or videos, or deleting the still images or videos from memory 102.

In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, and camera module 143, image management module 144 arranges still and/or video images. includes executable instructions for editing, modifying (eg, editing) or otherwise manipulating, labeling, deleting, presenting (eg, in a digital slide show or album), and storing.

In conjunction with the RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, the browser module 147 can display a web page or portions thereof, and Contains executable instructions for browsing the Internet according to a user's instructions, including searching for, linking to, receiving, and displaying attachments and other files linked to web pages.

Calendar module 148 in conjunction with RF circuitry 108 , touch screen 112 , display system controller 156 , contact module 130 , graphics module 132 , handwriting input module 157 , text input module 134 , email client module 140 , and browser module 147 includes executable instructions for creating, displaying, modifying, and storing a calendar and data associated with the calendar (eg, calendar items, to-do lists, etc.) according to a user's instructions.

In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, and browser module 147, widget module 149 can be downloaded by a user. , mini-applications that can be used (e.g., weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5), or created by the user. A mini-application (eg, a user-created widget 149-6) that can be used to create a widget. In some embodiments, widgets include HTML (hypertext markup language) files, CSS (cascading style sheets) files, and JavaScript files. In some embodiments, widgets include XML (Extensible Markup Language) files and JavaScript files (eg, Yahoo!® widgets).

In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, and browser module 147, widget creation module 150 creates widgets. (e.g., to turn a user-specified portion of a web page into a widget).

In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, search module 151 is configured to perform one or more search criteria (e.g. , one or more user-specified search criteria).

In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speakers 111, RF circuitry 108, and browser module 147, video and music player module 152 supports MP3 or AAC Executable instructions that enable a user to download and play recorded music and other audio files stored in one or more file formats, such as and executable instructions for displaying, presenting, or otherwise reproducing (on or on a display externally connected via external port 124). In some embodiments, device 100 may include MP3 player functionality, such as an iPod (a registered trademark of Apple Inc.).

In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, memo module 153 creates and creates notes, to-do lists, etc. according to user instructions. Contains executable instructions for managing.

In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, GPS module 135, and browser module 147, map module 154 includes: According to the user's instructions, receive a map and data associated with the map (e.g., driving directions, data about stores at or near a particular location and other points of interest, and other location-based data). , can be used to display, modify and store.

Touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, handwriting input module 157, text input module 134, email client module 140, and browser module 147 In conjunction with the online video module 155, the online video module 155 allows a user to access, view, and receive (e.g., by streaming and/or downloading) certain online videos (e.g., on a touch screen or via external port 124). play it (on an external display connected via H. The method includes instructions that allow online video to be managed in one or more file formats, such as H.264. In some embodiments, instant messaging module 141 is used rather than email client module 140 to send a link to a particular online video.

Each of the modules and applications identified above may perform one or more of the functions described above and the methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). ) corresponds to a set of executable instructions. These modules (i.e., instruction sets) need not be implemented as separate software programs, procedures, or modules; therefore, various embodiments may combine various subsets of these modules or otherwise It can be reconstructed using the following method. In some embodiments, memory 102 may store a subset of the modules and data structures identified above. Furthermore, memory 102 may store additional modules and data structures not described above.

In some embodiments, device 100 is a device in which the operation of a predetermined set of functions on the device is performed exclusively via a touch screen and/or touch pad. By using a touch screen and/or touch pad as the primary input control device for the operation of device 100, the number of physical input control devices (push buttons, dials, etc.) on device 100 can be reduced.

FIG. 2 illustrates a portable multifunction device 100 with a touch screen 112, according to some embodiments. The touch screen can display one or more graphics within a user interface (UI) 200. In this embodiment, as well as other embodiments described below, the user may, for example, use one or more fingers 202 (not drawn to scale in the illustration) or one or more stylus 203 (not drawn to scale in the illustration). (not drawn to scale) can be used to select one or more of the graphics by performing gestures on the graphics. In some embodiments, selection of one or more graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, the gesture is one or more taps, one or more swipes (left-to-right, right-to-left, upward, and/or downward), and/or contacting the device 100 ( (right-to-left, left-to-right, upward, and/or downward) finger rolling. In some embodiments, accidental contact with a graphic may not select that graphic. For example, if the gesture that corresponds to selection is a tap, a swipe gesture that sweeps over an application icon may not select the corresponding application.

Device 100 may also include one or more physical buttons, such as a “home” or menu button 204. As mentioned above, menu button 204 can be used to navigate to any application 136 within the set of applications that can run on device 100. Alternatively, in some embodiments, the menu button is implemented as a softkey within a GUI displayed on touch screen 112.

In one embodiment, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for powering the device on/off and locking the device, a volume control button 208, and a subscriber identity module (SIM) card slot. 210, a headset jack 212, and an external docking/charging port 124. The push button 206 turns on and off power on the device by pressing the button down and holding the button down for a predetermined time interval, and by pressing the button down and holding the button down for a predetermined time interval. Releasing the button can be used to lock the device and/or unlock the device or initiate an unlocking process. In an alternative embodiment, device 100 may also accept verbal input via microphone 113 regarding activation or deactivation of some features.

FIG. 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface, according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a child's learning toy), a gaming system, a telephone device, or a control device ( For example, a home or business controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more processors for interconnecting these components. Includes a communication bus 320. Communication bus 320 may include circuitry (sometimes referred to as a chipset) that interconnects and controls communications between system components. Device 300 includes an input/output (I/O) interface 330 with a display 340, typically a touch screen display. I/O interface 330 may also include a keyboard and/or mouse (or other pointing device) 350 and touch pad 355. Memory 370 includes high speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access semiconductor memory devices, and also includes one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other memory devices. may include non-volatile memory, such as non-volatile semiconductor storage devices. Memory 370 may optionally include one or more storage devices located remotely from CPU 310. In some embodiments, memory 370 stores programs, modules, and data structures similar to or similar to those stored within memory 102 of portable multifunction device 100 (FIG. 1). Remember the subset. Furthermore, memory 370 may store additional programs, modules, and data structures that are not present within memory 102 of portable multifunction device 100. For example, the memory 370 of the device 300 may store a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disk authoring module 388, and/or a spreadsheet module 390, while In this case, the memory 102 of the portable multifunction device 100 (FIG. 1) may not store these modules.

Each of the elements of FIG. 3 identified above may be stored in one or more of the aforementioned memory devices. Each of the modules identified above corresponds to a set of instructions that perform the functions described above. The modules or programs (i.e., instruction sets) identified above need not be implemented as separate software programs, procedures, or modules; therefore, various embodiments may implement various subsets of these modules. may be combined or otherwise reconfigured. In some embodiments, memory 370 may store a subset of the modules and data structures identified above. Furthermore, memory 370 may store additional modules and data structures not described above.

FIG. 4 shows an example screen on a device (e.g., device 300 of FIG. 3) that includes a touch-sensitive surface 451 (e.g., tablet or touchpad 355 of FIG. 3) separate from a display 450 (e.g., touchscreen display 112). Figure 2 shows a typical user interface. Although many of the examples below are described with reference to input on a touch screen display 112 (when a touch-sensitive surface and display are combined), in some embodiments the device 4, detecting input on a touch-sensitive surface separate from the display. In some embodiments, the touch-sensitive surface (e.g., 451 in FIG. 4) has a major axis (e.g., 453 in FIG. 4) that corresponds to a major axis (e.g., 453 in FIG. 4) on the display (e.g., 450). 452). According to these embodiments, the device includes a touch-sensitive surface 451 and a touch-sensitive surface 451 at locations corresponding to their respective locations on the display (e.g., in FIG. 4, 460 corresponds to 468 and 462 corresponds to 470). (eg, 460 and 462 in FIG. 4). In this manner, if the touch-sensitive surface is separate from the display, user inputs (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the multifunction device's display (eg, 450 in FIG. 4). It should be appreciated that similar methods can be used with other user interfaces described herein.

Attention is now directed to embodiments of handwriting input methods and user interfaces ("UI") that may be implemented on a multifunction device (eg, device 100).

FIG. 5 shows an I/O interface module 500 (e.g., I/O interface 330 in FIG. 3 or I/O subsystem 106 in FIG. 1) interacting with an I/O interface module 500 on a device, according to some implementations. FIG. 2 is a block diagram illustrating an example handwriting input module 157 that provides handwriting input capabilities to the user. As shown in FIG. 5, handwriting input module 157 includes an input processing module 502, a handwriting recognition module 504, and a result generation module 506. In some embodiments, input processing module 502 includes a segmentation module 508 and a normalization module 510. In some embodiments, result generation module 506 includes a radical clustering module 512 and one or more language models 514.

In some embodiments, input processing module 502 communicates with I/O interface module 500 (e.g., I/O interface 330 in FIG. 3 or I/O subsystem 106 in FIG. 1) to Receive handwritten input from. Handwriting is input via any suitable means, such as touch sensitive display system 112 in FIG. 1 and/or touch pad 355 in FIG. 3. These handwriting inputs include data representing each stroke provided by the user within a predefined handwriting input area within the handwriting input UI. In some embodiments, data representing each stroke of handwriting input includes the beginning and ending of a sustained contact within the handwriting input area (e.g., contact between a user's finger or stylus and a touch-sensitive surface of a device). Contains data such as location, intensity profile, and path of movement. In some embodiments, I/O interface module 500 passes an array of handwritten strokes 516, with associated temporal and spatial information, to input processing module 502 in real time. At the same time, the I/O interface module also provides a real-time rendering 518 of these handwritten strokes within the handwriting input area of the handwriting input user interface as visual feedback for this user's input.

In some embodiments, as data representing each handwritten stroke is received by input processing module 502, temporal and spatial information associated with the plurality of consecutive strokes is also recorded. For example, the data optionally includes the shape, size, spatial saturation of individual strokes with each stroke sequence number, and the relative of those strokes along the writing direction of the entire handwriting input. Contains a stack, indicating, for example, spatial location. In some embodiments, the input processing module 502 uses an I/O signal to render the received strokes on a display 518 of the device (e.g., display 340 in FIG. 3 or touch-sensitive display 112 in FIG. 1). /O interface module 500; In some embodiments, the rendering of the received strokes is animated to mimic the actual progression of writing on a writing surface (e.g., a piece of paper) using a writing instrument (e.g., a pen). , visual effects are provided. In some embodiments, the user is optionally allowed to specify the nib style, color, texture, etc. of the rendered strokes.

In some embodiments, input processing module 502 processes the currently accumulated strokes within the handwriting input area and assigns the strokes to one or more recognition units. In some embodiments, each recognition unit corresponds to a particular character recognized by handwriting recognition model 504. In some embodiments, each recognition unit corresponds to a particular output character or radical recognized by handwriting recognition model 504. Radicals are reproducible components found within multiple compound logograms. A compound logogram may include two or more radicals arranged according to a common layout (eg, left-right layout, top-down layout, etc.). In one embodiment, a single Chinese character "
” means two radicals, namely the left radical “
” and the right radical “
It is built using ``.

In some embodiments, input processing module 502 relies on a segmentation module to assign or separate currently accumulated handwritten strokes into one or more recognition units. For example, handwritten characters “
”, the splitting module 508 optionally splits the strokes that are clustered on the left side of the handwritten input into one (i.e., the left radical “
”) and assign the strokes that are clustered on the right side of the handwritten input to another (i.e., the right radical “
”) to the recognition unit. Alternatively, the segmentation module 508 also splits all of these strokes into a single (i.e., the character "
”) can also be assigned to the recognition unit.

In some embodiments, the segmentation module 508 may segment the currently accumulated handwriting input (e.g., one or more handwritten strokes) into groups of recognition units in several different ways to create a segmented bundle 520. create. For example, assume that so far a total of nine strokes have been accumulated within the handwriting input area. According to the first division chain of the division bundle 520, strokes 1, 2, 3 are grouped into a first recognition unit 522, and strokes 4, 5, 6 are grouped into a second recognition unit 526. Ru. According to the second division chain of the division bundle 520, all strokes 1 to 9 are grouped into one recognition unit 526.

In some embodiments, each segmentation chain is given a segmentation score to assess the likelihood that a particular segmentation chain is a correct segmentation of the current handwriting input. In some embodiments, factors optionally used to calculate the splitting score for each splitting strand include absolute and/or relative size of the stroke, various directions (e.g., the relative and/or absolute span of the stroke in the y-direction, z-direction), the average and/or variation of the saturation level of the stroke, the absolute and/or relative distance to neighboring strokes, the absolute and/or absolute span of the stroke; or the relative location, the order or arrangement in which the strokes are entered, the duration of each stroke, the average and/or variation in the speed (or slowness or steepness) at which each stroke is entered, the intensity of each stroke along the length of the stroke. Examples include profiles. In some embodiments, one or more functions or transformations are optionally applied to one or more of these factors to generate split scores for various split chains within split bundle 520. Ru.

In some embodiments, after the splitting module 508 splits the current handwritten input 516 received from the user, the splitting module 508 passes the split bundle 520 to the normalization module 510. In some embodiments, normalization module 510 generates an input image (eg, input image 528) for each recognition unit specified in split bundle 520 (eg, recognition units 522, 524, and 526). In some embodiments, the normalization module performs any required or desired normalization (e.g., stretching, cropping, etc.) on the input image so that the input image can be provided as input to the handwriting recognition model 504. , downsampling, or upsampling). In some embodiments, each input image 528 includes strokes that are assigned to a respective recognition unit and corresponds to one character or radical recognized by the handwriting recognition module 504.

In some embodiments, the input image generated by input processing module 502 does not include any temporal information associated with individual strokes, but rather spatial information (e.g., location and density of pixels within the input image). only the information represented by ) is preserved in the input image. A handwriting recognition model trained purely on the spatial information of training writing samples is capable of handwriting recognition based only on spatial information. As a result, the handwriting recognition model is stroke order and stroke direction independent, and during training the handwriting recognition model is stroke order and stroke direction independent for all characters in its vocabulary (i.e., all output classes). There is no exhaustive enumeration of all possible permutations of direction. In fact, in some embodiments, handwriting recognition module 502 does not distinguish between pixels belonging to one stroke and pixels belonging to another stroke in the input image.

As described in more detail below (e.g., in connection with FIGS. 25A-27), in some embodiments, some temporally derived stroke distribution information may be derived from purely spatial handwriting. It is reintroduced into the recognition model to improve recognition accuracy without compromising the stroke order and stroke direction independence of the recognition model.

In some embodiments, for one recognition unit, the input image generated by input processing module 502 does not overlap with the input image of any other recognition unit within the same split chain. In some embodiments, input images generated for different recognition units may have some degree of overlap. In some embodiments, some degree of overlap between input images includes handwritten input written in a cursive typeface, and/or continuous characters (e.g., one stroke connects two adjacent characters) Acceptable for recognizing handwritten input.

In some embodiments, some degree of normalization is performed before splitting. In some embodiments, the functions of segmentation module 508 and normalization module 510 may be performed by the same module or two or more other modules.

In some embodiments, when the input image 528 for each recognition unit is provided as input to the handwriting recognition model 504, the handwriting recognition model 504 recognizes that the recognition unit is the repertoire or vocabulary of the handwriting recognition model 504 (i.e., generates an output consisting of various likelihoods of being each output character in a list of all characters and radicals that can be recognized by the handwriting recognition module 504; As described in more detail below, handwriting recognition model 504 is trained to recognize multiple characters within multiple scripts (e.g., at least three non-overlapping scripts encoded by the Unicode standard). has been done. Examples of non-duplicate scripts include Latin script, Chinese script, Arabic script, Persian script, Cyrillic script, and man-made scripts such as glyphs. In some embodiments, handwriting recognition model 504 generates one or more output characters for each input image (i.e., for each recognition unit) and determines whether each output character has a confidence level associated with the character recognition. Assign a recognition score to each.

In some embodiments, the handwriting recognition model 504 generates a candidate bundle 530 according to the split bundle 520 and corresponds to each arc in the split chain within the split bundle 520 (e.g., a respective recognition unit 522, 524, 526). ) is expanded into one or more candidate arcs (eg, arcs 532, 534, 536, 538, 540, each corresponding to a respective output character) within candidate bundle 530. Each candidate chain in candidate bundle 530 is scored according to the respective segmentation scores of the segmentation chains on which the candidate chain is based and the recognition scores associated with the output characters within that candidate chain.

In some embodiments, after the handwriting recognition model 504 generates an output character from the input image 528 of the recognition unit, the candidate bundle 530 is passed to the result generation module 506 and is passed to the result generation module 506 to provide information about the currently accumulated handwriting input 516. One or more recognition results are generated.

In some embodiments, the results generation module 506 utilizes a radical clustering module 512 to combine one or more radicals in the candidate chain into compound characters. In some embodiments, the results generation module 506 uses one or more language models 514 to ensure that the character chains in the candidate bundle 530 are properly arranged in the particular language represented by the language model. Determine whether it exists or not. In some embodiments, the results generation module 506 generates the corrected candidate bundle 542 by removing certain arcs or by combining two or more arcs within the candidate bundle 530.

In some embodiments, the results generation module 506 calculates the combined recognition score for each character sequence (e.g., character sequences 544 and 546) still maintained in the correction candidate bundle 542 by the radical clustering module 512 and the linguistic Generate based on the recognition scores of output characters within those character arrays as modified (eg, increased or decreased) by model 514. In some embodiments, the results generation module 506 ranks the various character sequences maintained within the correction candidate bundle 542 based on their combined recognition scores.

In some embodiments, the results generation module 506 sends the top ranked character sequence as a ranked recognition result 548 to the I/O interface module 500 for display to the user. In some embodiments, the I/O interface module 500 receives the received recognition results 548 (e.g., “
"as well as"
") is displayed in the candidate display area of the handwriting input interface. In some embodiments, the I/O interface module provides multiple recognition results (e.g., "
"as well as"
”) to the user, allowing the user to select the recognition results and enter them as text input for the associated application. In some embodiments, the I/O interface module selects the top-ranked recognition result (e.g., "
”) is automatically input in response to other input or instructions for user confirmation of the recognition result. Effective autofilling of top-ranked results can improve the efficiency of the input interface and provide a better user experience.

In some embodiments, the results generation module 506 uses other factors to modify the combined recognition score of the candidate strands. For example, in some embodiments, results generation module 506 optionally maintains a log of the most frequently used characters for a particular user or for a number of users. Results generation module 506 generates an integrated recognition score for a particular candidate character or character sequence if the particular candidate character or character sequence is found in the list of most frequently used characters or character sequences. Optionally raise.

In some embodiments, handwriting input module 157 provides real-time updates regarding the recognition results displayed to the user. For example, in some embodiments, for each additional stroke by the user, the input processing module 502 optionally subdivides the currently accumulated handwriting input into a segmented bundle provided to the handwriting recognition model 504. and correct the input image. Similarly, handwriting recognition model 504 optionally corrects the candidate bundle provided to results generation module 506. As a result, results generation module 506 optionally updates the recognition results presented to the user. As used herein, real-time handwriting recognition refers to handwriting recognition in which handwriting recognition results are presented to the user immediately or within a short period of time (eg, within tens of milliseconds to seconds). Real-time handwriting recognition starts immediately and runs virtually simultaneously with the receipt of handwriting input, rather than all at once after the current user session, from recorded images that are stored for later retrieval. This differs from offline recognition (eg, as in offline optical character recognition (OCR) applications) in that the Furthermore, since offline character recognition is performed without the use of any temporal information regarding individual strokes and stroke sequences, segmentation is performed without the benefit of such information. Further disambiguation between similar-looking candidate characters also does not benefit from such temporal information.

In some embodiments, handwriting recognition model 504 is implemented as a convolutional neural network (CNN). FIG. 6 shows an example convolutional neural network 602 trained on a multi-script training corpus 604 that includes writing samples for characters in multiple non-overlapping scripts.

As shown in FIG. 6, convolutional neural network 602 includes an input surface 606 and an output surface 608. Between the input surface 606 and the output surface 608 are a plurality of convolutional layers 610 (eg, including an initial convolutional layer 610a, zero or more intermediate convolutional layers (not shown), and a final convolutional layer 610n). Each convolutional layer 610 is followed by a respective subsampling layer 612 (eg, a first subsampling layer 612a, zero or more intermediate subsampling layers (not shown), and a final subsampling layer 612n). After these convolutional and subsampling layers, and just before the output surface 608, there is a hidden layer 614. Hidden layer 614 is the final layer before output surface 608. In some embodiments, a kernel layer 616 (e.g., including a first kernel layer 616a, zero or more intermediate kernel layers (not shown), and a final kernel layer 612n) is inserted before each convolutional layer 610. This improves computational efficiency.

As shown in FIG. 6, the input surface 606 receives the input image 614 of the handwritten recognition unit (eg, handwritten characters or a brainer), and the output surface 608 is a corresponding output class (recognized unit). For example, it outputs a set of probabilities indicating the likelihood of belonging to a particular character in the set of output characters that the neural network is configured to recognize. The entire output class of this neural network (or set of output characters of the neural network) is also referred to as the repertoire or vocabulary of the handwriting recognition model. The convolutional neural networks described herein can be trained to have a repertoire of tens of thousands of characters.

As the input image 614 is processed through the various layers of the neural network, various spatial features embedded within the input image 614 are extracted by the convolution layer 610. Each convolutional layer 610 is also referred to as a set of feature maps and acts as a filter to find specific features within the input image 614 to distinguish between images corresponding to different characters. Sub-sampling layer 612 ensures that features are captured at increasing scales from input image 614. In some embodiments, subsampling layer 612 is implemented using a max pooling technique. This max pooling layer creates positional invariance over a larger local area and downsamples the output image of the previous convolutional layer by a factor of Kx and Ky along each direction (Kx and Ky is the size of the maximum pooling rectangle). Maximum pooling provides faster convergence speed and improves generalization ability by selecting top invariant features. In some embodiments, subsampling is accomplished using other methods.

In some embodiments, after the final set of convolutional layers 610n and subsampling 612n, there is a fully connected layer, or hidden layer 614, before the output surface 608. Fully connected hidden layer 614 is a multilayer perceptron that fully connects nodes in final subsampling layer 612n and nodes in output surface 608. Hidden layer 614 takes the output image received from the previous layer and arrives at one of the output characters in output layer 608 through logistic regression.

During training of the convolutional neural network 602, the features in the convolutional layer 610 and the respective weights associated with those features, as well as the weights associated with the parameters in the hidden layer 614, are assigned to the known output classes in the training corpus 604. is adjusted such that the classification error is minimized for written samples with . Once the convolutional neural network 602 has been trained and optimal sets of parameters and associated weights have been established for the various layers within the network, the convolutional neural network 602 can be used to calculate real-time handwriting input received from a user. New writing samples 618 that are not part of the training corpus 604 can be recognized, such as input images generated based on the training corpus 604 .

As described herein, the convolutional neural network of the handwriting input interface is trained using a multi-script training corpus to enable multi-script or mixed-script handwriting recognition. In some embodiments, the convolutional neural network is trained to recognize a wide repertoire of 30,000 to over 60,000 characters (eg, all characters encoded according to the Unicode standard). Most state-of-the-art handwriting recognition systems are based on stroke order-dependent hidden Markov methods (HMMs). Moreover, most existing handwriting recognition models are language-specific and can only be used from a small repertoire of a few dozen characters (e.g., the English alphabet, the Greek alphabet, all 10 numbers, etc.). Contains up to several thousand characters (e.g., the most commonly used set of Chinese characters). Therefore, the general purpose recognizer described herein can handle several orders of magnitude more characters than most existing systems.

Some conventional handwriting systems may include several individually trained handwriting recognition models, each tailored for a particular language or small set of characters. The writing samples are propagated through various recognition models until they can be classified. For example, a handwriting sample can be provided to a series of concatenated language-specific or script-specific character recognition models, and if the handwriting sample cannot be conclusively classified by the first recognition model, then The next recognition model attempts to classify the handwriting sample within its own repertoire. This approach to classification is time consuming and the memory requirements increase rapidly with each additional recognition model that needs to be employed.

Other state-of-the-art models require the user to specify a preferred language and use the selected handwriting recognition model to classify the current input. Such implementations are cumbersome to use, consume significant memory, and cannot be used to recognize mixed language input. It is impractical to require a user to switch language settings in the middle of inputting mixed languages or mixed scripts.

The multi-script or universal recognizer described herein addresses at least some of the above problems with conventional recognition systems. FIG. 7 shows how a large-scale multi-script training corpus is used to train a handwriting recognition module (e.g., a convolutional neural network) to perform real-time multi-language and multi-script handwriting on a user's handwriting input. 7 is a flow diagram of an example process 700 for subsequent use to provide recognition.

In some embodiments, training of the handwriting recognition model is performed on the server device and then the trained handwriting recognition model is provided to the user device. This handwriting recognition model optionally performs real-time handwriting recognition locally on the user device without requiring further assistance from a server. In some embodiments, both training and recognition are provided on the same device. For example, a server device can receive a user's handwriting input from a user device, perform handwriting recognition, and send the recognition results to the user device in real time.

In the example process 700, a device having one or more processors and memory performs a multi-script handwriting process based on spatially derived features (e.g., stroke-order independent features) of a multi-script training corpus. A recognition model is trained (702). In some embodiments, the spatially derived features of the multi-script training corpus are stroke order independent and stroke direction independent (704). In some embodiments, training of the multi-script handwriting recognition model is independent of temporal information associated with each stroke within the handwriting samples (706). Specifically, images of handwriting samples are normalized to a predetermined size, and the images do not contain any information about the order in which individual strokes are entered to form the image. Moreover, the images also do not contain any information about the direction in which the individual strokes are input to form the image. In fact, during training, features are extracted from handwritten images regardless of how the images are formed in time by individual strokes. Therefore, no temporal information related to individual strokes is required during recognition. As a result, the recognition robustly provides consistent recognition results despite delayed, out-of-order strokes, and arbitrary stroke directions within the handwritten input.

In some embodiments, the multi-script training corpus includes handwriting samples corresponding to characters of at least three non-overlapping scripts. As shown in Figure 6, the multi-script training corpus includes handwriting samples collected from many users. Each handwriting sample corresponds to one character of the respective script represented in the handwriting recognition model. In order to properly train the handwriting recognition model, the training corpus contains a large number of writing samples for each character of the script represented within the handwriting recognition model.

In some embodiments, the at least three non-overlapping scripts include Chinese characters, emoji, and Latin script (708). In some embodiments, the multi-script handwriting recognition model has at least 30,000 output classes representing 30,000 characters across at least three non-overlapping scripts (710).

In some embodiments, the multi-script training corpus includes all or a substantial portion of all Chinese characters encoded in the Unicode standard (e.g., all CJK (Chinese-Japanese-Korean) integrated Chinese characters). Contains a respective writing sample for each character in the section). The Unicode standard defines a total of approximately 74,000 CJK unified kanji. The CJK Integrated Kanji Basic Block (4E00-9FFF) includes 20,941 basic Chinese characters used in Chinese, as well as Japanese, Korean, and Vietnamese. In some embodiments, the multi-script training corpus includes writing samples for all characters in the basic block of CJK unified Kanji. In some embodiments, the multi-script training corpus further includes writing samples for CJK radicals that can be used to structurally compose one or more compound Chinese characters. In some embodiments, the multi-script training corpus further includes writing samples for less frequently used Chinese characters, such as Chinese characters encoded with one or more of the CJK unified Chinese character extensions.

In some embodiments, the multi-script training corpus further includes a respective writing sample for each character of all characters in the Latin script encoded by the Unicode standard. Characters in the basic Latin script include Latin uppercase and Latin lowercase letters, as well as basic symbols and numbers commonly used on standard Latin keyboards. In some embodiments, the multi-script training corpus further includes characters in the extended Latin script (eg, various emphasized forms of the basic Latin script).

In some embodiments, the multi-script training corpus includes writing samples corresponding to each character of the artificial script that is not associated with any natural human language. For example, in some embodiments, a set of emojis is optionally defined in an emoji script, and writing samples corresponding to each of those emojis are included in a multi-script training corpus. For example, the hand-drawn heart symbol is the emoji in the training corpus.
This is a handwritten sample regarding ``. Similarly, a hand-drawn smile (e.g., two points above an upturned arc) is the emoji in the training corpus.
This is a handwritten sample regarding ``. Other emojis include various emotions (e.g., joy, sadness, anger, embarrassment, surprise, laughter, crying, irritation, etc.), various objects and characters (e.g., cat, dog, rabbit, heart, fruit, eyes, icons representing various actions (e.g., shaking hands, kissing, running, dancing, jumping, sleeping, eating, meeting, loving, liking, voting, etc.) There are several categories. In some embodiments, the strokes in the handwriting sample that correspond to glyphs are simplified and/or stylized lines of the actual lines that form the corresponding glyph. In some embodiments, each device or application may use different designs for the same emoji. For example, a smiling emoji presented to a female user may be different from a smiling emoji presented to a male user, even if the handwritten input received from those two users is substantially the same. It can be different.

In some embodiments, the multi-script training corpus also includes Greek script (e.g., includes Greek letters and symbols), Cyrillic script, Hebrew script, and one or more others encoded according to the Unicode standard. It also includes writing samples for characters in other scripts, such as the script. In some embodiments, the at least three non-redundant scripts included in the multi-script training corpus include Chinese characters, emojis, and characters in Latin script. Chinese characters, pictograms, and characters in Latin script are inherently non-overlapping scripts. Many other scripts may overlap each other, at least with respect to some characters. For example, some characters in Latin script (eg, A, Z) can be found in many other scripts (eg, Greek, and Cyrillic). In some embodiments, the multi-script training corpus includes Chinese scripts, Arabic scripts, and Latin scripts. In some embodiments, the multi-script training corpus includes other combinations of overlapping and/or non-overlapping scripts. In some embodiments, the multiscript training corpus includes writing samples for all characters encoded by the Unicode standard.

As shown in FIG. 7, in some embodiments, to train a multi-script handwriting recognition model, the device combines a single convolutional neural network with a single input surface and a single output surface. , provides handwriting samples of the multi-script training corpus (712). The device comprises spatially derived features (e.g., stroke order independent features) of handwriting samples for distinguishing between characters of at least three non-overlapping scripts represented in a multi-script training corpus; and respective weights for the spatially derived features are determined using a convolutional neural network (714). This multi-script handwriting recognition model has a single input surface and a single output surface, in that a single handwriting recognition model is trained using all the samples in the multi-script training corpus. This is different from traditional multi-script handwriting recognition models. A single convolutional neural network is divided into separate subnetworks, each handling a small subset of the training corpus (e.g., each trained on a particular script or character used within a particular language). It is trained to identify all characters represented in the multi-script training corpus, independently. Furthermore, this single convolutional neural network can handle characters from multiple non-overlapping scripts, rather than characters from a small number of overlapping scripts (e.g., with overlapping characters A, B, E, Z, etc.), such as Latin and Greek scripts. trained to identify a large number of characters over a range of characters.

In some embodiments, the device provides real-time handwriting recognition for the user's handwriting input using a multi-script handwriting recognition model that is trained on spatially derived features of a multi-script training corpus ( 716). In some embodiments, providing real-time handwriting recognition for a user's handwriting input includes continuously providing recognition output for the user's handwriting input as the user continues to provide additions and corrections to the handwriting input. This includes making corrections. In some embodiments, providing real-time handwriting recognition for the user's handwriting input further includes providing a multi-script handwriting recognition model to the user device (718), the user device receiving handwriting input from the user. and locally perform handwriting recognition on the handwritten input based on the multi-script handwriting recognition model.

In some embodiments, the device provides a multi-script handwriting recognition model to the plurality of devices for which there is no overlap in their respective input languages, and the multi-script handwriting recognition model is associated with each of those user devices. used in each of those multiple devices for handwriting recognition in different languages. For example, if a multi-script handwriting recognition model is trained to recognize characters in many different scripts and languages, that same handwriting recognition model can be used around the world to recognize characters in many different scripts and languages. It is possible to provide handwritten input regarding information. A first device for users who only want to type in English and Hebrew can perform handwriting input using the same handwriting recognition model as a second device for users who only want to type in Chinese and emojis. functionality. A user of the first device has an English handwriting input keyboard (e.g., one implemented with an English-specific handwriting recognition model) and a separate Hebrew handwriting input keyboard (e.g., one implemented with a Hebrew-specific handwriting recognition model). Once installed on the first device, the same generic multi-script handwriting recognition model provides handwriting input functionality for both English and Hebrew, as well as can be used to provide mixed input in multiple languages. Additionally, the second user may have a Chinese handwriting input keyboard (e.g., one implemented with a Chinese-specific handwriting recognition model) and a separate emoji handwriting input keyboard (e.g., one implemented with an emoji handwriting recognition model). Once installed on the second device, the same generic multi-script handwriting recognition model provides handwriting input functionality for both Chinese and emoji, as well as both scripts. can be used to provide mixed input. By using the same multi-script handwriting model to handle a wide repertoire across multiple scripts (e.g., a substantial portion or all of the characters encoded in close to 100 different scripts), device suppliers and users The usefulness of the recognizer is increased without imposing any substantial burden on the user.

Training a multi-script handwriting recognition model using a large multi-script training corpus is different from traditional HMM-based handwriting recognition systems and does not rely on temporal information associated with individual strokes of a character. . Furthermore, the resource and memory requirements for the multi-script recognition system do not increase proportionately as the number of symbols and languages covered by the multi-script recognition system increases. For example, in a traditional handwriting system, an increase in the number of languages means adding another independently trained model, and memory requirements increase to adapt to the increasing capabilities of that handwriting recognition system. It will be at least doubled. In contrast, if a multiscript model is trained with a multiscript training corpus, increasing language coverage requires retraining the handwriting recognition model using additional handwriting samples. , increases the size of the output surface, but the amount of increase is very modest. The multi-script training corpus contains handwriting samples corresponding to n different languages, the multi-script handwriting recognition model occupies memory of size m, and increases the language coverage to N (N>n) languages. If the device retrains the multi-script handwriting recognition model based on the spatially derived features of a second multi-script training corpus, the device retrains the multi-script handwriting recognition model based on the spatially derived features of a second multi-script training corpus, , and a second handwriting sample corresponding to . When the change in N/n is between 1 and 100, the change in M/m remains substantially constant within the range of 1 and 2. After the multi-script handwriting recognition model is retrained, the device can use the retrained multi-script handwriting recognition model to provide real-time handwriting recognition for the user's handwriting input.

8A and 8B illustrate example user interfaces for providing real-time multi-script handwriting recognition and input on a portable user device (eg, device 100). 8A and 8B, a handwriting input interface 802 is displayed on a touch-sensitive display screen (eg, touch screen 112) of a user device. Handwriting input interface 802 includes a handwriting input area 804 , a candidate display area 806 , and a text input area 808 . In some embodiments, the handwriting input interface 802 further includes a plurality of control elements, each of which can be called to cause the handwriting input interface to perform a predetermined function. As shown in FIG. 8A, the handwriting input interface includes a delete button, a space button, a carriage return or enter button, and a keyboard toggle button. Other control elements are also possible and can optionally be provided within the handwriting input interface to suit each different application that utilizes the handwriting input interface 802. The layout of the various components of handwriting input interface 802 is merely exemplary and may vary with different devices and different applications.

In some embodiments, handwriting input area 804 is a touch sensitive area for receiving handwriting input from a user. A sustained contact on the touch screen within the handwriting input area 804 and its associated path of movement is registered as a handwritten stroke. In some embodiments, the handwritten strokes registered by the device are visually rendered within the handwriting input area 804 at the same location traced by the sustained touch. As shown in FIG. 8A, the user enters some handwritten Chinese characters (e.g., "
''), some handwritten English characters (e.g., "Happy"), and hand-drawn emojis (e.g., a smiley face). These handwritten characters are distributed within multiple lines (eg, two lines) within handwriting input area 804.

In some embodiments, candidate display area 806 displays one or more recognition results (eg, 810 and 812) for the handwriting input currently stored within handwriting input area 804. Generally, the highest ranking recognition result (eg, 810) is displayed in the first position within the candidate display area. As shown in Figure 8A, the handwriting recognition model described herein is capable of recognizing characters from multiple non-overlapping scripts, including Chinese characters, Latin scripts, and emojis; The recognition results (eg, 810) provided by the recognition model accurately include Chinese characters, English characters, and emojis represented by handwritten input. The user is not required to interrupt writing input in order to select or switch recognition languages.

In some embodiments, text input area 808 is an area for displaying text input provided to each application employing this handwriting input interface. As shown in FIG. 8A, text entry area 808 is used by the notes application to include the text currently shown within text entry area 808 (e.g., "America
”) is a text input already provided to the Notes application. In some embodiments, cursor 813 indicates the current text entry position within text entry area 808.

In some embodiments, the user may, for example, make an explicit selection input (e.g., a tap gesture on one of the displayed recognition results) or an implicit confirmation input (e.g., an "Enter" button). a tap gesture above or a double tap gesture within the handwriting input area) can select a particular recognition result displayed within the candidate display area 806. As shown in FIG. 8B, the user explicitly selects the top-ranked recognition result 810 using a tap gesture (as indicated by the contact 814 on the recognition result 810 in FIG. 8A). Selected. In response to this selection input, the text of the recognition result 810 is inserted into the text input area 808 at the insertion point indicated by the cursor 813. As shown in FIG. 8B, when the text of the selected recognition result 810 is entered into the text input area 808, the handwriting input area 804 and the candidate display area 806 are both cleared. At this point, the handwriting input area 804 is ready to accept new handwriting input, and the candidate display area 806 can now be used to display recognition results for the new handwriting input. . In some embodiments, the implicit confirmation input causes the top-ranked recognition result to be entered into the text entry area 808 without requiring the user to interrupt and select the top-ranked recognition result. Well-designed implicit confirmation input speeds up text entry and reduces the cognitive burden imposed on the user during text composition.

In some embodiments (not shown in FIGS. 8A, 8B), the highest ranking recognition result of the current handwriting input is optionally provisionally displayed within the text input area 808. The provisional text entry shown within the text entry area 808 is visually distinguished from other text entries within the text entry area by, for example, a provisional entry box surrounding the provisional text entry. The text shown in this provisional input box has not yet been finalized or provided to the associated application (e.g. a notes application) and may be e.g. , will be automatically updated when the top-ranked recognition result is changed by the handwriting input module.

9A-9B are a flowchart of an example process 900 for providing multi-script handwriting recognition on a user device. In some embodiments, as shown in diagram 900, a user device receives (902) a multi-script handwriting recognition model that includes spatially derived features of a multi-script training corpus. (e.g., stroke order and stroke direction independent features), and the multi-script training corpus includes handwriting samples corresponding to characters of at least three non-overlapping scripts. In some embodiments, the multi-script handwriting recognition model is a single convolutional neural network (906) with a single input surface and a single output surface, and is represented in a multi-script training corpus. It includes spatially derived features and respective weights for those spatially derived features for distinguishing between characters of at least three non-overlapping scripts. In some embodiments, the multi-script handwriting recognition model is configured to recognize characters (908) based on respective input images of one or more recognition units identified within the handwriting input; Each spatially derived feature used for is independent of the respective stroke order, stroke direction, and stroke continuity within the handwriting input.

In some embodiments, the user device receives (908) handwriting input from the user, the handwriting input comprising one or more handwritten strokes provided on a touch-sensitive surface coupled to the user device. . For example, handwriting input includes data about the location and movement of contact between a finger or stylus and a touch-sensitive surface coupled to a user device, respectively. In response to receiving this handwriting input, the user device generates one or more handwriting recognition results in real-time based on a multi-script handwriting recognition model that is trained on the spatially derived features of the multi-script training corpus. (910) (912).

In some embodiments, when providing real-time handwriting recognition results to a user, the user device divides the user's handwriting input into one or more recognition units (914), and each recognition unit is Contains one or more of the provided handwritten strokes. In some embodiments, the user device segments the user's handwriting input according to the shape, location, and size of discrete strokes produced by contact between the user's finger or stylus and a touch-sensitive surface of the user device. do. In some embodiments, this handwriting input segmentation further takes into account the relative order and position of individual strokes produced by contact between the user's finger or stylus and the touch-sensitive surface of the user device. In some embodiments, the user's handwritten input is in a cursive typeface, and each successive stroke in the handwritten input may correspond to multiple strokes in the recognized character in printed form. In some embodiments, the user's handwritten input may include continuous strokes across multiple recognized characters in printed form. In some embodiments, segmenting the handwritten input produces one or more input images, each corresponding to each recognition unit. In some embodiments, some of the input images optionally include some overlapping pixels. In some embodiments, the input images do not include any overlapping pixels. In some embodiments, the user device generates a split bundle, with each split chain of the split bundle representing a respective manner in which the current handwriting input is split. In some embodiments, each arc in the split chain corresponds to a respective group of strokes in the current handwriting input.

As shown in diagram 900, a user device provides (914) a respective corresponding image of one or more recognition units as input to a multi-script recognition model. For at least one of the one or more recognition units, the user device generates at least a first output character from a first script and a second output character different from the first script from the multi-script handwriting recognition model. At least a second output character from the script is obtained (916). For example, the same input image can cause a multi-script recognition model to output two or more similar-looking output characters from different scripts as recognition results for that same input image. For example, handwritten input for the letter "a" in Latin script and the letter "α" in Greek script are often similar. Furthermore, the letter “J” in Latin script and the Chinese letter “
” handwritten inputs are often similar. Similarly, the emoji “
”, the handwritten input for the CJK radical “
” may be similar to handwritten input. In some embodiments, the multi-script handwriting recognition model generates multiple candidate recognition results that are likely to correspond to the user's handwriting input, often because the visual appearance of the handwriting input is , even a human reader would have difficulty deciphering it. In some embodiments, the first script is a CJK basic character block and the second script is a Latin script, such as encoded by the Unicode standard. In some embodiments, the first script is a CJK basic character block and the second script is a set of glyphs. In some embodiments, the first script is a Latin script and the second script is an emoticon.

In some embodiments, the user device displays both the first output character and the second output character within a candidate display area of the handwriting input interface of the user device (918). In some embodiments, the user device determines whether one of the first script and the second script is a corresponding script used with a soft keyboard currently installed on the user device. selectively displays one of the first output character and the second output character based on (920). For example, the handwriting recognition model selects the Chinese character "
” and the Greek letter “λ”, the user device has either a Chinese soft keyboard (e.g., a keyboard that uses the Pinyin input method) or a Greek input keyboard on the user device. Determine whether it is installed. If the user device determines that only a Chinese soft keyboard is installed, the user device may optionally display Chinese characters as a recognition result to the user.
", but not the Greek letter "λ".

In some embodiments, the user device provides real-time handwriting recognition and input. In some embodiments, the user device, in response to continued additions or corrections to the handwritten input by the user, before the user performs an explicit or implicit selection of the recognition results displayed to the user. One or more recognition results for the user's handwritten input are continually corrected (922). In some embodiments, in response to each correction of the one or more recognition results, the user displays the respective corrected one or more recognition results to the user in a candidate display area of the handwriting input user interface. (924).

In some embodiments, the multi-script handwriting recognition model is configured to recognize all characters of at least three non-redundant scripts, including Chinese characters, emojis, and Latin scripts, that are encoded according to the Unicode standard. Trained (926). In some embodiments, the at least three non-overlapping scripts include a Chinese script, an Arabic script, and a Latin script. In some embodiments, the multi-script handwriting recognition model has at least 30,000 output classes representing at least 30 characters across at least three non-overlapping scripts (928).

In some embodiments, the user device allows the user to enter multi-script handwritten input, such as words that include characters from more than one script. For example, a user may write continuously, without interrupting mid-writing, to manually switch recognition languages, and receive handwriting recognition results that include characters from two or more scripts. For example, a user may write the multiscript sentence "Hello means" within the handwriting input area of the user device.
in Chinese. ”, in which case the Chinese character “
It is not necessary to switch the input language from English to Chinese before writing the word "in Chinese," or to switch the input language from Chinese back to English when writing the English word "in Chinese."

As described herein, a multi-script handwriting recognition model is used to provide real-time handwriting recognition for user input. In some embodiments, real-time handwriting recognition is used to provide real-time multi-script handwriting input functionality on a user's device. 10A-10C are a flowchart of an example process 1000 for providing real-time handwriting recognition and input on a user device. Specifically, this real-time handwriting recognition is stroke order independent at the character level, phrase level, and sentence level.

In some embodiments, stroke order-independent handwriting recognition at the character level is such that, regardless of the arrangement of the individual strokes of a particular character provided by the user, the handwriting recognition model It is required to provide the same recognition results. For example, the individual strokes of Chinese characters are typically written in a particular order. Although native Chinese speakers are often trained in school to write each character in a specific order, many users develop personal styles and patterns that deviate from that traditional stroke order. Stroke arrangement was adopted later. Furthermore, cursive typefaces are highly individualized, and the multiple strokes in printed forms of Chinese characters are often fused into a single stylized stroke, which can be distorted or distorted. , the direction may be changed, and even the next letter may be continued. Stroke order independent recognition models are trained on images of writing samples that do not contain temporal information associated with individual strokes. Therefore, this recognition is independent of stroke order information. For example, the Chinese character “
”, regardless of whether the user writes the horizontal stroke first or the vertical stroke first, the same recognition result “
” is given by the handwriting recognition model.

As shown in FIG. 10A, in process 1000, a user device receives 1002 a plurality of handwritten strokes from a user, the plurality of handwritten strokes corresponding to one handwritten character. For example, the character "
” typically includes substantially horizontal handwritten strokes intersecting substantially vertical handwritten strokes.

In some embodiments, the user device generates an input image based on the plurality of handwritten strokes (1004). In some embodiments, the user device provides an input image to a handwriting recognition model (1006) to perform real-time handwriting recognition of the handwriting, and the handwriting recognition model includes stroke order independent handwriting. Provide recognition. The user device then processes the same first and second handwritten strokes (e.g., horizontal strokes and vertical strokes) received from the user in real time, regardless of their respective order. 1 output character (for example, the character "
”) is displayed (1008).

In some conventional handwriting recognition systems, minor stroke order variations in a small number of characters are tolerated by specifically including such variations in the training of the handwriting recognition system. Such traditional handwriting recognition systems are not scalable to accommodate arbitrary stroke order variations in large numbers of complex characters, such as Chinese characters; Even so, this already gives rise to numerous variations in stroke order. Furthermore, by simply including more permutations of acceptable stroke order for a particular character, traditional recognition systems combine multiple strokes into a single stroke (e.g., hypercursive Handwritten input (as in writing) or where one stroke is decomposed into multiple sub-strokes (as in characters captured with a very coarse sampling of the input strokes) can still be handled. It's impossible. Therefore, multi-script handwriting systems trained on spatially derived features, such as those described herein, offer advantages over traditional recognition systems.

In some embodiments, stroke order independent handwriting recognition is performed independently of temporal information associated with individual strokes within each handwritten character. In some embodiments, stroke order-independent handwriting recognition is performed in conjunction with stroke distribution information that takes into account the spatial distribution of individual strokes before being fused into a flat input image. Ru. Further details on how this temporally derived stroke distribution information is used to enhance the stroke order independent handwriting recognition described above are provided below (e.g., in FIGS. 25A-27). (related to) provided herein. The techniques described in connection with FIGS. 25A-27 do not compromise the stroke order independence of handwriting recognition systems.

In some embodiments, the handwriting recognition model provides stroke direction independent handwriting recognition (1010). In some embodiments, stroke direction-independent recognition includes, in response to receiving the plurality of handwriting inputs, the user device, in response to receiving the plurality of handwriting inputs, recognizes each of the plurality of handwritten strokes provided by the user, regardless of the respective stroke direction. , it is required to display the same first output character. For example, if a user enters the Chinese characters "
”, the handwriting recognition model outputs the same recognition result regardless of whether the user drew the horizontal stroke from left to right or from right to left. Similarly, handwriting recognition models output the same recognition results regardless of whether the user draws vertical strokes in a downward or upward direction. In another example, many Chinese characters are constructed structurally with two or more radicals. Some Chinese characters each include a left radical and a right radical, and people traditionally write the left radical first and the right radical second. In some embodiments, the handwriting recognition model recognizes that when a user completes a handwritten character, as long as the resulting handwriting input shows left radicals to the left of right radicals, Provides the same recognition result whether the left radical is written first or the right radical is written first. Similarly, some Chinese characters each contain an upper radical and a lower radical, and people traditionally write the upper radical first and the lower radical last. In some embodiments, the handwriting recognition model determines whether the user wrote the upper radical first, or Provides the same recognition result regardless of whether the lower radical is written first. In other words, the handwriting recognition model determines the identity of handwritten characters without depending on the direction of individual strokes of the handwritten characters provided by the user.

In some embodiments, the handwriting recognition model provides handwriting recognition based on an image of the recognition unit regardless of the number of substrokes of the recognition unit provided by the user. In other words, in some embodiments, the handwriting recognition model provides stroke count independent handwriting recognition (1014). In some embodiments, the user device, in response to receiving the plurality of handwritten strokes, determines how many handwritten strokes are identical, regardless of how many handwritten strokes are used to form a continuous stroke in the input image. Display the first output character. For example, if the user writes the Chinese character "+" in the handwriting input area, the handwriting recognition model will write the Chinese character "+" in the handwriting input area.
' provided four strokes (e.g. two short horizontal strokes and two short vertical strokes to form a cross-shaped character) or two strokes (e.g. , L-shaped strokes and 7-shaped strokes, or horizontal strokes and vertical strokes), or any other number of strokes (e.g., hundreds of extremely short strokes or points). Outputs the same recognition result regardless of the

In some embodiments, the handwriting recognition model is not only capable of recognizing the same characters regardless of the order, direction, and number of strokes in which each single character is written; It is also possible to recognize multiple characters regardless of the temporal order of the strokes of the multiple characters provided by the user.

In some embodiments, the user device not only receives the first plurality of handwritten strokes from the user, but also receives the second plurality of handwritten strokes (1016) and the second plurality of handwritten strokes. The plurality of handwritten strokes correspond to a second handwritten character. In some embodiments, the user device generates a second input image based on the second plurality of handwritten strokes (1018). In some embodiments, the user device provides the second input image to the handwriting recognition model to perform real-time recognition of the second handwritten characters (1020). In some embodiments, the user device displays a second output character corresponding to the second plurality of handwritten strokes in real time upon receiving the second plurality of handwritten strokes (1022). In some embodiments, the second output character and the first output character are independent of the respective order of the first plurality of handwritten strokes and the second plurality of handwritten strokes provided by the user. displayed simultaneously in a spatial array. For example, if a user enters two Chinese characters (e.g., "
"as well as"
''), the user device determines that the handwriting input currently stored within the handwriting input area is the character ``
To the left of the stroke related to ``, the character ``
” as long as the user indicates a stroke related to the character “
” stroke first, or the character “
” regardless of whether you wrote the stroke first, the recognition result “
" is displayed. In fact, when the user inputs the characters ``
” before some of the strokes (e.g. vertical strokes), the character “
Even if you are writing some of the strokes (for example, strokes slanted to the left) that relate to the character "
To the left of all strokes related to ``
”, the user device recognizes the recognition result “
” is shown.

In other words, as shown in FIG. 10B, in some embodiments, the spatial arrangement of the first output character and the second output character is a 1024 corresponds to a spatial distribution of the first plurality of handwritten strokes and the second plurality of strokes along a writing direction (from to the right). In some embodiments, the second plurality of handwritten strokes is received later in time than the first plurality of handwritten strokes (1026), and the second output character is a second plurality of handwritten strokes of a handwriting input interface of a user device. A spatial alignment along the default (eg, left-to-right) writing direction that precedes the first output character.

In some embodiments, the handwriting recognition model provides stroke order independent recognition at the sentence-to-sentence level. For example, in the handwriting input area, the handwritten character "
” exists in the first handwritten text, and the handwritten text “
'' in the second handwritten text, and the two handwritten characters are separated by one or more other handwritten characters and/or words, the handwriting recognition model still recognizes their Put the two characters into a spatial arrangement ``
” will still be provided. This recognition result and the spatial arrangement of the two recognized characters are such that when the user completes handwriting input, the recognition units for those two characters are arranged in the array "
'' remains the same regardless of the temporal order of the strokes of those two characters provided by the user. In some embodiments, the first handwritten character (e.g., "
”) is the first handwritten text (e.g., “
is a number. '') and the second handwritten character (e.g., ``
”) is the second handwritten text (e.g., “
is another number. ”) provided by the user, the first handwritten text and the second handwritten text are displayed simultaneously within the handwriting input area of the user device. In some embodiments, this recognition result (e.g., "
is a number.
is another number. ") is the correct recognition result, those two sentences are entered into the text input area of the user device, and the handwriting input area is opened for the user to input another handwritten input. , cleared.

In some embodiments, the handwriting recognition model is stroke order independent, not only at the character level, but also at the word and sentence level, so that the user can make corrections to previous unfinished characters, as well as subsequent characters. It can be executed after writing. For example, if a user forgets to write a particular stroke for a character before moving on to writing one or more subsequent characters within the handwriting input area, the user may still You can fill in the missing strokes at the correct location within the character later and receive the correct recognition result.

In traditional stroke order dependent systems (e.g. HMM-based recognition systems), once a character is written, it is fixed and the user can no longer perform any changes to it. . If the user wishes to make any changes, he must delete that character and all subsequent characters and start over. Some traditional recognition systems require the user to complete a handwritten character within a short predefined time window, and any strokes entered outside that predefined time window are It cannot be included within the same recognition unit as other strokes provided between windows. Such conventional systems are difficult to use and cause a great deal of frustration to users. Stroke order independent systems are free from these drawbacks and allow the user to complete characters in any order and in any time frame as the user sees fit. The user may also perform modifications to previously written characters (eg, adding one or more strokes) after subsequently writing the one or more characters in the handwriting input interface. In some embodiments, the user can also individually delete previously written characters (e.g., using the methods described below in connection with FIGS. 21A, 22B) within the handwriting input interface. You can also rewrite the character in the same place.

As shown in FIGS. 10B and 10C, the second plurality of handwritten strokes spatially follows the first plurality of handwritten strokes along the default writing direction of the handwriting input interface of the user device ( 1028), the second output character follows the first output character in a spatial arrangement along its default writing direction within the candidate display area of the handwriting input interface. The user device receives (1030) a third handwritten stroke from the user to correct the first handwritten character (i.e., the handwritten character formed by the first plurality of handwritten strokes); The handwritten strokes are received later in time than the first plurality of handwritten strokes and the second plurality of handwritten strokes. For example, a user may enter two characters in a left-to-right spatial arrangement (e.g., "
”) is written. The first plurality of strokes is the handwritten character "
” is formed. The user is actually writing the character "
” but missed one stroke. The second plurality of strokes is the handwritten character "
” is formed. If the user says “
"not"
”, the user simply writes the characters “
Another vertical stroke can be written below the stroke for ", and the user device writes that vertical stroke to the first recognition unit (e.g., "
”). The user device generates a new output character (e.g., "
”), and this new output character is the same as the previous output character in the recognition result (for example, “
”) will be replaced. As shown in FIG. 10C, in response to receiving the third handwritten stroke, the user device determines the third handwritten stroke based on the relative proximity of the third handwritten stroke to the first plurality of handwritten strokes. The third handwritten stroke is assigned to the same recognition unit as the first plurality of handwritten strokes (1032). In some embodiments, the user device generates a corrected input image based on the first plurality of handwritten strokes and the third handwritten strokes (1034). The user device provides the corrected input image to the handwriting recognition model to perform real-time recognition of the corrected handwriting (1036). In some embodiments, the user device displays (1040) a third output character corresponding to the corrected input image in response to receiving the third handwritten input, the third output character comprising: It replaces the first output character and is displayed simultaneously with the second output character in a spatial arrangement along the default writing direction.

In some embodiments, the handwriting recognition module recognizes handwritten input written in a default writing direction of left to right. For example, a user can write characters from left to right and in one or more lines. In response to this handwriting input, the handwriting input module presents a recognition result, optionally including characters in a left-to-right spatial arrangement and in one or more rows. If the user selects a recognition result, the selected recognition result is entered into the text entry area of the user device. In some embodiments, the default writing direction is top to bottom. In some embodiments, the default writing direction is right to left. In some embodiments, the user optionally changes the default writing direction to an alternative writing direction after the recognition result is selected and the handwriting input area is cleared.

In some embodiments, the handwriting input module allows a user to enter multi-character handwritten input within the handwriting input area, and from the handwriting input at one time rather than all recognition units at once. Allows deletion of strokes in one recognition unit. In some embodiments, the handwriting input module allows deletion of one stroke at a time from the handwriting input. In some embodiments, the deletion of recognition units proceeds one by one in a direction opposite to the default writing direction, regardless of the order of the recognition units or strokes entered to produce the current handwriting input. do. In some embodiments, the deletion of strokes proceeds one by one in the reverse order in which the strokes were entered within each recognition unit, and when all strokes within a recognition unit are deleted, the strokes are deleted. Deletion proceeds to the next recognition unit in the opposite direction to the default write direction.

In some embodiments, while the third output character and the second output character are simultaneously displayed as candidate recognition results in the candidate display area of the handwriting input interface, the user device receives a delete input from the user. receive. In response to the delete input, the user device deletes the second output character from the recognition result while maintaining the third output character within the recognition result displayed within the candidate display area.

In some embodiments, as shown in FIG. Render in real time as provided by the user (1042). In some embodiments, in response to receiving the delete input from the user, the user device renders each of the second plurality of handwritten inputs (e.g., corresponding to the second handwritten character) from the handwriting input area. while deleting respective renderings of the first plurality of handwritten strokes and the third handwritten strokes (e.g., corresponding together to the corrected first handwritten strokes) within the handwriting input area. Maintain (1044). For example, if a user creates a character array '
If the user enters a delete input after providing the missing vertical stroke within the character "
" are removed from the handwriting input area and the strokes within the recognition unit for the character "
” is the recognition result in the candidate display area of the user device “
' will be removed from '. After this deletion, the characters “
" stroke is kept within the handwriting input area, and the recognition result is the character "
" is shown only.

In some embodiments, the handwritten characters are multi-stroke Chinese characters. In some embodiments, the first plurality of handwritten inputs are provided in a cursive typeface. In some embodiments, the first plurality of handwritten inputs are provided in a cursive typeface, and the handwritten characters are multi-stroke Chinese characters. In some embodiments, the handwritten characters are written in cursive Arabic. In some embodiments, the handwritten characters are written in other scripts, such as cursive.

In some embodiments, the user device establishes respective default constraints for a set of acceptable dimensions for handwriting input, and applies multiple currently accumulated handwritten strokes to the respective default constraints. Based on the handwriting recognition model, the input image is divided into a plurality of recognition units, and each input image is generated from each of the recognition units and provided to the handwriting recognition model, where it is recognized as a corresponding output character.

In some embodiments, the user device receives additional handwritten strokes from the user after dividing the currently accumulated plurality of handwritten strokes. The user device assigns the further handwritten stroke to a respective one of the plurality of recognition units based on the spatial location of the further handwritten stroke relative to the plurality of recognition units.

Attention is now directed to an example user interface for providing handwriting recognition and input on a user device. In some embodiments, the exemplary user interface is provided on a user device based on a multi-script handwriting recognition model that provides real-time, stroke order-independent handwriting recognition of a user's handwriting input. . In some embodiments, the example user interface includes an example handwriting input area 804, a candidate display area 804, and a text input area 808 (e.g., as shown in FIGS. 8A and 8B). This is a user interface of input interface 802. In some embodiments, the example handwriting input interface 802 also includes a plurality of control elements 1102, such as a delete button, a spacebar, an enter button, a keyboard toggle button, etc. One or more other areas and/or elements may be provided within the handwriting input interface 902 to enable further functionality as described below.

As described herein, multi-script handwriting recognition models can have an extremely wide repertoire of tens of thousands of characters in many different scripts and languages. As a result, for handwritten input, this recognition model is very likely to identify a large number of output characters, all of which have a reasonably good likelihood of being the character intended by the user. has. On user devices with limited display area, it is advantageous to initially provide only a subset of recognition results, while keeping other results available upon user request.

11A-11G are for displaying a subset of recognition results within a normal view of a candidate display area, along with an affordance for invoking an expanded view of the candidate display area for displaying the remainder of the recognition results. 1 shows an example user interface of . Furthermore, within the expanded view of the candidate display area, the recognition results are sorted into different categories and displayed on different tabbed pages within the expanded view.

FIG. 11A shows an example handwriting input interface 802. The handwriting input interface includes a handwriting input area 804, a candidate display area 806, and a text input area 808. One or more control elements 1102 are also included within handwriting input interface 1002.

As shown in FIG. 11A, candidate display area 806 optionally includes an area for displaying one or more recognition results and an affordance 1104 for invoking an expanded version of candidate display area 806 (e.g., expanded icon).

FIGS. 11A-11C illustrate that when a user provides one or more handwritten strokes (e.g., strokes 1106, 1108, and 1110) within handwriting input area 804, the user device Indicates that each set of recognition results corresponding to the stroke being specified is identified and displayed. As shown in FIG. 11B, after the user inputs the first stroke 1106, the user device generates three recognition results 1112, 1114, and 1116 (e.g., characters "/", "1", and ", ”) and display them. In some embodiments, a small number of candidate characters are displayed within candidate display area 806 in an order according to recognition confidence associated with each character.

In some embodiments, the top-ranked candidate result (eg, "/") is tentatively displayed within text entry area 808, such as in box 1118. The user confirms that the top-ranked candidate is the intended input with a simple confirmation input (e.g., by pressing the "Enter" key or providing a double-tap gesture within the handwriting input area). , can optionally be verified.

FIG. 11C shows that if the user enters two more strokes 1108 and 1110 into the handwriting input area 804 before the user selects any candidate recognition result, those additional strokes along with the initial stroke 1106 It is rendered within the handwriting input area 804 to show that the candidate results are updated to reflect those changes in the recognition units identified from the currently stored handwriting input. As shown in FIG. 11C, based on the three strokes, the user device has identified a single recognition unit. Based on this single recognition unit, the user device identifies and displays several recognition results 1118-1124. In some embodiments, one or more of these recognition results (e.g., 1118 and 1122) currently displayed within candidate display area 806 each represent a plurality of similar recognition results for the current handwriting input. These are representative candidate characters selected from candidate characters with different appearances.

As shown in FIGS. 11C and 11D, when the user selects affordance 1104 (e.g., using a tap gesture with contact 1126 on affordance 1104), the candidate display area (e.g., 11C) to an expanded view (eg, shown in FIG. 11D). In some embodiments, this expanded view shows all of the recognition results (eg, candidate characters) that have been identified for the current handwriting input.

In some embodiments, the initially displayed normal view of candidate display area 806 shows only the most commonly used characters used in the respective script or language, while the expanded The view shows all candidate characters, including characters that are used infrequently in the script or language. This expanded view of the candidate display area can be designed in various ways. 11D-11G illustrate example designs of expanded candidate display areas, according to some embodiments.

As shown in FIG. 11D, in some embodiments, expanded candidate display area 1128 includes one or more tabbed pages (e.g., pages 1130, 1132, 1134, and 1136). The tabbed design shown in FIG. 11D allows the user to quickly locate the desired character category and then, within its corresponding tabbed page, find the character that the user intended to enter. Become.

In FIG. 11D, the first tabbed page 1130 displays all candidate characters that have been identified for the currently stored handwriting input, including both commonly used characters as well as less frequent characters. indicate. As shown in FIG. 11D, the tabbed page 1130 includes all of the characters shown in the initial candidate display area 806 in FIG. 11C, and some that were not included in the initial candidate display area 806. Further characters (for example, “
”, “β”, “
”, etc.).

In some embodiments, the characters displayed within the initial candidate display area 806 are characters from a set of commonly used characters associated with a particular script (e.g., encoded according to the Unicode standard). Contains only all characters in the basic blocks of CJK script, such as In some embodiments, the characters displayed within the expanded candidate display area 1128 are a set of infrequent characters associated with the script (e.g., an extension of the CJK script, such as encoded according to the Unicode standard). Also includes all characters in the block. In some embodiments, expanded candidate display area 1128 further includes candidate characters from other scripts not commonly used by users, such as Greek script, Arabic script, and/or emoji script.

In some embodiments, as shown in FIG. 11D, the expanded candidate display areas 1128 each include a respective category of candidate characters (e.g., all characters, infrequent characters, characters from the Latin script, respectively). , and characters from the emoji script), respectively. 11E-11G illustrate that a user can select each of the various tabbed pages to reveal candidate characters within the corresponding category. FIG. 11E shows only infrequent characters (eg, characters from the extended block of the CJK script) that correspond to the current handwritten input. FIG. 11F shows only the Latin and Greek characters that correspond to the current handwritten input. FIG. 11G shows only the pictograms that correspond to the current handwritten input.

In some embodiments, the expanded candidate display area 1128 displays candidate characters in each tabbed page based on respective criteria (e.g., based on phonetic spelling, based on number of strokes, and and one or more affordances for sorting (based on radicals, etc.). The ability to sort candidate characters within each category according to criteria other than recognition confidence score provides users with additional ability to quickly find desired candidate characters for text input.

FIGS. 11H-11K illustrate that in some embodiments, similar-looking candidate characters can be grouped together, and only representative characters from each group of similar-looking candidate characters are initialized. Indicates that the candidate is presented within the candidate display area 806. Because the multi-script recognition model described herein can produce many candidate characters that are approximately equally valid for a given handwriting input, the recognition model replaces one candidate with another similar-looking character. It is not always possible to sacrifice candidates for deletion. On devices with limited display area, displaying many similar-looking candidates all at once does not help the user choose the correct character, but it does make it easier to make small distinctions. and even if the desired character is visible to the user, selecting it from a very crowded display using a finger or stylus is difficult. This is because it may be difficult.

In some embodiments, to address the above issues, the user device selects candidate characters that have a high degree of similarity to each other (e.g., according to a concordance or dictionary of similar-looking characters, or some image-based criteria). identify and group those candidate characters into their respective groups. In some embodiments, one or more groups of similar-looking characters may be identified from a set of candidate characters for a given handwriting input. In some embodiments, the user device has identified a representative candidate character among a plurality of similar-looking candidate characters within the same group, and displays only the representative candidate in the initial candidate display area 806. Display within. If a commonly used character does not appear sufficiently similar to any other candidate character, then the character itself is displayed. In some embodiments, a representative candidate character of each group (e.g., candidate character 1118 "
” and 1122 “T”) are candidate characters that do not belong to any group (e.g., candidate character 1120 “
” and 1124 “J”) (e.g., in a thick-framed box). In some embodiments, the criteria for selecting a representative character of a group is based on the relative frequency of usage of the candidate character within the group. In some embodiments, other criteria may be used.

In some embodiments, once the representative character is displayed to the user, the user can optionally enlarge the candidate display area 806 to show similar-looking candidate characters in the enlarged view. Can be done. In some embodiments, selection of a particular representative character can result in an expanded view of only candidate characters within the same group as the selected representative character.

Various designs are possible for providing a magnified view of similar-looking candidates. 11H-11K illustrate one embodiment in which a predetermined gesture (e.g., a zoom gesture) detected on a representative candidate character (e.g., representative character 1118) invokes a magnified view of the representative candidate character. shows. The default gesture for invoking a magnified view (eg, a zoom gesture) is different from the default gesture for selecting a representative character for text input (eg, a tap gesture).

As shown in FIGS. 11H and 11I, the user makes a magnification gesture (e.g., as indicated by two contacts 1138 and 1140 moving away from each other) over the first representative character 1118. When provided, the area displaying representative character 1118 is enlarged to include three similar-looking candidate characters (e.g., “
”, “
",as well as"
'') but other candidate characters that are not in the same expansion group (for example, ``
”) are presented in an enlarged view (e.g., in enlarged boxes 1142, 1144, and 1146, respectively).

As shown in FIG. 11I, three similar-looking candidate characters (e.g., “
”, “
",as well as"
”) can be more easily seen by the user. If one of these three candidate characters is the intended character input, the user can select that candidate character, for example by touching the area where that character is displayed. can. As shown in FIGS. 11J and 11K, the user selects the second character (e.g., "
”) is selected (at contact 1148). Depending on this, the selected character (e.g. "
”) is entered into the text entry area 808 at the insertion point indicated by the cursor. As shown in FIG. 11K, once a character is selected, the handwriting input in the handwriting input area 804 and the candidate characters in the candidate display area 806 (or an enlarged view of the candidate display area) are used for subsequent handwriting input. will be deleted for.

In some embodiments, if the user does not see the desired candidate character within the expanded view of the first representative candidate character 1142, the user optionally uses the same gesture to Other characters displayed within candidate display area 806 can be enlarged. In some embodiments, by enlarging another representative character within candidate display area 806, the currently presented enlarged view is automatically restored to the normal view. In some embodiments, the user optionally uses a zoom out gesture to restore the currently zoomed view to the normal view. In some embodiments, the user can scroll the candidate display area 806 (eg, to the left or to the right) to reveal other candidate characters that are not visible within the candidate display area 806.

12A and 12B show that a first subset of recognition results is presented within the initial candidate display area, while a second subset of recognition results is hidden until specifically invoked by the user. 12 is a flowchart of an example process 1200 presented in a hidden expanded candidate display area. In the example process 1200, the device identifies, from multiple handwriting recognition results for handwriting input, a subset of recognition results that have a level of visual similarity that exceeds a predetermined threshold. The user device then selects a representative recognition result from the subset of recognition results and displays the selected representative recognition result in a candidate display area of the display. Process 1200 is shown in FIGS. 11A-11K.

As shown in FIG. 12A, in example process 1200, a user device receives handwritten input from a user (1202). This handwriting input includes one or more handwritten strokes (e.g., 1106 in FIG. 11C) provided within a handwriting input area (e.g., 806 in FIG. 11C) of a handwriting input interface (e.g., 802 in FIG. 11C). , 1108, 1110). The user device identifies a plurality of output characters (eg, the characters shown in tabbed page 1130 in FIG. 11C) for the handwriting input based on the handwriting recognition model (1204). The user device sorts the plurality of output characters into two or more categories based on predefined categorization criteria (1206). In some embodiments, the predefined categorization criteria determines whether each character is a commonly used character or a low frequency character (1208).

In some embodiments, the user device displays a first category of the two or more categories (e.g., 806 as shown in FIG. , commonly used characters) (1210), and this initial view of the candidate display area is used to invoke an expanded view of the candidate display area (e.g., 1128 in FIG. 11D). (eg, 1104 in FIG. 11C).

In some embodiments, the user device receives user input selecting an affordance for invoking an expanded view (1212), for example, as shown in FIG. 11C. In response to this user input, the user device displays each output character in the first category in an expanded view of the candidate display area and in the initial view of the candidate display area, as shown in FIG. 11D, for example. displays each output character in at least a second category of the two or more categories that was not previously displayed (1214).

In some embodiments, each character in the first category is a character found in a dictionary of commonly used characters, and each character in the second category is a low-frequency character. is a character found in the dictionary. In some embodiments, the dictionary of commonly used characters and the dictionary of infrequent characters are dynamically adjusted or updated based on usage history associated with the user device.

In some embodiments, the user device determines, from the plurality of output characters, according to a predetermined similarity criterion (e.g., based on a dictionary of similar characters or based on some spatially derived feature). Groups of characters that are visually similar to each other are identified (1216). In some embodiments, the user device selects a representative character from a group of visually similar characters based on predetermined selection criteria (eg, based on historical frequency of usage). In some embodiments, this predetermined selection criterion is based on the relative frequency of usage of the characters within the group. In some embodiments, this predetermined selection criteria is based on the preferred input language associated with the device. In some embodiments, the representative candidates are based on other factors that indicate the likelihood that each candidate is the intended input by the user. These factors include, for example, whether the candidate character belongs to a soft keyboard script currently installed on the user's device, or whether the candidate character is associated with a specific user or user device. for example, whether it is in the most commonly used character set for that language.

In some embodiments, the user device displays a representative character (e.g., "
”) to other characters in its visually similar group of characters (e.g., “
”, “
) (1220). In some embodiments, each candidate character is displayed within the initial view of the candidate display area to indicate whether it is a representative character of a group or a regular candidate character that does not exist within any group. , visual instructions (e.g., selective visual highlighting, special backgrounds) are provided. In some embodiments, the user device performs a predefined magnification input (e.g., a magnification gesture) that targets the representative character displayed within the initial view of the candidate display area, e.g., as shown in FIG. 11H. is received from the user (1222). In some embodiments, in response to receiving this default magnification input, the user device displays a magnified view of the representative character and a group of visually similar characters, as shown in FIG. 11I, for example. simultaneously displaying (1224) each magnified view of one or more other characters within.

In some embodiments, this default magnification input is a magnification gesture detected over the representative character displayed within the candidate display area. In some embodiments, the predetermined magnification input is a touch detected on the representative character displayed within the candidate display area and sustained for more than a predetermined threshold time. In some embodiments, this sustained touch to expand the group has a longer threshold duration than the tap gesture to select representative characters for text input.

In some embodiments, each representative character is displayed simultaneously with a corresponding affordance (eg, a corresponding zoom button) to invoke a magnified view of the group of similar-looking candidate characters. In some embodiments, the default magnification input is a selection of corresponding affordances associated with the representative character.

As described herein, in some embodiments, the repertoire of multi-script handwriting recognition models includes emoji scripts. The handwriting input recognition module can recognize emoticons based on a user's handwritten input. In some embodiments, the handwriting recognition module presents both the glyphs identified directly from the handwriting and the characters or words in natural human language that represent the identified glyphs. In some embodiments, the handwriting input module recognizes characters or words in a natural human language based on the user's handwritten input, and corresponds to the recognized characters or words and the recognized characters or words. Present both sides with emoji. In other words, the handwriting input module provides a way to input emojis without switching from a handwriting input interface to an emoji keyboard. Additionally, the handwriting input module also provides a method for inputting regular natural language characters by hand-drawing glyphs. 13A-13E provide example user interfaces that illustrate these various methods of entering emoji and regular natural language characters.

FIG. 13A shows an example handwriting input interface 802 invoked under a chat application. Handwriting input interface 802 includes a handwriting input area 804 , a candidate display area 806 , and a text input area 808 . In some embodiments, once the user is satisfied with the text composition in text entry area 808, the user can choose to send the text composition to another participant in the current chat session. The interaction history of the chat session is shown within the interaction panel 1302. In this illustrative example, the user may receive a chat message 1304 (e.g., "Happy Birthday") displayed within the interaction panel 1302.
”) is being received.

As shown in FIG. 13B, the user has provided handwritten input 1306 within handwritten input area 804 for the English word "Thanks." In response to handwriting input 1306, the user device has identified several candidate recognition results (eg, recognition results 1308, 1310, and 1312). The highest ranked recognition result 1303 has been provisionally entered in box 1314 within text input area 808 .

As shown in FIG. 13C, after the user enters the handwritten word "Thanks" within the handwriting input area 806, the user then enters a stroke 1316 (e.g., a rounded circle down) within the handwriting input area 806. (with an elongated circle) depicting a stylized exclamation point. The user device recognizes that this additional stroke 1316 forms a separate recognition unit from other recognition units previously recognized from the accumulated handwritten strokes 1306 within the handwriting input area 806. Based on this newly input recognition unit (i.e., the recognition unit formed by stroke 1316), the user device uses the handwriting recognition model to identify the emoticon (e.g., a stylized "!") do. Based on the recognized emoji, the user device presents a first recognition result 1318 (eg, “Thanks!” with a stylized “!”) in the candidate display area 806. Furthermore, the user device also identifies the number "8", which is also visually similar to the newly entered recognition unit. Based on this recognized digit, the user device presents a second recognition result 1322 (eg, “Thanks 8”) within candidate display area 806. Further, based on the identified emoji (eg, a stylized "!"), the user device also identifies a regular character (eg, a regular character "!") that corresponds to the emoji. Based on this indirectly recognized regular character, the user device presents a third recognition result 1320 (eg, "Thanks!" with a regular "!") in the candidate display area 806. At this point, the user can select any one of candidate recognition results 1318, 1320, and 1322 and enter that candidate recognition result into text entry area 808.

As shown in FIG. 13D, the user continues to provide additional handwriting strokes 1324 within handwriting input area 806. This time, the user has drawn a stylized exclamation mark followed by a heart symbol. In response to this new handwritten stroke 1324, the user device recognizes that the newly provided handwritten stroke 1324 forms yet another new recognition unit. Based on this new recognition unit, the user device selects the emoji "
”, and alternatively the number “0”. Based on these new candidate characters recognized from the new recognition unit, the user device generates two updated candidate recognition results 1326 and 1330 (e.g., “Thanks
” and “Thanks 80”). In some embodiments, the user device identifies an emoji (e.g., "
”) that corresponds to the canonical characters or words (eg, “Love”). Based on the identified canonical character or word for the recognized emoji, the user device presents a third recognition result 1328 in which the recognized emoji is replaced with its corresponding canonical character or word. do. As shown in FIG. 13D, in the recognition result 1328, the pictogram "
” is replaced with the regular exclamation mark “!” and the emoji “!”
” has been replaced with the regular letter or word “Love.”

As shown in Figure 13E, the user selects one of the candidate recognition results (e.g., the text "Thanks" of the mixed script).
” and the text of this selected recognition result is entered into the text entry area 808 and then sent to the other participants in the chat session. A message bubble 1332 shows the text of the message within the dialog panel 1302.

FIG. 14 is a flow diagram of an example process 1400 in which a user enters an emoji using handwritten input. 13A-13E illustrate an example process 1400, according to some embodiments.

In process 1400, a user device receives (1402) handwritten input from a user. The handwriting input includes a plurality of handwritten strokes provided within a handwriting input area of the handwriting input interface. In some embodiments, the user device recognizes (1404) a plurality of output characters from its handwriting input based on a handwriting recognition model. In some embodiments, those output characters include at least the first glyph (e.g., the stylized exclamation mark "
” or emoji “
”) and at least a first character from a natural human language script (eg, the character from the word “Thanks” in FIG. 13D). In some embodiments, the user device displays their first glyph (e.g., the stylized exclamation mark "
” or emoji “
”) and a first character from the natural human language script (e.g., the character from the word “Thanks” in FIG. 13D), the recognition result (e.g., result 1326 in FIG. 13D) is input to the handwriting input interface. (1406).

In some embodiments, based on the handwriting recognition model, the user device optionally recognizes (1408) at least a first semantic unit (e.g., the word "Thanks") from the handwriting input; This first semantic unit includes each letter, word, or phrase capable of conveying a respective semantic meaning in a respective human language. In some embodiments, the user device identifies a second emoji (e.g., a "handshake" emoji) that is associated with a first semantic unit (e.g., the word "Thanks") recognized from the handwritten input. (1410). In some embodiments, the user device displays a second glyph that includes at least a second glyph identified from the first semantic unit (e.g., the word "Thanks") within the candidate display area of the handwriting input interface. Recognition results (for example, after the "handshake" emoji, "
"as well as"
” is displayed (1412). In some embodiments, displaying the second recognition result includes displaying a third recognition result (e.g., the recognition result "Thanks") that includes at least the first semantic unit (e.g., the word "Thanks").
'') and simultaneously displaying the second recognition result.

In some embodiments, a user receives user input selecting a first recognition result displayed within the candidate display area. In some embodiments, in response to the user input, the user device enters text of the selected first recognition result within the text input area of the handwriting input interface, the text being at least the first recognition result. 1 glyph and a first character from a natural human language script. In other words, the user can write text in a mixed script using a single handwriting input (albeit a handwriting input containing multiple strokes) within the handwriting input area without having to switch between the natural language keyboard and the emoji keyboard. It is possible to enter input.

In some embodiments, the handwriting recognition model is trained on a multi-script training corpus that includes writing samples corresponding to characters in at least three non-overlapping scripts, where the three non-overlapping scripts include emoji, Chinese Contains a set of characters, and Latin script.

In some embodiments, the user device detects a first emoji (e.g., "
A second semantic unit (eg, the word "Love") is identified (1414) that corresponds to the word "Love" (pictogram). In some embodiments, the user device displays a first emoji (e.g., "
A fourth recognition result (e.g., 1328 in FIG. 13D) is displayed (1416), including at least the second semantic unit (e.g., the word "Love") identified from the emoji). In some embodiments, the user device displays a first recognition result (e.g., result "Thanks" in the candidate display area, as shown in FIG. 13D).
"), and at the same time, a fourth recognition result (for example, result 1328 "Thanks! Love") is displayed.

In some embodiments, the user device allows the user to enter regular text by drawing emoticons. For example, if the user does not know how to spell the word "elephant," the user optionally draws a stylized glyph for "elephant" in the handwriting input area and the user device If the handwritten input can be correctly recognized as an emoji for "elephant," the user device also optionally displays normal text as one of the recognition results displayed within the candidate display area. Also presents the word "elephant". In another example, the user may enter the Chinese characters “
Instead of writing ``, you can draw a stylized cat. If the user device identifies an emoji for "cat" based on the handwritten input provided by this user, the user device also optionally identifies in the candidate display area along with the emoji for "cat". , the Chinese character ``, which means ``cat'' in Chinese
” will also be presented. By presenting normal text for recognized emojis, the user device inputs complex characters or words using a small number of stylized strokes commonly associated with well-known emojis. Provide an alternative method. In some embodiments, the user device displays emoji as their corresponding regular text (e.g., a character, word, phrase, symbol, etc.) in one or more preferred scripts or languages (e.g., English or Chinese). etc.) and memorize the dictionary.

In some embodiments, the user device recognizes the emoji based on the emoji's visual similarity to an image generated from handwritten input. In some embodiments, to enable recognition of emojis from handwritten input, the handwriting recognition model used on the user device includes handwriting samples that correspond to characters in a natural human language script, and artificially It is trained using a training corpus that includes both handwriting samples corresponding to the designed set of emoji. In some embodiments, emojis related to the same semantic concept may have different appearances when used within mixed input having text in different natural languages. For example, an emoji for the semantic concept of "Love" could be a "heart" emoji when presented in regular text in one natural language (e.g. Japanese), and an emoji for a "heart" in another natural language (e.g. Japanese). , English or French), it could be a "kiss" emoji.

As described herein, when performing recognition of multi-character handwritten input, the handwriting input module performs segmentation of the currently accumulated handwritten input within the handwriting input area, and divides those accumulated handwritten inputs. separate the strokes into one or more recognition units. One of the parameters used to determine how to partition handwriting input is how those strokes are clustered within the handwriting input area, and the differences between clusters of different strokes. The distance can be People have different typefaces. Some people tend to write with very low density, with large distances between strokes, or distances between different parts of the same letter, while others tend to write with very low density, with large distances between strokes, or different parts of the same letter. There is a tendency to write with extremely high density, with extremely small distances between characters. Even for the same user, imperfect planning can cause handwriting to deviate from a balanced appearance and become lopsided, stretched, or compressed in various ways. As described herein, multi-script handwriting recognition models provide stroke order independent recognition, thus allowing a user to write characters or parts of characters out of order. . As a result, achieving spatial uniformity and balance of handwritten input between characters can be difficult.

In some embodiments, the handwriting input model described herein allows the user to instruct the handwriting input module to either merge two adjacent recognition units into a single recognition unit or A method is provided for notifying whether a recognition unit should be separated into two separate recognition units. With this user assistance, the handwriting input module can correct the initial segmentation and produce the results intended by the user.

15A-15J illustrate some example user interfaces and processes in which a user provides predefined pinch and zoom gestures to modify recognition units identified by a user device.

As shown in FIGS. 15A and 15B, the user has entered a plurality of handwritten strokes 1502 (eg, three strokes) within the handwriting input area 806 of the handwriting input interface 802. The user device has identified a single recognition unit based on the currently accumulated handwritten strokes 1502 and has three candidate characters 1508 (e.g., "
”, “
",as well as"
”) is presented.

FIG. 15C shows the user inputting a small number of additional strokes 1510 within the handwriting input area 606 to the right of the initial handwritten strokes 1502. The user device determines (eg, based on the dimensions and spatial distribution of strokes 1502 and 1510) that stroke 1502 and stroke 1510 should be considered as two separate recognition units. Based on this separation of recognition units, the user device provides the input images of the first recognition unit and the second recognition unit to the handwriting recognition model and obtains a set of two candidate characters. The user device then generates multiple recognition results (eg, 1512, 1514, 1516, and 1518) based on various combinations of those recognized characters. Each recognition result includes a recognized character regarding the first recognition unit and a recognized character regarding the second recognition unit. As shown in FIG. 15C, the plurality of recognition results 1512, 1514, 1516, and 1518 each include two recognized characters.

In this example, the user actually intended this handwritten input to be recognized as a single character, but the handwritten character (e.g. "
") (for example, the left radical "
”) and the right part (for example, the right radical “
”), we assume that you have inadvertently left an excessively large space between the two. Upon viewing the results presented in candidate display area 806 (e.g., 1512, 1514, 1516, and 1518), the user determines that the user device has incorrectly split the current handwritten input into two recognition units. Understand that Although this division can be based on objective criteria, it would be useful for the user to delete the current handwritten input and rewrite the entire character with a smaller distance between the left and right parts. , which is not desirable.

Instead, as shown in FIG. 15D, the user uses a pinch gesture over the two clusters of handwritten strokes 1502 and 1510 so that the two recognition units identified by their handwriting input module are Indicates to the handwriting input module that it should be fused as a single recognition unit. This pinch gesture is indicated by two contacts 1520 and 1522 on the touch-sensitive surface moving towards each other.

FIG. 15E shows that in response to a user's pinch gesture, the user device corrects the segmentation of currently accumulated handwritten input (e.g., strokes 1502 and 1510) and combines the handwritten strokes into a single recognition unit. It shows that it is fused with. As shown in FIG. 15E, the user device provides an input image based on the corrected recognition unit to the handwriting recognition model and generates three new candidate characters 1524, 1526, and 1528 (for example, “
”, “
",as well as"
”) is obtained. In some embodiments, as shown in FIG. 15E, the user device optionally renders the handwriting input within the handwriting input area 806 such that the distance between the left and right clusters of handwritten strokes is reduced. Adjust so that In some embodiments, the user device does not change the rendering of the handwriting input shown within the handwriting input area 608 in response to the pinch gesture. In some embodiments, the user device identifies pinch gestures from input strokes based on two simultaneous contacts (as opposed to a single contact) detected within handwriting input area 806.

As shown in FIG. 15F, the user inputs the previously input handwritten input (i.e., the character "
Two more strokes 1530 are input to the right of the stroke related to "." The user device determines that this newly inputted stroke 1530 is a new recognition unit, and selects a candidate character (for example, "
”). The user device then displays that newly identified character (e.g., "
”) with the candidate characters for the previously identified recognition unit to present several different recognition results (eg, results 1532 and 1534) within the candidate display area 806.

Following the handwritten stroke 1530, the user continues to write additional strokes 1536 (eg, three additional strokes) to the right of the stroke 1530, as shown in FIG. 15G. Since the horizontal distance between stroke 1530 and stroke 1536 is extremely small, the user device determines that stroke 1530 and stroke 1536 belong to the same recognition unit, and the handwriting recognition model includes the following: Provide input image. The handwriting recognition model identifies three different candidate characters for the corrected recognition unit and produces two corrected recognition results 1538 and 1540 for the currently stored handwriting input.

In this example, the last two sets of strokes 1530 and 1536 are actually two separate characters (e.g., "
” and “±”). After the user observes that the user device is erroneously combining the sets of two strokes 1530 and 1536 into a single recognition unit, the user subsequently provides a magnification gesture to cause the user device to , signals that the set of two strokes 1530 and 1536 should be separated into two separate recognition units. As shown in FIG. 15H, the user makes two contacts 1542 and 1544 around strokes 1530 and 1536, and then makes those two contacts in a generally horizontal direction (i.e., the default writing move away from each other (along the direction).

FIG. 15I shows that in response to this user's magnification gesture, the user device corrects the previous segmentation of the currently stored handwriting input and assigns stroke 1530 and stroke 1536 into two consecutive recognition units. to show that Based on the input images generated for those two separate recognition units, the user device identifies one or more candidate characters for the first recognition unit based on stroke 1530 and the first recognition unit based on stroke 1536. One or more candidate characters are identified for the recognition unit No. 2. The user device then generates two new recognition results 1546 and 1548 based on the various combinations of those recognized characters. In some embodiments, the user device optionally modifies the rendering of strokes 1536 and 1536 to reflect the previously identified recognition unit separation.

As shown in FIGS. 15J and 15K, the user has selected one of the candidate recognition results displayed within candidate display area 806 (as indicated by contact 1550), and the user has selected one of the candidate recognition results displayed within candidate display area 806 and The resulting recognition results (eg, results 1548) have been entered into the text entry area 808 of the user interface. After the selected recognition result is entered into text input area 808, candidate display area 806 and handwriting input area 804 are both cleared and ready for displaying subsequent user input.

16A and 16B inform the user how to split or correct an existing split of the current handwriting input using predefined gestures (e.g., pinch gesture and/or zoom gesture) , which is a flow diagram of an example process 1600. 15J and 15K provide an illustration of an example process 1600, according to some embodiments.

In some embodiments, a user device receives handwritten input from a user (1602). The handwriting input includes a plurality of handwritten strokes provided in a touch-sensitive surface coupled to the device. In some embodiments, the user device renders the plurality of handwritten strokes in real time (1604) within a handwriting input area of a handwriting input interface (eg, handwriting input area 806 of FIGS. 15A-15K). The user device receives one of a pinch gesture input and a zoom gesture input over the plurality of handwritten strokes, for example, as shown in FIGS. 15D and 15H.

In some embodiments, upon receiving a pinch gesture input, the user device recognizes multiple handwritten strokes by processing them as a single recognition unit, for example, as shown in FIGS. 15C-15E. A first recognition result based on the plurality of handwritten strokes is generated (1606).

In some embodiments, upon receiving the magnification gesture input, the user device converts the multiple handwritten strokes into two separate recognition units separated by the magnification gesture input, as shown in FIGS. 15G-15I, for example. A second recognition result is generated based on the plurality of handwritten strokes (1608).

In some embodiments, upon generating each one of the first recognition result and the second recognition result, the user device generates the generated recognition result, e.g., as shown in FIGS. 15E and 15I. The results are displayed in the candidate display area of the handwriting input interface.

In some embodiments, the pinch gesture input includes two simultaneous contacts on the touch-sensitive surface that converge toward each other within an area occupied by multiple handwritten strokes. In some embodiments, the magnified gesture input includes two simultaneous contacts on the touch-sensitive surface that are dispersed from each other within the area occupied by multiple handwritten strokes.

In some embodiments, the user device identifies two adjacent recognition units from the plurality of handwritten strokes (eg, 1614). The user device may generate an initial recognition result (e.g., result 1512 in FIG. 15C, 1514, 1516, and 1518) are displayed (1616). In some embodiments, when displaying a first recognition result (e.g., results 1524, 1526, or 1528 in FIG. 15E) in response to a pinch gesture, the user device displays an initial recognition result within the candidate display area. The recognition result is replaced with the first recognition result (1618). In some embodiments, the user device receives a pinch gesture input (1620) while the initial recognition results are displayed within the candidate display area, as shown in FIG. 15D. In some embodiments, in response to this pinch gesture input, the user device may reduce the distance between two adjacent recognition units within the handwriting input area, for example, as shown in FIG. 15E. Re-render the multiple handwritten strokes (1622).

In some embodiments, the user device identifies a single recognition unit from these multiple handwritten strokes (1624). The user device displays characters recognized from that single recognition unit (e.g., "
""
”), the initial recognition results (eg, results 1538 or 1540 in FIG. 15G) are displayed (1626). In some embodiments, when displaying a second recognition result (e.g., result 1546 or 1548 in FIG. 15I) in response to a magnification gesture, the user device Then, within the candidate display area, the initial recognition result (eg, result 1538 or 1540) is replaced with its second recognition result (eg, result 1546 or 1548) (1628). In some embodiments, the user device receives magnified gesture input (1630) while the initial recognition results are displayed within the candidate display area, as shown in FIG. 15H. In some embodiments, in response to this magnified gesture input, the user device generates a first stroke within the handwriting input area that is assigned to the first recognition unit, as shown in FIGS. 15H and 15I. and the second subset of handwritten strokes assigned to the second recognition unit (1632).

In some embodiments, immediately after the user provides the strokes and realizes that the strokes may be too spread apart for the correct splitting based on the standard splitting process: The user optionally provides a pinch gesture to signal the user device to process the multiple strokes as a single recognition unit. A user device can distinguish a pinch gesture from a regular stroke based on the presence of two simultaneous contacts in the pinch gesture. Similarly, in some embodiments, a user provides strokes and understands that the strokes may be too dense for correct segmentation based on a standard segmentation process. Immediately thereafter, the user optionally provides a magnification gesture to notify the user device to process the multiple strokes as two separate recognition units. A user device can distinguish a magnification gesture from a regular stroke based on the two simultaneous contacts present in a pinch gesture.

In some embodiments, the direction of movement of the pinch or expand gesture is optionally used to provide further guidance on how to split the strokes under the gesture. For example, for a handwriting input area, if multi-line handwriting input is enabled, a pinch gesture with two vertically moving contacts will cause the handwriting input module to have two contacts identified within two adjacent lines. Recognition units may be signaled to be fused into a single recognition unit (eg, as an upper radical and a lower radical). Similarly, a magnification gesture with two contacts moving vertically can tell the handwriting input module to separate a single recognition unit into two recognition units in two adjacent lines. . In some embodiments, the pinch and zoom gestures also provide guidance for splitting character input into sub-parts, such as merging two sub-components, or compound characters (
Guidance can also be provided for separating a single component within a compound character (e.g., the top, bottom, left, or right part) into different parts of the compound character. This is particularly useful for recognizing complex compound Chinese characters since users tend to lose correct proportion and balance when writing complex compound characters by hand. Being able to adjust the proportion and balance of the handwriting input after completion of the handwriting input, for example by pinch and zoom gestures, allows the user to perform several attempts to arrive at the correct proportion and balance. Particularly useful for entering the correct characters without having to do so.

As described herein, the handwriting input module allows a user to input handwriting input of multiple characters, within a handwriting input area, within one character, across multiple characters, or even allows out-of-order strokes for those multi-character handwritten inputs over multiple words, sentences, and/or lines. In some embodiments, the handwriting input module also provides character-by-character deletion within the handwriting input area, where the order of character deletion is in the opposite direction of writing, and the strokes for each character It is independent of whether it is provided within the handwriting input area. In some embodiments, the deletion of each recognition unit (e.g., character or radical) within the handwriting input area is optionally performed on a stroke-by-stroke basis, where the strokes are deleted within the recognition unit. Delete in reverse chronological order provided. 17A-17H illustrate example user interfaces for providing character-by-character deletion within multi-character handwritten input in response to deletion input from a user.

As shown in FIG. 17A, a user has provided a plurality of handwriting strokes 1702 within a handwriting input area 804 of a handwriting input user interface 802. Based on these currently accumulated strokes 1702, the user device presents three recognition results (eg, results 1704, 1706, and 1708) within candidate display area 806. As shown in FIG. 17B, the user has provided additional strokes 1710 within handwriting input area 806. The user device recognizes the three new output characters and replaces the three previous recognition results 1704, 1706, and 1708 with the three new recognition results 1712, 1714, and 1716. In some embodiments, even if the user device identifies two separate recognition units from the current handwriting input (e.g., stroke 1702 and stroke 1710), as shown in FIG. 17B, , the cluster of strokes 1710 does not correspond well to any known characters in the handwriting recognition module's repertoire. As a result, the candidate character identified for the recognition unit containing stroke 1710 (e.g., "
”, “
”) all have recognition confidences below a predefined threshold. In some embodiments, the user device determines the candidate character (e.g., "
A partial recognition result (e.g., result 1712) is presented in candidate display area 806 that includes only a character (") but not any candidate characters for the second recognition unit. In some embodiments, the user device generates a complete recognition result (e.g., result 1714 or 1716) that includes candidate characters for both recognition units, regardless of whether the recognition confidence passes a predefined threshold. Show more. Providing partial recognition results informs the user which parts of their handwriting input require correction. Additionally, the user may also choose to first enter the correctly recognized portions of their handwriting input and then rewrite the incorrectly recognized portions.

FIG. 17C shows the user continuing to provide additional handwritten strokes 1718 to the left of stroke 1710. Based on the relative location and distance of stroke 1718, the user device determines that this newly added stroke belongs to the same recognition unit as the cluster of handwritten strokes 1702. Based on this corrected recognition unit, a new character (e.g. "
”) are recognized and a new set of recognition results 1720, 1722, and 1724 is generated. Again, the first recognition result 1720 is a partial recognition result because none of the candidate characters identified for the stroke 1710 meet the predetermined confidence threshold.

FIG. 17D shows that the user has now entered a number of new strokes 1726 between strokes 1702 and 1710. The user device assigns this newly input stroke 1726 to the same recognition unit as stroke 1710. At this point, the user has written two Chinese characters (e.g. "
”) and the correct recognition result 1728 is shown in the candidate display area 806.

FIG. 27E shows the user entering the initial portion of the delete input, for example, by performing a light touch 1730 on the delete button 1732. If the user maintains contact with the delete button 1732, the user can delete the current handwritten input characters character by character (or recognition unit by recognition unit). This deletion is not performed simultaneously for all handwritten inputs.

In some embodiments, when the user's finger first touches the delete button 1732 on the touch-sensitive screen, the default (e.g., left-to-right) writing direction is selected, as shown in FIG. 17E. The last recognition unit (for example, the character "
”) is visually highlighted (e.g., highlighted with a border 1734 or a bright background) relative to other recognition units that are simultaneously displayed within the handwriting input area 804.

In some embodiments, when the user device detects that the user maintains contact 1730 on the delete button 1732 for more than a threshold duration, the user device , removes the highlighted recognition unit (eg, within box 1734) from handwriting input area 806. Additionally, the user device also deletes the recognition results shown in candidate display area 608 to delete any output characters that were generated based on the deleted recognition unit, as shown in FIG. 17F. correct.

FIG. 17F shows the last recognition unit (e.g., the character "
If the user continues to maintain contact 1730 on the delete button 1732 after a recognition unit related to "" is deleted, the recognition unit adjacent to the deleted recognition unit (for example, the character "
”) will be the next recognition unit to be deleted. As shown in FIG. 17F, this remaining recognition unit is visually highlighted (eg, within box 1736) and is ready to be deleted. In some embodiments, the visual highlighting of the recognition unit provides a preview of the recognition unit that will be deleted if the user continues to maintain contact with the delete button. If the user discontinues contact with the delete button before the threshold duration is reached, the visual highlighting is removed from the last recognition unit and that recognition unit is not deleted. As will be recognized by those skilled in the art, the contact duration is reset each time a recognition unit is deleted. Additionally, in some embodiments, the intensity of the contact (e.g., the pressure the user is applying to the contact 1730 with the touch-sensitive screen) is optionally used to determine the currently highlighted recognition unit. Adjust the threshold duration to confirm user intent to delete. 17F and 17G show that the user aborts contact 1730 with the delete button 1732 before the threshold duration is reached, and the characters "
” is stored in the handwriting input area 806. When the user selects a first recognition result (e.g., result 1738) for this recognition unit (e.g., as indicated by contact 1740), the first recognition result, as shown in FIGS. 17G and 17H, The text in result 1738 is entered into text entry area 808.

18A and 18B are a flowchart of an example process 1800 in which a user device provides character-by-character deletion within multi-character handwritten input. In some embodiments, this deletion of the handwritten input is performed before characters recognized from the handwritten input are verified and entered into the text entry area of the user interface. In some embodiments, the deletion of characters within the handwritten input proceeds according to the reverse spatial order of the recognition units identified from the handwriting input, and is independent of the temporal order in which those recognition units are formed. It is. 17A-17H illustrate an example process 1800, according to some embodiments.

As shown in FIG. 18A, in an example process 1800, a user device receives handwriting input from a user (1802), where the handwriting input is located in a handwriting input area of a handwriting input interface (e.g., area 804 in FIG. 17D). ), including multiple handwritten strokes. The user device identifies (1804) a plurality of recognition units from the plurality of handwritten strokes, each recognition unit including a corresponding subset of the plurality of handwritten strokes. For example, as shown in FIG. 17D, a first recognition unit includes strokes 1702 and 1718, and a second recognition unit includes strokes 1710 and 1726. The user device generates (1806) a multi-character recognition result (eg, result 1728 in FIG. 17D) that includes each character recognized from the plurality of recognition units. In some embodiments, the user device displays the multi-character recognition results (eg, results 1728 in FIG. 17D) within a candidate display area of the handwriting input interface. In some embodiments, while the multi-character recognition results are displayed within the candidate display area, the user device receives a delete input (e.g., a delete button) from the user, e.g., as shown in FIG. 17E. A contact 1730) on 1732 is received (1810). In some embodiments, in response to receiving this delete input, the user device deletes the plurality of items displayed within the candidate display area (e.g., candidate display area 806), as shown in FIGS. 17E, 17F, for example. From the character recognition results (e.g., result 1728), the last character (e.g., spatial array '
The character that appears at the end of ``
" is removed (1812).

In some embodiments, the user device includes a plurality of handwriting strokes within the handwriting input area of the handwriting input interface when the plurality of handwriting strokes are provided by the user, for example, as shown in FIGS. 17A-17D. The handwritten strokes of are rendered in real time (1814). In some embodiments, in response to receiving the delete input, the user device deletes a line from a handwriting input area (e.g., handwriting input area 804 in FIG. 17E) formed by a plurality of recognition units within the handwriting input area. A corresponding subset of the plurality of handwritten strokes that corresponds to the last recognition unit (eg, the recognition unit that includes strokes 1726 and 1710) in the spatial array is removed (1816). This last recognition unit is the last character (e.g., the character "
”).

In some embodiments, this last recognition unit does not include the temporally last handwritten stroke among the plurality of handwritten strokes provided by the user (1818). For example, if the user provides stroke 1718 after providing strokes 1726 and 1710, the last recognition unit that includes strokes 1726 and 1710 is still deleted first.

In some embodiments, in response to receiving the initial portion of the deletion input, the user device deletes the last recognition unit from other recognition units identified within the handwriting input area, e.g., as shown in FIG. 17E. (1820). In some embodiments, the initial portion of the delete input is an initial contact detected on the delete button within the handwriting input interface (1822), and the initial contact is detected over a predetermined threshold amount of time. If persisted, a delete input is detected.

In some embodiments, the last recognition unit corresponds to handwritten Chinese characters. In some embodiments, the handwritten input is written in a cursive typeface. In some embodiments, the handwritten input corresponds to a plurality of Chinese characters written in a cursive typeface. In some embodiments, at least one of the handwritten strokes is separated into two adjacent recognition units of the plurality of recognition units. For example, in some cases, a user may use long strokes that are continued into multiple characters, and in such cases, the splitting module of the handwriting input module optionally Separate strokes into several recognition units. When deletion of handwritten input is performed character by character (or recognition unit by recognition unit), only segments of this long stroke (eg, divisions within the corresponding recognition unit) are deleted at a time.

In some embodiments, the deletion input is a sustained touch on a delete button provided within the handwriting input interface (1824), and removing the corresponding subset of the plurality of handwritten strokes The method further includes removing the subset of handwritten strokes within the recognition unit from the handwriting input area stroke by stroke in the reverse temporal order in which the subset of handwritten strokes was provided by the user.

In some embodiments, the user device generates (1826) a partial recognition result that includes a subset of each recognized character from the plurality of recognition units, such as shown in FIGS. 17B and 17C. , each of the respective subsets of characters satisfy a predetermined confidence threshold. In some embodiments, the user device displays partial recognition results (e.g., result 1712 in FIG. 17B and result 1720 in FIG. 17C) within the candidate display area of the handwriting input interface as a multi-character recognition result. (eg, results 1714 and 1722) are displayed simultaneously (1828).

In some embodiments, the partial recognition result does not include at least the last character in the multi-character recognition result. In some embodiments, the partial recognition result does not include at least the first character in the multi-character recognition result. In some embodiments, the partial recognition result does not include at least an intermediate character within the multi-character recognition result.

In some embodiments, the smallest unit of deletion is a radical, and the handwriting input is deleted whenever the radical happens to be the last recognized unit in the handwriting input that still remains within the handwriting input area. One radical is removed at a time.

As described herein, in some embodiments, a user device provides both horizontal and vertical write modes. In some embodiments, the user device allows the user to enter text in a horizontal writing mode and in one or both of a left-to-right writing direction and a right-to-left writing direction. . In some embodiments, the user device allows the user to enter text in a vertical writing mode in one or both of a top-to-bottom writing direction and a bottom-to-top writing direction. . In some embodiments, the user device provides various affordances (e.g., writing mode or writing direction buttons) on the user interface to invoke respective writing modes and/or writing directions for the current handwriting input. provide. In some embodiments, the text input direction within the text input area is by default the same as the handwriting input direction in the handwriting input direction. In some embodiments, the user device allows the user to manually set the input direction within the text input area and the writing direction within the handwriting input area. In some embodiments, the text display direction within the candidate display area is by default the same as the handwriting input direction within the handwriting input area. In some embodiments, the user device allows the user to manually set the text display direction within the text input area, independent of the handwriting input direction within the handwriting input area. In some embodiments, the user device associates a writing mode and/or writing direction of the handwriting input interface with a corresponding device orientation, such that a change in the device orientation causes a change in the writing mode and/or writing direction. Triggered. In some embodiments, the change in writing direction automatically causes the input of the highest ranking recognition result to be input into the text input area.

19A-19F illustrate example user interfaces for user devices that provide both horizontal and vertical input modes.

FIG. 19A shows the user device in horizontal input mode. In some embodiments, a horizontal input mode is provided when the user device is in a landscape display orientation, as shown in FIG. 19A. In some embodiments, a horizontal input mode is optionally associated and provided when the device is operated in a portrait display orientation. For different applications, the relationship between device orientation and writing mode may be different.

In horizontal input mode, a user can provide handwriting in a horizontal writing direction (eg, with a default writing direction going from left to right, or a default writing direction going from right to left). In horizontal input mode, the user device performs segmentation of the handwritten input into one or more recognition units along this horizontal writing direction.

In some embodiments, the user device only allows a single line of input within the handwriting input area. In some embodiments, as shown in FIG. 19A, the user device allows multiple lines of input (eg, two lines of input) within the handwriting input area. In FIG. 19A, the user has provided multiple handwritten strokes in several lines within the handwriting input area 806. Based on the arrangement of handwritten strokes provided by the user and the relative locations of the handwritten strokes and the distances between the strokes, the user device determines that the user is inputting two lines of characters. do. After splitting this handwritten input into two separate lines, the device determines the recognition units within each line.

As shown in FIG. 19A, the user device has recognized a respective character for each recognition unit identified within the current handwriting input 1902, producing several recognition results 1904 and 1906. As further shown in FIG. 19A, in some embodiments, if the output character (e.g., the letter "I") for a particular set of recognition units (e.g., the recognition units formed by the initial stroke) is low , the user device optionally generates partial recognition results (eg, results 1906) that indicate only those output characters with sufficient recognition confidence. In some embodiments, the user understands from this partial recognition result 1906 that the first stroke can be corrected or individually deleted in order for the recognition model to produce the correct recognition result. Can be done. In this particular example, editing the first recognition unit 1904 is not necessary because it indicates the desired recognition result for the first recognition unit.

In this example, as shown in FIGS. 19A and 19B, the user has rotated the device to a portrait viewing orientation (eg, as shown in FIG. 19B). In response to this change in device orientation, the handwriting input interface changes from horizontal input mode to vertical input mode, as shown in FIG. 19B. In vertical input mode, the layout of handwriting input area 804, candidate display area 806, and text input area 808 may differ from that shown in horizontal input mode. The specific layout of the horizontal input mode and vertical input mode can be varied to suit various device geometries and application requirements. In some embodiments, upon rotation of the device orientation and change of input mode, the user device automatically places the top-ranked result (e.g., result 1904) within text input area 808 as text input 1910. input. The orientation and position of cursor 1912 also reflects changes in input mode and writing direction.

In some embodiments, a change in input mode is optionally triggered by a user touching a special input mode selection affordance 1908. In some embodiments, this input mode selection affordance is a graphical user interface element that also indicates the current writing mode, current writing direction, and/or current paragraph direction. In some embodiments, the input mode selection affordance can cycle through all available input modes and writing directions provided by the handwriting input interface 802. As shown in FIG. 19A, affordance 1908 indicates that the current input mode is a horizontal input mode with a left-to-right writing direction and a top-to-bottom paragraph direction. In FIG. 19B, affordance 1908 indicates that the current input mode is a vertical input mode with a top-to-bottom writing direction and a right-to-left paragraph direction. Other combinations of writing direction and paragraph direction are possible according to various embodiments.

As shown in FIG. 19C, the user creates a plurality of new strokes 1914 (e.g., two Chinese characters "
” handwritten strokes). This handwritten input is written in a vertical writing direction. The user device divides this vertical handwriting input into two recognition units and displays two recognition results 1916 and 1918, each including two recognized characters laid out vertically.

19C and 19D show that when a user selects a displayed recognition result (eg, result 1916), the selected recognition result is entered vertically within text entry area 808.

19E and 19F show the user entering a further line of handwritten input 1920 in a vertical writing direction. These lines proceed from left to right, following the traditional Chinese writing paragraph direction. In some embodiments, candidate display area 806 also shows recognition results (eg, results 1922 and 1924) in the same writing direction and paragraph direction as the direction with respect to the handwriting input area. In some embodiments, other writing directions and Paragraph direction can be provided by default.

19E and 19F also show that when a user selects a recognition result (eg, result 1922), the text of the selected recognition result is entered into text entry area 808. As shown in FIG. 19F, the text currently being entered within the text input area 808 is therefore written in horizontal mode, with a left-to-right writing direction, and text written in a horizontal mode, with a writing direction from top to bottom. Including both text written in vertical mode, with orientation. The paragraph direction for horizontal text is top to bottom, while the paragraph direction for vertical text is right to left.

In some embodiments, the user device allows the user to separately establish a preferred writing direction, paragraph direction, for each of the handwriting input area 804, the candidate display area 806, and the text input area 808. . In some embodiments, the user device allows the user to establish a preferred writing direction and paragraph direction for each of the handwriting input area 804, candidate display area 806, and text input area 808 that are associated with the respective device orientations. make it possible to

20A-20C are a flowchart of an example process 2000 for changing the text input direction and handwriting input direction of a user interface. 19A-19F illustrate a process 2000, according to some embodiments.

In some embodiments, a user device determines an orientation of the device (2002). Device orientation and changes in device orientation may be detected by accelerometers and/or other orientation sensing elements within the user device. In some embodiments, the user device provides a handwriting input interface on the device in a horizontal input mode according to the first orientation of the device (2004). Each line of handwritten input input in this horizontal input mode is divided into one or more respective recognition units along the horizontal writing direction. In some embodiments, the device provides a handwriting input interface on the device in a vertical input mode according to the second orientation of the device (2006). Each line of handwritten input input in this vertical input mode is divided into one or more respective recognition units along the vertical writing direction.

In some embodiments, while operating in horizontal input mode (2008), the device detects a change in orientation of the device from a first orientation to a second orientation (2010). In some embodiments, in response to the change in orientation of the device, the device switches from horizontal input mode to vertical input mode (2012). This is shown, for example, in FIGS. 19A and 19B. In some embodiments, while operating in vertical input mode (2014), the user device detects a change in orientation of the device from the second orientation to the first orientation (2016). In some embodiments, in response to the change in orientation of the device, the user device switches from vertical input mode to horizontal input mode (2018). In some embodiments, the association between device orientation and input mode may be reversed as described above.

In some embodiments, while operating in horizontal input mode (2020), the user device receives (2022) a first multi-word handwritten input from a user. In response to the handwritten input of the first plurality of words, the user device presents recognition results of the first plurality of words in a candidate display area of the handwriting input interface according to a horizontal writing direction (2024). This is shown, for example, in FIG. 19A. In some embodiments, while operating in vertical input mode (2026), the user device receives (2028) a second multi-word handwritten input from the user. In response to the handwritten input of the second plurality of words, the user device presents recognition results of the second plurality of words within the candidate display area and according to a vertical writing direction (2030). This is shown, for example, in FIGS. 19C and 19E.

In some embodiments, the user device receives (2032) a first user input selecting a first multi-word recognition result, e.g., as shown in FIGS. 19A, 19B, and the selection is implicitly performed using input to change the input direction (eg, rotation of the device or selection of affordance 1908). The user device receives a second user input selecting a second multi-word recognition result (2034), for example, as shown in FIG. 19C or FIG. 19E. The user device simultaneously displays (2036) corresponding text of the recognition results of the first plurality of words and the recognition result of the second plurality of words in a text input area of the handwriting input interface; The corresponding text of the recognition result is displayed according to a horizontal writing direction, and the corresponding text of the second multi-word recognition result is displayed according to a vertical writing direction. This is shown, for example, in text entry area 808 in FIG. 19F.

In some embodiments, the handwriting input area accepts multiple lines of handwritten input in a horizontal writing direction and has a default top-to-bottom paragraph direction. In some embodiments, this horizontal writing direction is left to right. In some embodiments, this horizontal writing direction is right to left. In some embodiments, the handwriting input area accepts multiple lines of handwritten input in a vertical writing direction and has a default left-to-right paragraph direction. In some embodiments, the handwriting input area accepts multiple lines of handwritten input in a vertical writing direction and has a default right-to-left paragraph direction. In some embodiments, this vertical writing direction is from top to bottom. In some embodiments, the first orientation is a default landscape orientation and the second orientation is a default portrait orientation. In some embodiments, the user device provides a corresponding affordance within the handwriting input interface to manually switch between horizontal and vertical input modes, regardless of the orientation of the device. In some embodiments, the user device provides a corresponding affordance within the handwriting input interface to manually switch between two alternative writing directions. In some embodiments, the user device provides a corresponding affordance within the handwriting input interface to manually switch between two alternative paragraph directions. In some embodiments, this affordance is a toggle button that, when called one or more times in succession, rotates through each available combination of input direction and paragraph direction.

In some embodiments, the user device receives handwritten input from the user (2038). The handwriting input includes a plurality of handwritten strokes provided within a handwriting input area of the handwriting input interface. In response to the handwriting input, the user device displays one or more recognition results within a candidate display area of the handwriting input interface (2040). While the one or more recognition results are displayed within the candidate display area, the user device detects a user input to switch from the current handwriting input mode to an alternative handwriting input mode (2042). . In response to this user input (2044), the user device switches from the current handwriting input mode to an alternate handwriting input mode (2046). In some embodiments, the user device erases the handwriting input from within the handwriting input area (2048). In some embodiments, the user device automatically enters the highest ranking recognition result of the one or more recognition results displayed within the candidate display area into the text input area of the handwriting input interface. (2050). This is shown, for example, in FIGS. 19A and 19B, where the current handwriting input mode is a horizontal input mode and the alternative handwriting input mode is a vertical input mode. In some embodiments, the current handwriting input mode is a vertical input mode and the alternative handwriting input mode is a horizontal input mode. In some embodiments, the current handwriting input mode and the alternate handwriting input mode are modes in which any two different handwriting input directions or paragraph directions are provided. In some embodiments, the user input is a rotation of the device from a current orientation to a different orientation (2052). In some embodiments, this user input is an invocation of an affordance to manually switch the current handwriting input mode to an alternative handwriting input mode.

As described herein, the handwriting input module allows a user to input handwritten strokes and/or characters in any temporal order. Therefore, deleting individual handwritten characters in a multi-character handwritten input and rewriting the same handwritten character or a different handwritten character in the same place as the deleted character is not possible for the user to delete the entire handwritten input. This is advantageous because it helps correct long handwritten entries without having to delete them.

20A-20H illustrate an exemplary user for visually highlighting and/or deleting identified recognition units within a plurality of handwritten strokes currently accumulated within a handwriting input area. Indicates an interface. Enabling a user to individually select, view, and delete any one of a plurality of recognition units identified within a plurality of inputs may be useful for multi-character handwritten input; It is particularly useful if multiple lines of handwritten input are allowed by the user device. By allowing the user to delete a particular recognition unit at the beginning or middle of handwritten input, without requiring the user to delete all recognition units located after the undesired recognition unit. , the user is able to perform corrections to long inputs.

As shown in FIGS. 21A-21C, a user has provided a plurality of handwritten strokes (eg, strokes 2102, 2104, and 2106) within handwriting input area 804 of handwriting input user interface 802. As the user continues to provide additional strokes to the handwriting input area 804, the user device updates and updates the recognition units identified from the handwriting input currently accumulated within the handwriting input area. The recognition result is corrected according to the output characters recognized from the recognized recognition unit. As shown in FIG. 20C, the user device identifies two recognition units from the current handwriting input and presents three recognition results (e.g., 2108, 2010, and 2112), each containing two Chinese characters. are doing.

In this example, after the user writes two handwritten characters, the user realizes that the first recognition unit is written incorrectly, and as a result, the user device identifies and places the desired recognition result within the candidate display area. Understand what is not presented.

In some embodiments, when a user provides a tap gesture (e.g., a contact followed by an immediate lift-off at the same location) on a touch-sensitive display, the user device provides that tap gesture within the handwriting input area. Interpret as input to trigger visual highlighting of the currently identified individual recognition unit. In some embodiments, another predefined gesture (e.g., a multi-finger wipe gesture over the handwriting input area) is used to prompt the user device to identify individual recognition units within the handwriting input area 804. to be highlighted. Tap gestures may be preferred because they are relatively easy to distinguish from handwritten strokes (which typically involve a sustained contact of longer duration and movement of the contact within the handwriting input area 804). Multi-tap gestures may be preferred because they are relatively easy to distinguish from handwritten strokes (which typically involve a single touch within the handwritten input area 804). In some embodiments, the user device allows the user (e.g., via contact 2114) to visually highlight the individual recognition units (e.g., as shown by boxes 2108 and 2110). An affordance 2112 that can be invoked is provided within the user interface. In some embodiments, this affordance is preferred if there is sufficient screen space to accommodate such an affordance. In some embodiments, this affordance can be invoked multiple times in succession by the user, which causes the user device to visually highlight recognition units identified according to different split chains within the split bundle. and turn off the highlighting when all split chains are shown.

As shown in FIG. 21D, when a user provides the necessary gestures to highlight individual recognition units within handwriting input area 804, the user device Further displaying respective delete affordances (eg, small delete buttons 2116 and 2118) above. 21E and 21F show that when a user touches (e.g., via contact 2120) the delete affordance of the respective recognition unit (e.g., the delete button 2116 for the first recognition unit in box 2118), the respective recognition unit Indicates that the recognition unit (eg, within box 2118) is removed from handwriting input area 804. In this particular example, the deleted recognition unit is not the temporally last input recognition unit, nor is it the spatially last recognition unit along the writing direction. In other words, the user can delete any recognition unit, regardless of when and where the recognition unit was provided within the handwriting input area. FIG. 21F shows that in response to the deletion of this first recognition unit within the handwriting input area, the user device also updates the recognition results displayed within candidate display area 806. As shown in FIG. 21F, the user device also deletes the candidate characters corresponding to the deleted recognition units from the recognition results. As a result, new recognition results 2120 are shown within candidate display area 806.

As shown in FIGS. 21G and 21H, after the first recognition unit is removed from the handwriting input interface 804, the user can create multiple recognition units within the area previously occupied by the deleted recognition unit. A new handwritten stroke 2122 is provided. The user device has subdivided the currently accumulated handwriting input within the handwriting input area 804. Based on the recognition units identified from this handwritten input, the user device has regenerated recognition results (eg, results 2124 and 2126) within candidate display area 806. 21G and 21H show that the user has selected (e.g., via contact 2128) one of the recognition results (e.g., result 2124), and the text of the selected recognition result is , is entered into the text input area 808.

22A and 22B show that individual recognition units identified within the current handwriting input are visually presented and can be deleted individually, regardless of the temporal order in which they were formed. 22 is a flowchart for an example process 2200. 21A-21H illustrate a process 2200, according to some embodiments.

In the example process 2200, a user device receives handwritten input from a user (2202). The handwriting input includes a plurality of handwritten strokes provided on a touch-sensitive surface coupled to the device. In some embodiments, the user device renders the plurality of handwritten strokes within a handwriting input area (eg, handwriting input area 804) of the handwriting input interface (2204). In some embodiments, the user device divides (2206) the plurality of handwritten strokes into two or more recognition units, each recognition unit being a corresponding subset of the plurality of handwritten strokes. including.

In some embodiments, the user device receives an edit request from the user (2208). In some embodiments, the edit request is a touch detected (2210) over a predefined affordance (eg, affordance 2112 in FIG. 21D) provided within the handwriting input interface. In some embodiments, the edit request is a tap gesture detected over a predetermined area within the handwriting input interface (2212). In some embodiments, this predetermined area exists within the handwriting input area of the handwriting input interface. In some embodiments, this predetermined area is outside the handwriting input area of the handwriting input interface. In some embodiments, another predefined gesture outside the handwriting input area (eg, crosshair gesture, horizontal swipe gesture, vertical swipe gesture, tilt swipe gesture) may be used as an edit request. Gestures outside the handwriting input area can be easily distinguished from handwritten strokes because they are provided outside the handwriting input area.

In some embodiments, in response to this edit request, the user device visually identifies two or more recognition units within the handwriting input area using, for example, boxes 2108 and 2110 in FIG. 21D. (2214). In some embodiments, visually identifying the two or more recognition units further includes highlighting respective boundaries between the two or more recognition units within the handwriting input area (2216). ). In various embodiments, various methods of visually identifying identified recognition units within the current handwriting input may be used.

In some embodiments, the user device provides a means for individually deleting each of the two or more recognition units from the handwriting input area (2218). In some embodiments, the means for individually deleting each of the two or more recognition units is provided proximal to each recognition unit, e.g., as indicated by delete buttons 2116 and 2118 in FIG. 21D. These are the respective delete buttons that are displayed. In some embodiments, the means for individually deleting each of the two or more recognition units is means for detecting a default deletion gesture input on each recognition unit. In some embodiments, the user device does not visually display a separate deletion affordance above the highlighted recognition unit. Instead, in some embodiments, the user may use a delete gesture to delete the respective recognition unit under the delete gesture. In some embodiments, when the user device is displaying the recognition unit in a visually highlighted manner, the user device does not accept further handwriting strokes within the handwriting input area. Instead, a predetermined gesture or any gesture detected over a visually highlighted recognition unit causes the user device to remove the recognition unit from the handwriting input area and display it within the candidate display area. The resulting recognition results are corrected as appropriate. In some embodiments, the tap gesture causes the user device to visually highlight individual recognition units identified within the handwriting recognition area, and the user then uses a delete button to In the opposite direction, individual recognition units can be deleted one by one.

In some embodiments, the user device inputs a delete input to individually delete a first recognition unit of the two or more recognition units from the handwriting input area, e.g., as shown in FIG. 21E. , from the user through the provided means (2224). In response to the deletion input, the user device removes (2226) the corresponding subset of handwritten strokes within the first recognition unit from the handwriting input area, eg, as shown in FIG. 21F. In some embodiments, the first recognition unit is a spatially first recognition unit within the two or more recognition units. In some embodiments, the first recognition unit is a spatially intermediate recognition unit among the two or more recognition units, eg, as shown in FIGS. 21E and 21F. In some embodiments, the first recognition unit is the spatially last recognition unit of the two or more recognition units.

In some embodiments, the user device generates (2228) a segmented bundle from the plurality of handwritten strokes, each segmented bundle having a set of corresponding recognition units identified from the plurality of handwritten strokes. , including multiple alternative split strands. For example, FIG. 21G shows recognition results 2024 and 2026, where recognition result 2024 is generated from one split chain with two recognition units and recognition result 2026 is generated from another split chain with three recognition units. be done. In some embodiments, the user device receives two or more consecutive edit requests from the user (2230). For example, those two or more consecutive edit requests may be several consecutive taps on affordance 2112 in FIG. 21G. In some embodiments, in response to each of the two or more consecutive editing requests, the user device recognizes, within the handwriting input area, a correspondence from a different one of the plurality of alternative split chains. A set of units is visually identified (2232). For example, in response to a first tap gesture, two (e.g., each character "
"as well as"
”) is highlighted, and in response to the second tap gesture, three (e.g., respectively, the letters “
”, “
",as well as"
”) is highlighted. In some embodiments, in response to a third tap gesture, the visual highlighting is optionally removed from all recognition units and the handwriting input area returns to its normal state to allow further strokes. Ready to accept. In some embodiments, the user device provides a means for individually deleting each of the set of corresponding recognition units currently represented within the handwriting input area (2234). In some embodiments, this means is a separate delete button for each highlighted recognition unit. In some embodiments, the means detects a default delete gesture above each highlighted recognition unit, and deletes the highlighted recognition unit below the default delete gesture. It is a means to call the function to do.

As described herein, in some embodiments, the user device provides a continuous input mode within the handwriting input area. Since the area of the handwriting input area is limited on portable user devices, handwriting input provided by the user can be cached to free up screen space without the user having to commit to previously provided handwriting input. It may be desirable to provide a method to enable reuse. In some embodiments, the user device causes the input area to gradually transition by a certain amount (e.g., one recognition unit at a time) when the user is approaching the end of the handwriting input area sufficiently. , provides a scrolling handwriting input area. In some embodiments, the recognition units are dynamically moved, as migrating existing recognition units within the handwriting input area may interfere with the user's writing process and possibly prevent the recognition units from being correctly split. It may be advantageous to reclaim previously used areas of the input area without migration. In some embodiments, if a user reuses an area occupied by handwriting input that has not yet been entered within the text input area, the top recognition result for that handwriting input area is Automatically entered, the user can continue to provide new handwriting input without explicitly selecting the top-ranked recognition result.

Some conventional systems allow a user to write on top of existing handwriting input that is still shown within the handwriting input area. In such systems, temporal information is used to determine whether a new stroke is part of a previous recognition unit or a new recognition unit. Systems that rely on such temporal information impose strict requirements on the speed and rapidity with which users provide handwritten input, requirements that are difficult for many users to meet. . Furthermore, the visual rendering of the handwritten input can be cluttered that is difficult for the user to decipher. Therefore, the writing process can be frustrating and confusing for the user, resulting in a poor user experience.

To indicate when a user can reuse an area occupied by a previously written recognition unit to continue writing within that handwriting input area, as described herein; A fading process is used. In some embodiments, this fading process gradually reduces the visibility of each recognition unit provided within the handwriting input area for a threshold amount of time when a new stroke is written over it. , existing text does not visually conflict with the new strokes. In some embodiments, by writing over a faded recognition unit, the user can interrupt writing and select input for the top-ranked recognition result without having to explicitly provide selection input for that recognition unit. The recognition result with the highest rank for is automatically entered into the text input area. This implicit and automatic confirmation of the top-ranked recognition results improves the efficiency and speed of input in the handwriting input interface and reduces the cognitive burden imposed on the user to maintain the current flow of thought of the text composition. Reduce. In some embodiments, writing over a faded recognition unit does not cause automatic selection of the top-ranked search result. Instead, the faded recognition unit is cached in the handwriting input stack and combined with the new handwriting input as the current handwriting input. The user can view the recognition results generated based on all of those recognition units accumulated in the handwriting input stack before making a selection.

23A-23J show that recognition units provided within different regions of a handwriting input area gradually fade out from their corresponding regions, e.g. after a predetermined amount of time, and that this fade-out occurs within a particular region. 3 illustrates an example user interface and process that allows a user to provide new handwritten strokes within that region.

As shown in FIG. 23A, the user has provided a plurality of handwritten strokes 2302 (eg, three handwritten strokes for the capital letter "I") within the handwriting input area 804. Handwritten strokes 2302 are identified by the user device as one recognition unit. In some embodiments, the handwriting input currently shown within the handwriting input area 804 is cached in a first layer within the handwriting input stack of the user device. Several recognition results generated based on this identified recognition unit are provided in candidate display area 804.

FIG. 23B shows that as the user continues to write one or more strokes 2302 to the right of stroke 2304, the handwritten strokes 2302 within the first recognition unit begin to gradually fade out within the handwriting input area 804. Show that. In some embodiments, an animation is displayed that mimics a gradual fading or disappearance of the visual rendering of this first recognition unit. For example, this animation can create the visual effect of ink evaporating from a whiteboard. In some embodiments, the fading of the recognition unit is not uniform across the recognition unit. In some embodiments, the fading of the recognition unit increases over time until the recognition unit becomes completely invisible within the handwriting area. However, even if a recognition unit is no longer visible within the handwriting input area 804, in some embodiments the invisible recognition unit is maintained at the top of the handwriting input stack and removed from the recognition unit. The generated recognition results are continuously displayed within the candidate display area. In some embodiments, a faded recognition unit is not completely removed from view until new handwriting input is written over it.

In some embodiments, the user device allows new handwriting input to be provided over the area occupied by the faded recognition unit as soon as the fading animation begins. In some embodiments, the user device detects the new handwriting only after the fading has progressed to a certain stage (e.g., to the faintest level or until recognition is completely invisible within that region). Allows input to be provided over the area occupied by the faded recognition unit.

FIG. 23C shows that the first recognition unit (i.e., stroke 2302) has completed its fading process (e.g., the ink color has stabilized at a very faint level or has become invisible). ). The user device identifies additional recognition units (e.g., recognition units for handwritten characters "a" and "m") from additional handwritten strokes provided by the user and presents recognition results in candidate display area 804. is being updated.

22D-22F show that over time, the user provides multiple additional handwriting strokes (eg, 2304 and 2306) within handwriting input area 804. At the same time, the previously identified recognition units gradually fade away from the handwriting input area 804. In some embodiments, each recognition unit requires a predetermined amount of time to begin its own fading process after the recognition unit is identified. In some embodiments, the fading process for each recognition unit does not begin until the user begins inputting a second recognition unit downstream of that recognition unit. As shown in FIGS. 23B-23F, when the handwriting input is provided in cursive, a single stroke (e.g., stroke 2304 or stroke 2306) may have multiple recognition units (e.g., (recognition unit for each handwritten character within the word "am" or "back").

FIG. 22G shows that even after the recognition unit has started its fading process, the user can perform a default playback input, e.g., indicated by an immediate liftoff following contact 2308 (e.g., on the delete button 2310). indicates that the recognition unit can be restored to its non-fade state by a tap gesture (such as When the recognition unit is regenerated, its appearance returns to its normal visibility level. In some embodiments, the reproduction of faded recognition units is performed character by character in the opposite direction of writing within the handwriting input area 804. In some embodiments, the reproduction of faded recognition units is performed word by word within the handwriting input area 804. As shown in FIG. 23G, the recognition unit corresponding to the word "back" has been regenerated from a fade complete state to a completely non-fade state. In some embodiments, the clock for starting the fading process is reset for each recognition unit when the recognition unit is regenerated to a non-fading state.

FIG. 22H shows that a sustained touch on the delete button causes the last recognition unit in the default writing direction (e.g., the recognition unit for the letter "k" in the word "back") to be deleted from the handwriting input area 804. to show that If this deletion input is maintained continuously, more recognition units (e.g., recognition units for the letters "c", "a", "b" in the word "back") will be written in the opposite direction of writing. They are deleted one by one. In some embodiments, the recognition unit deletion is word-by-word, and all characters of the handwritten word "back" that are deleted from the handwriting input area 804 are removed at the same time. FIG. 22H also shows that when contact 2308 is maintained on delete button 2310 after deletion of the recognition unit for the letter "b" in the handwritten word "back", the previously faded recognition unit "m" is It also shows that it will be played in the same way.

FIG. 23I shows that if the deletion input is stopped before the deletion of the reproduced recognition unit "m" in the handwritten word "am" is performed, the reproduced recognition unit is again gradually Indicates a fade. In some embodiments, the state of each recognition unit (eg, a state selected from a set of one or more fade and non-fade states) is maintained and updated within the handwriting input stack.

FIG. 23J shows a case where the user provides one or more strokes 2312 over the area in the handwriting input area that was occupied by the faded recognition unit (e.g., the recognition unit for the letter "I"). , in some embodiments, the text of the top-ranked recognition result (e.g., result 2314) for the handwriting input performed before the stroke 2312 is included in the text input area 808, as shown in FIGS. 23I, 23J. indicates that it will be automatically entered. As shown in FIG. 23J, the text "I am" is no longer shown as being tentative, but instead has been confirmed within the text entry area 808. In some embodiments, when text input is performed for a fully faded handwriting input or a partially faded handwriting input, the handwriting input is removed from the handwriting input stack. The newly input stroke (eg, stroke 2312) becomes the current input in the handwriting input stack.

As shown in FIG. 23J, the text "I am" is no longer shown as being tentative, but instead has been confirmed within the text entry area 808. In some embodiments, when text input is performed for a fully faded handwriting input or a partially faded handwriting input, the handwriting input is removed from the handwriting input stack. The newly input stroke (eg, stroke 2312) becomes the current input in the handwriting input stack.

In some embodiments, if the stroke 2312 is provided over an area in the handwriting input area that was occupied by a faded recognition unit (e.g., a recognition unit for the letter "I"), the previous stroke 2312 The text of the top-ranked recognition result (eg, result 2314) for the handwritten input performed on the handwriting input is not automatically entered into the text entry area 808. Instead, the current handwriting input (both faded and non-faded) within the handwriting input area 804 is erased and cached in the handwriting input stack. The new stroke 2312 is appended to the cached handwriting input in that handwriting input stack. The user device determines the recognition result based on the total amount of handwriting input currently stored in the handwriting input stack. Those recognition results are displayed within the candidate display area. In other words, even if only a portion of the currently accumulated handwriting input is shown within the handwriting input area 804, the recognition result is cached within the handwriting input stack (the visible portion and no longer visible). generated based on the entire handwritten input (both non-visible parts).

FIG. 23K shows that the user has entered additional strokes 2316 within the handwriting input area 804, and the strokes are fading over time. FIG. 23L shows that a new stroke 2318 written above the faded strokes 2312 and 2316 has entered the text of the top recognition result 2320 for the faded strokes 2312 and 2316 into the text input area 808. Show that.

In some embodiments, the user optionally provides handwritten input in multiple lines. In some embodiments, the same fading process may be used to clear the handwriting input area for new handwriting input when multiple lines of input are valid.

24A and 24B are a flowchart of an example process 2400 for providing a fading process within a handwriting input area of a handwriting input interface. 23A-23K illustrate a process 2400, according to some embodiments.

In some embodiments, the device receives (2402) a first handwritten input from a user. The first handwriting input includes a plurality of handwritten strokes, the plurality of handwritten strokes having a plurality of recognition units distributed along respective writing directions associated with a handwriting input area of the handwriting input interface. Form. In some embodiments, the user device renders each of the plurality of handwritten strokes within the handwriting input area as the handwritten strokes are provided by the user (2404).

In some embodiments, the user device initiates a respective fading process for each of the plurality of recognition units after the recognition units are completely rendered (2406). In some embodiments, during each fading process, the rendering of the recognition unit within the first handwriting input fades out. This is illustrated in FIGS. 23A-23F in some embodiments.

In some embodiments, the user device receives handwriting input that was occupied by the faded recognition unit of the plurality of recognition units, as shown, for example, in FIGS. 23I, 23J, and 23K, 23L. A second handwritten input is received from the user over the area of the area (2408). In some embodiments, in response to receiving this second handwriting input (2410), the user device renders (2412) the second handwriting input within the handwriting input area and all faded The recognition unit is deleted from the handwriting input area (2414). In some embodiments, all recognition units that were entered within the handwriting input area prior to the second handwriting input, whether or not the recognition unit has started its fading process, , erased from the handwriting input area. This is shown, for example, in FIGS. 23I, 23J, and 23K, 23L.

In some embodiments, the user device generates (2416) one or more recognition results regarding the first handwritten input. In some embodiments, the user device displays the one or more recognition results within a candidate display area of the handwriting input interface (2418). In some embodiments, in response to receiving the second handwriting input, the user device selects the top-ranked recognition result displayed within the candidate display area within the text input area of the handwriting input interface without user selection. (2420). This is shown, for example, in FIGS. 23I, 23J, and 23K, 23L.

In some embodiments, the user device stores an input stack that includes the first handwritten input and the second handwritten input (2422). In some embodiments, the user device generates one or more characters, each comprising a corresponding spatial arrangement of characters, recognized from the concatenation of the first handwriting input and the second handwriting input. A recognition result is generated (2424). In some embodiments, the user device is configured to display, within the candidate display area of the handwriting input interface, while the rendering of the second handwriting input replaces the rendering of the first handwriting input within the handwriting input area. The recognition results for one or more of the characters are displayed (2426).

In some embodiments, a respective fading process for each recognition unit is initiated if a predetermined period of time has elapsed after that recognition unit was completed by the user.

In some embodiments, the fading process for each recognition unit is initiated if, after that recognition unit, the user begins inputting strokes for the next recognition unit.

In some embodiments, the end state of each fading process for each recognition unit is a predetermined minimum visibility state for that recognition unit.

In some embodiments, the end state of each fading process for each recognition unit is a state with zero visibility for that recognition unit.

In some embodiments, after the last recognition unit in the first handwritten input fades, the user device receives a default playback input from the user (2428). In response to receiving this default playback input, the user device returns its last recognition unit from the faded state to the non-fade state (2430). This is shown, for example, in FIGS. 23F-23H. In some embodiments, this default playback input is an initial touch detected on a delete button provided within the handwriting input interface. In some embodiments, a sustained touch detected on the delete button deletes the last recognition unit from the handwriting input area and regenerates the penultimate recognition unit from a faded state to a non-fade state. This is shown, for example, in FIGS. 23G and 23H.

As described herein, the multi-script handwriting recognition model performs stroke order independent and stroke direction independent recognition of handwritten characters. In some embodiments, the recognition model is trained only on spatially derived features contained within flat images of writing samples that correspond to various characters within the vocabulary of the handwriting recognition model. Because images of writing samples do not contain any temporal information related to the individual strokes contained within those images, the resulting recognition model is stroke order-independent and stroke direction-independent. It is.

As explained above, stroke order and stroke direction independent handwriting recognition relies on information related to the temporal production of characters (e.g., the temporal arrangement of strokes within those characters). provides many advantages over other recognition systems. However, in real-time handwriting recognition scenarios, temporal information related to individual strokes is available, and it may be beneficial to utilize this information to improve the recognition accuracy of handwriting recognition systems. In the following, a technique is described that integrates temporally derived stroke distribution information in the extraction of spatial features of a handwriting recognition model. This does not impair the stroke order and/or stroke direction independence of the stroke order and/or stroke direction. Based on stroke distribution information associated with various characters, disambiguation between similar-looking characters produced by distinctly different sets of strokes is possible.

In some embodiments, when handwriting input is converted to an input image (e.g., an input bitmap image) for a handwriting recognition model (e.g., CNN), temporal information associated with individual strokes is lost. For example, the Chinese character “
”, eight strokes (eg, labeled #1 to #8 in FIG. 27) can be used to write this character. The arrangement and direction of strokes for this character provides some unique characteristics associated with that character. A naive method to capture stroke order and stroke direction information without compromising the stroke order and stroke direction independence of the recognition system is to include all possible permutations of stroke order and stroke direction in the training samples. , to list them explicitly. However, even for characters of even moderate complexity, this permutation amounts to over a billion possibilities, making it impractical, if not impossible, to implement in practice. As described herein, a stroke distribution profile is generated for each writing sample that abstracts the temporal aspects (i.e., temporal information) of stroke generation. The stroke distribution profiles of those writing samples are trained to extract a set of temporally derived features, which are then extracted from the input bitmap image (e.g. ) in combination with spatially derived features to improve recognition accuracy without affecting the stroke order and stroke direction independence of handwriting recognition systems.

As described herein, temporal information associated with a single character is extracted by calculating various pixel distributions that characterize each handwritten stroke. Every handwritten stroke of a character produces a deterministic pattern (or profile) when projected onto a given direction. This pattern by itself may not be sufficient to clearly recognize that stroke, but when combined with other similar patterns it is adequate to capture certain characteristics unique to this particular stroke. It can be. This type of stroke representation can also be integrated with spatial feature extraction (e.g., feature extraction based on input images in a CNN) to address disambiguation between similar-looking characters in the repertoire of handwriting recognition models. Orthogonality information is provided, which can be useful for removing morphism.

25A and 25B are flowcharts of an example process 2500 for integrating temporally and spatially derived features of handwriting samples during training of a handwriting recognition model. , in which case the resulting recognition model remains stroke order and stroke direction independent. In some embodiments, example process 2500 is executed on a server device that provides the trained recognition model to a user device (eg, portable device 100). In some embodiments, the server device includes one or more processors and memory that includes instructions that, when executed by the one or more processors, perform process 2500.

In the example process 2500, the device separately trains (2502) a set of spatially derived features and a set of temporally derived features of a handwriting recognition model; The set of features is trained on a corpus of training images, each of which is an image of handwritten samples for each character of the output character set, and the set of temporally derived features is trained on a corpus of stroke distribution profiles. , each stroke distribution profile numerically characterizes the spatial distribution of multiple strokes within the handwriting sample for each character of the output character set.

In some embodiments, separately training the set of spatially derived features further comprises training a convolutional neural network having an input layer, an output layer, and a plurality of convolutional layers (2504 ), the plurality of convolutional layers includes an initial convolutional layer, a final convolutional layer, zero or more intermediate convolutional layers between the initial convolutional layer and the final convolutional layer, and a hidden layer between the final convolutional layer and the output layer. Contains layers. An example convolutional network 2602 is shown in FIG. 26. This example convolutional network 2602 may be implemented in substantially the same manner as convolutional network 602 shown in FIG. 6. Convolutional network 2602 includes an input layer 2606, an output layer 2608, and a plurality of convolutional layers, including an initial convolutional layer 2610a, zero or more intermediate convolutional layers, and a final convolutional layer 2610n. A hidden layer 2614 is included between the final convolutional layer and the output layer 2608. Convolutional network 2602 also includes a kernel layer 2616 and a subsampling layer 2612, according to the arrangement shown in FIG. The training of this convolutional network is based on images 2614 of writing samples in the training corpus 2604. Spatially derived features are obtained and respective weights associated with those various features are determined by minimizing recognition error with respect to training samples within the training corpus. Those same features and weights are trained and then used for recognition of new handwriting samples that are not present in the training corpus.

In some embodiments, training the set of temporally derived features separately provides multiple stroke distribution profiles to the statistical model to classify each character in the output character set. The method further includes determining a plurality of temporally derived parameters and respective weights for the plurality of temporally derived parameters (2506). In some embodiments, as shown in FIG. 26, a stroke distribution profile 2620 is derived from each writing sample in a training corpus 2622. Training corpus 2622 optionally includes the same writing samples as corpus 2604, but also includes temporal information associated with stroke generation within each writing sample. The stroke distribution profile 2622 is provided to a statistical modeling process 2624 during which temporally derived features are extracted and respective weights for those various features are determined using statistical modeling methods (e.g. (CNN, K-Nearest Neighbors, etc.), by minimizing recognition or classification errors. As shown in FIG. 26, the set of temporally derived features and their respective weights are transformed into a set of feature vectors (e.g., feature vector 2626 or feature vector 2628) and their respective weights in convolutional neural network 2602. Introduced within the layer. The resulting network therefore contains spatially derived parameters and temporally derived parameters that are orthogonal to each other and jointly contribute to character recognition. .

In some embodiments, the device combines (2508) a set of spatially derived features and a set of temporally derived features within a handwriting recognition model. In some embodiments, combining the set of spatially derived features and the set of temporally derived features within the handwriting recognition model is performed in one of the convolutional layers of the convolutional neural network. or introducing within the hidden layer a plurality of spatially derived parameters and a plurality of temporally derived parameters (2510). In some embodiments, the plurality of temporally derived parameters and the respective weights for the plurality of temporally derived parameters are determined in the final convolutional layer of the convolutional neural network for handwriting recognition (e.g., Figure 26 is introduced into the final convolutional layer 2610n). In some embodiments, the plurality of temporally derived parameters and the respective weights for the plurality of temporally derived parameters are defined in the hidden layers of a convolutional neural network for handwriting recognition (e.g., as shown in FIG. 26). hidden layer 2614).

In some embodiments, the device uses this handwriting recognition model to provide real-time handwriting recognition for the user's handwriting input (2512).

In some embodiments, the device generates a corpus of stroke distribution profiles from the plurality of writing samples (2514). In some embodiments, each of the plurality of handwriting samples corresponds to a character in the output character set (2516), and for each constituent stroke of the handwriting sample, the respective spatial Store information separately. In some embodiments, to generate a corpus of stroke distribution profiles, the device performs the following steps (2518).

For each of the plurality of handwriting samples (2520), the device identifies (2522) constituent strokes within the handwriting sample, and for each of the identified strokes of the handwriting sample, the device identifies each of the plurality of predetermined directions. (2524), where the occupancy is the ratio of the projected span in each stroke direction to the maximum projected span of the writing sample, and For each, the device also calculates a respective saturation rate for each stroke based on the ratio of the respective number of pixels in each stroke to the total number of pixels in the written sample (2526). The user device then generates (2528) a feature vector for the handwritten sample as a stroke distribution profile of the handwritten sample, the feature vector representing the respective occupancies and occupancies of the at least N strokes in the handwritten sample. Including each saturation rate, N is a predetermined natural number. In some embodiments, N is less than the maximum number of strokes observed in any single writing sample within the plurality of writing samples.

In some embodiments, for each of the plurality of handwriting samples, the device sorts the occupancy of each of the identified strokes in each of the predetermined directions in descending order, and arranges the occupancy of each of the N top-ranked strokes in descending order. Only the written samples with and saturation rate are included in the feature vector of that written sample.

In some embodiments, the plurality of predetermined directions include a written sample horizontal direction, a vertical direction, a positive 45 degree direction, and a negative 45 degree direction.

In some embodiments, to provide real-time handwriting recognition regarding a user's handwriting input using this handwriting recognition model, the device receives the user's handwriting input and responds to the receipt of the user's handwriting input. and provides handwriting recognition output to the user substantially simultaneously with receiving the handwriting input.

The characters shown in Figure 27 “
” are used herein to describe example embodiments for purposes of explanation. In some embodiments, each input image of handwritten characters is optionally normalized to a square. Measure the span of each individual handwritten stroke (e.g., strokes #1, #2...and #3) when projected horizontally, vertically, diagonally +45 degrees, and diagonally -45 degrees on this square. be done. The span of each stroke Si is recorded as x span (i), y span (i), c span (i), and d span (i) for the four projection directions, respectively. Additionally, the maximum span observed across the image is also recorded. The maximum spans of this character are recorded as x span, y span, c span, and d span, respectively, for the four projection directions. For purposes of explanation, four projection directions are optionally considered herein, but in principle any arbitrary set of projections could be used in various embodiments. I can do it. The maximum span (e.g., designated as x span, y span, c span, and d span) and in the four projection directions, the character "
'' (e.g., stroke #4) has a span (e.g., shown as x-span (4), y-span (4), c-span (4), and d-span (4)) , shown in FIG.

In some embodiments, the span is measured for all strokes 1-5 (where 5 is the number of distinct handwritten strokes associated with this input image) along each projection direction. , the respective occupancy rates are calculated. For example, the corresponding occupancy rate R _x (i) along the x direction for the stroke S _i is calculated as R _x (i)=x span (i)/x span. Similarly, the corresponding occupancy along other projection directions can be calculated, R _y (i) = y span (i) / y span, R _c (i) = c span (i) / c span, R _d (i) = d span (i)/d span.

In some embodiments, the occupancy of all strokes in each direction is sorted separately in descending order, such that the respective ranking of all strokes in that input image is , taken in terms of the occupancy of those strokes in that direction. The ranking of strokes in each projection direction reflects the relative importance of each stroke along the associated projection direction. This relative importance is independent of the order and direction in which the strokes were produced within the writing sample. Therefore, this occupancy-based ranking is stroke order and stroke direction independent, time-derived information.

In some embodiments, each stroke is given a relative weight that indicates the importance of the stroke relative to the overall character. In some embodiments, this weight is measured by the ratio of the number of pixels in each stroke to the total number of pixels in the character. This ratio is referred to as the saturation rate associated with each stroke.

In some embodiments, a feature vector can be created for each stroke based on the occupancy and saturation rate of each stroke. For each character, a set of feature vectors is created containing 5S number of features. This set of features is called the stroke distribution profile of the character.

In some embodiments, only a predetermined number of top ranked strokes are used in constructing the stroke distribution profile for each character. In some embodiments, this predetermined number of strokes is ten. Based on those top 10 strokes, 50 stroke-derived features are generated for each character. In some embodiments, these features are introduced in the final convolutional layer of the convolutional neural network or in subsequent hidden layers.

In some embodiments, during real-time recognition, an input image of a recognition unit is provided to a handwriting recognition mode that is trained on both spatially derived features and temporally derived features. The input image is processed through each layer of the handwriting recognition model shown in FIG. When the processing of this input image reaches a layer where stroke distribution profile input is required (eg, a final convolutional layer or a hidden layer), the stroke distribution profile of that recognition unit is introduced into that layer. Processing of this input image and stroke distribution profile continues until an output classification (eg, one or more candidate characters) is provided at output layer 2608. In some embodiments, the stroke distribution profiles of all recognition units are computed and provided as input to a handwriting recognition model, along with the input images of those recognition units. In some embodiments, an input image of a recognition unit is first passed through a handwriting recognition model (without the benefit of temporally trained features). If two or more similar-looking candidate characters are identified with close recognition confidence values, then the stroke distribution profile of that recognition unit is determined by a layer trained with temporally derived features (e.g. , final convolutional layer, or hidden layer) into the handwriting recognition model. When the input image and stroke distribution profile of that recognition unit pass through the final layer of the handwriting recognition model, two or more similar-looking candidate characters can be better distinguished by the difference in their stroke distribution profiles. can. Therefore, temporally derived information about how recognition units are formed by individual handwritten strokes can be used to improve recognition without compromising the stroke order and stroke direction independence of handwriting recognition systems. Improves accuracy.

The foregoing description has been described with reference to specific embodiments for purposes of explanation. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. These embodiments are intended to best explain the principles of the invention and its practical application, and are intended to enable others skilled in the art to make various modifications as appropriate to the specific application contemplated. They were chosen and described in order to enable the invention and its various embodiments to be best utilized.

Claims (12)

A method,
an electronic device having one or more processors, a touch-sensitive surface and a display;
receiving handwritten input, the handwritten input comprising a plurality of handwritten strokes provided on the touch-sensitive surface;
Displaying the plurality of handwritten strokes within a handwriting input area of a handwriting input interface;
dividing the plurality of handwritten strokes into two or more recognition units, each recognition unit comprising a distinct subset of the plurality of handwritten strokes;
receiving editing requests; and
visually identifying at least one of the two or more recognition units within the handwriting input area in response to the editing request;
displaying separate delete buttons in proximity to each of the two or more recognition units;
detecting a default deletion gesture input on the at least one of the two or more recognition units within the handwriting input area;
in response to detecting the predetermined deletion gesture input, deleting the discrete subset of handwritten strokes corresponding to the at least one of the two or more recognition units within the handwriting input area;
A method, comprising: 12. Visually identifying the two or more recognition units further comprises highlighting distinct boundaries between the two or more recognition units within the handwriting input area. The method described in Section 1 . 3. A method according to claim 1 or 2 , characterized in that the editing request corresponds to a touch detected on a predetermined affordance displayed in the handwriting input interface. 4. A method according to any preceding claim, characterized in that the editing request corresponds to a tap gesture detected on a predetermined area within the handwriting input interface. 5. The method of claim 4 , wherein the predetermined area corresponds to an area within the handwriting input area of the handwriting input interface. 5. The method of claim 4 , wherein the predetermined area corresponds to an area outside the handwriting input area of the handwriting input interface. Any one of claims 1 to 6 , characterized in that the at least one of the two or more recognition units corresponds to a spatially first recognition unit within the two or more recognition units. The method described in. Any one of claims 1 to 6 , wherein the at least one of the two or more recognition units corresponds to a spatially intermediate recognition unit among the two or more recognition units. The method described in section. A method,
an electronic device having one or more processors, a touch-sensitive surface and a display;
receiving handwritten input, the handwritten input comprising a plurality of handwritten strokes provided on the touch-sensitive surface;
Displaying the plurality of handwritten strokes within a handwriting input area of a handwriting input interface;
dividing the plurality of handwritten strokes into two or more recognition units, each recognition unit comprising a distinct subset of the plurality of handwritten strokes;
receiving editing requests; and
visually identifying at least one of the two or more recognition units within the handwriting input area in response to the editing request;
detecting a default deletion gesture input on the at least one of the two or more recognition units within the handwriting input area;
in response to detecting the predetermined deletion gesture input, deleting the discrete subset of handwritten strokes corresponding to the at least one of the two or more recognition units within the handwriting input area;
generating a segmented bundle from the plurality of handwritten strokes, the segmented bundle comprising a plurality of alternative segmentation chains, each representing a distinct set of recognition units identified from the plurality of handwritten strokes; And,
receiving two or more consecutive edit requests;
visually identifying, within the handwriting input area, the distinct set of recognition units from a different one of the plurality of alternative split chains in response to each of the two or more consecutive editing requests; And,
detecting a second default delete gesture input on at least one of the recognition units currently represented within the handwriting input area;
in response to detecting the second predetermined deletion gesture input, deleting the discrete subset of handwritten strokes corresponding to the at least one of the recognition units currently represented within the handwriting input area; to do and
A method, comprising : A computer program product comprising instructions for carrying out a method according to any one of claims 1 to 9 . A memory storing the computer program according to claim 10 ;
one or more processors capable of executing the instructions stored in the memory;
An electronic device comprising: Electronic device comprising means for carrying out the method according to any one of claims 1 to 9 .

Images