JP6154550B2 - Real-time handwriting recognition management - Google Patents

Description

  This document relates to providing handwriting input functionality on a computing device, and more particularly, real-time, multi-script, stroke order independent, handwriting recognition and input on a computing device. Relating to providing functionality.

  The handwriting input method is an important alternative input method for computing devices equipped with a touch-sensitive surface (eg, touch-sensitive display screen or touchpad). Many users, especially those in some Asian or Arab countries, are accustomed to writing in cursive, and feel comfortable writing with normal handwriting, as opposed to typing on the keyboard. it can.

  For certain ideographic character writing systems, such as Chinese Kanji and Japanese Kanji (also called Chinese characters), alternative syllable input methods (eg Pinyin or Kana) have corresponding ideographic characters. If the user does not know how to pronounce and spell a phonetic character, even though it can be used to enter characters in the writing system, and an incorrect phonetic expression of that phonetic character Such a syllable input method is inappropriate when using spelling. Therefore, the ability to use handwriting input on a computing device is extremely important for users who are unable to pronounce enough or no words with respect to the associated ideographic writing system.

  While the functionality of handwriting input has become popular to some extent within certain regions of the world, improvements are still needed. Specifically, human handwriting is very variable (eg, in terms of stroke order, size, typeface, etc.), and high quality handwriting recognition software is complex and requires extensive training. Therefore, providing efficient real-time handwriting recognition on mobile devices with limited memory and computing resources has become a difficult task.

  Furthermore, in today's multicultural world, users in many countries use many languages and write in more than one script (eg write Chinese messages that mention movie titles in English) )) May be needed frequently. However, manually switching the recognition system to the desired script or language during writing is cumbersome and inefficient. Furthermore, the usefulness of conventional multi-script handwriting recognition technology is strictly limited, but this expands the device's recognition capability to handle multiple scripts simultaneously, making the recognition system complex. This is to greatly increase the demands on performance and computer resources.

  Furthermore, conventional handwriting techniques rely heavily on language or script specific peculiarities to achieve recognition accuracy. Such peculiarities are not easily portable to other languages or scripts. Therefore, adding handwriting input capabilities for new languages or scripts is a difficult task that is not easily undertaken by software and device suppliers. As a result, multilingual users are not provided with significant alternative input methods for their electronic devices.

  A conventional user interface for providing handwriting input includes an area for accepting handwriting input from a user and an area for displaying handwriting recognition results. On portable devices with small form factors, significant improvements in user interface are still needed to improve efficiency, accuracy, and overall user experience.

  This specification describes a technique for providing multi-script handwriting recognition using a universal recognition unit. This universal recognizer is trained using a large multi-script corpus of written samples for characters in various languages or scripts. This general-purpose recognition unit training is a language-independent expression, a script-independent expression, a stroke order-independent expression, and a stroke direction-independent expression. Therefore, the same recognition unit can recognize handwritten input of mixed languages and mixed scripts without requiring manual switching between input languages during use. Furthermore, the universal recognizer is lightweight enough to be deployed as a stand-alone module on a mobile device, allowing handwriting input in various languages and scripts used in various areas around the world. It becomes possible.

  In addition, this universal recognizer is trained on spatially derived features of stroke order independent and stroke direction independent and does not require temporal or sequence information at the stroke level, This universal recognizer provides a number of additional features and advantages over conventional time-based recognition methods (eg, recognition methods based on Hidden Markov Methods (HMM)). For example, the user can still obtain the same recognition result by inputting one or more characters, phrases, and sentence strokes in any order. Therefore, out-of-order multiple character entry and out-of-order correction (eg, addition or rewriting) of previously entered characters are possible in this case.

  Furthermore, this general-purpose recognition unit is also used for real-time handwriting recognition, in which case temporal information about each stroke is available, and ambiguity of handwriting input before character recognition is performed by the general-purpose recognition unit. Is optionally used to remove or split handwritten input. The real-time, stroke order-independent recognition described herein is different from traditional offline recognition methods (eg, optical character recognition (OCR)) and Good performance can be provided. In addition, the universal recognizer described herein explicitly identifies significant features of various variations (eg, variations in speed, slowness, stroke order, stroke direction, stroke continuity, etc.) within the recognition system. The ability to handle high variability in individual writing (for example, variability in speed, slowness, stroke order, stroke direction, stroke continuity, etc.) without incorporating it into the overall recognition system Complexity is reduced.

  As described herein, in some embodiments, temporally derived stroke distribution information is optionally re-introduced into the universal recognition unit to improve recognition accuracy and the same input. The ambiguity between similar visual recognition outputs for images is removed. The reintroduction of the temporally derived stroke distribution information does not impair the stroke order and stroke direction independence of the general-purpose recognition unit, but this is spatially derived from the temporally derived features. This feature is obtained through a separate training process and is combined in the handwriting recognition model only after the separate training process is completed. In addition, temporally derived stroke distribution information captures significant temporal characteristics of similar-looking characters without relying on explicit knowledge of stroke order differences between similar-looking characters. To be carefully designed.

  A user interface for providing handwriting input functionality is also described herein.

  In some embodiments, a method for providing multiscript handwriting recognition is to train a multiscript handwriting recognition model based on spatially derived features of a multiscript training corpus, the multiscript training The multi-script handwriting recognition that the corpus is trained on the spatially derived features of the multi-script training corpus, including each hand-written sample corresponding to at least three non-overlapping script characters Using the model to provide real-time handwriting recognition for the user's handwriting input.

  In some embodiments, a method for providing multi-script handwriting recognition is to receive a multi-script handwriting recognition model, wherein the multi-script recognition model is for spatially derived features of a multi-script training corpus. Trained, the multiscript training corpus receives a multiscript handwriting recognition model including each handwritten sample corresponding to at least three non-overlapping script characters, and receives handwritten input from a user And receiving the handwriting input including one or more handwriting strokes provided on a touch-sensitive surface coupled to the user device, and in response to receiving the handwriting input, On spatially derived features of multiscript training corpora It is kneaded, based on the multi-Script handwriting recognition models, one or more handwriting recognition results, and providing real-time to the user.

  In some embodiments, a method for providing real-time handwriting recognition is to receive a plurality of handwriting strokes from a user, the plurality of handwriting strokes corresponding to one handwritten character. Receiving an input image, generating an input image based on the plurality of handwritten strokes, and providing an input image to a handwriting recognition model to perform real-time recognition of the handwritten character, This handwriting recognition model provides input images that provide stroke order independent handwriting recognition, and receives multiple handwritten strokes regardless of their order of multiple handwritten strokes received from the user Displaying the same first output character in real time.

  In some embodiments, the method receives a second plurality of handwritten strokes from the user, the second plurality of handwritten strokes corresponding to the second handwritten character. And generating a second input image based on the second plurality of handwritten strokes, and executing the real-time recognition of the second handwritten character, the second input image in the handwriting recognition model And displaying a second output character corresponding to the second plurality of handwritten strokes in real time to receive the second plurality of handwritten strokes, the first output character and the second Are output simultaneously in a spatial arrangement, independent of the respective order of the first plurality of handwriting inputs and the second plurality of handwriting inputs provided by the user, Further and displaying the force character.

  In some embodiments, the second plurality of handwriting strokes spatially follows the first plurality of handwriting strokes along the default writing direction of the handwriting input interface of the user device and the second output. The character follows the first output character in a spatial arrangement along its default writing direction, and this method is to receive a third handwritten stroke from the user to correct the handwritten character. The third handwriting stroke is received in time after the first plurality of handwriting strokes and the second plurality of handwriting strokes, and the third handwriting stroke is received. In response to receiving the handwriting stroke, the first plurality of handwriting strokes based on the relative proximity of the third handwriting stroke to the first plurality of handwriting strokes. Assigning the handwriting stroke to the same recognition unit as the writing stroke, generating a corrected input image based on the first plurality of handwriting strokes and the third handwriting stroke, and correcting the handwritten character Providing the corrected input image to the handwriting recognition model to perform real-time recognition and receiving the third handwritten input, the third output character corresponding to the corrected input image is received. A third output, wherein the third output character is replaced with the first output character and displayed simultaneously with the second output character in a spatial arrangement along the default writing direction. And further displaying characters.

  In some embodiments, the method includes a deletion input from the user while the third output character and the second output character are simultaneously displayed as recognition results in the candidate display area of the handwriting input interface. Receiving further and deleting the second output character from the recognition result in response to the deletion input while maintaining the third output character in the recognition result.

  In some embodiments, a first plurality of handwriting strokes, a second plurality of handwriting strokes, and a third handwriting stroke are each included in a handwriting input area of the handwriting input interface, each of the handwriting strokes by a user. Rendering each of the second plurality of handwriting strokes from the handwriting input area in response to receiving the deletion input and rendering in real time as provided, while the first plurality of handwriting strokes And maintaining the respective rendering of the third handwriting stroke within the handwriting input area.

  In some embodiments, a method for providing real-time handwriting recognition is to receive handwriting input from a user, wherein the handwriting input is provided within a handwriting input area of a handwriting input interface. Receiving handwritten input, including handwritten strokes, identifying a plurality of output characters for the handwriting input based on a handwriting recognition model, and determining the plurality of output characters based on a predefined categorization criterion By classifying into two or more categories, and displaying each output character in the first category of the two or more categories in the initial view of the candidate display area of the handwriting input interface And an initial view of this candidate display area is provided at the same time as an affordance for invoking an expanded view of the candidate display area Displaying output characters, selecting affordances for invoking this extended view, receiving user input, and in response to the user input, within the expanded view of the candidate display area, within the first category Displaying each output character and each output character in at least a second of the two or more categories that were not previously displayed in the initial view of the candidate display area. Including.

  In some embodiments, a method for providing real-time handwriting recognition is to receive handwriting input from a user, wherein the handwriting input is provided within a handwriting input area of a handwriting input interface. Receiving a handwritten input, and recognizing a plurality of output characters from the handwritten input based on a handwriting recognition model, the output characters being at least a first pictogram and a natural human language Recognizing a plurality of output characters including at least a first character from the script, and a recognition result including the first pictogram and the first character from a script in a natural human language as a handwriting input interface Display within the candidate display area.

  In some embodiments, a method for providing handwriting recognition is to receive handwriting input from a user, wherein the handwriting input is provided in a touch-sensitive surface coupled to the device. Receiving the handwriting input, rendering the plurality of handwriting strokes in real time within the handwriting input area of the handwriting input interface, pinching gesture input on the plurality of handwriting strokes, and Receiving a first of the enlarged gesture inputs and receiving a pinch gesture input, by processing the plurality of handwritten strokes as a single recognition unit, Multiple handwriting when generating recognition results and receiving enlarged gesture input Generating a second recognition result based on the plurality of handwritten strokes by processing the stroke as two separate recognition units separated by an enlarged gesture input; Displaying one of the two recognition results in the candidate display area of the handwriting input interface when generating one of the two recognition results.

  In some embodiments, a method for providing handwriting recognition is to receive handwriting input from a user, wherein the handwriting input comprises a plurality of handwriting strokes provided within a handwriting input area of a handwriting input interface. And identifying a plurality of recognition units from the plurality of handwritten strokes, each recognition unit identifying a plurality of recognition units including a corresponding subset of the plurality of handwritten strokes Generating a recognition result of a plurality of characters including each character recognized from a plurality of recognition units, and displaying the recognition result of the plurality of characters in a candidate display area of a handwriting input interface. , Receiving the deletion input from the user and receiving the deletion input while the recognition result of the plurality of characters is displayed in the candidate display area. Depending on, from the recognition results of a plurality characters displayed in the candidate display area, and removing the last character.

  In some embodiments, a method for providing real-time handwriting recognition provides a handwriting input interface in a horizontal input mode on a device in accordance with determining the orientation of the device and that the device is in a first orientation. In the horizontal input mode, each line of handwritten input input in this horizontal input mode is divided into one or more respective recognition units along the horizontal writing direction. And providing a handwriting input interface on the device in vertical input mode according to the device being in the second orientation, wherein each line of handwriting input input in this vertical input mode is Providing in a vertical input mode that is divided along the vertical writing direction into one or more respective recognition units.

  In some embodiments, a method for providing real-time handwriting recognition is to receive handwriting input from a user, wherein the handwriting input is provided on a touch-sensitive surface coupled to the device. Receiving handwritten input, including strokes, rendering the plurality of handwritten strokes within a handwriting input area of the handwriting input interface, and converting the plurality of handwritten strokes into two or more recognition units Splitting into two or more recognition units, each recognition unit including a corresponding subset of the plurality of handwritten strokes; receiving an edit request from a user; In response to the editing request, two or more recognition units in the handwriting input area are visually identified and handwritten input From frequency, each of the two or more recognition units, and providing a means to remove individual.

  In some embodiments, a method for providing real-time handwriting recognition is to receive a first handwriting input from a user, the first handwriting input including a plurality of handwriting strokes, Receiving a first handwriting input forming a plurality of recognition units, wherein the handwriting strokes are distributed along respective writing directions associated with the handwriting input area of the handwriting input interface; Rendering each of those multiple handwritten strokes within the handwriting input area as provided by the user, and each fading for each of the plurality of recognition units after the recognition unit is fully rendered Starting the process, between each fading process The rendering of the recognition units in the first handwriting input gradually fades, starting the respective fading process and the area of the handwriting input area occupied by the faded recognition units of the plurality of recognition units And receiving a second handwritten input from the user, rendering the second handwritten input in the handwritten input area in response to receiving the second handwritten input, and all fading Erasing the recognized recognition unit from the handwriting input area.

  In some embodiments, a method for providing handwriting recognition is to separately train a set of spatially derived features and a set of temporally derived features of a handwriting recognition model. The set of spatially derived features is trained on a corpus of training images, each of which is an image of a handwritten sample for each character of the output character set, and the set of temporally derived features is a stroke Trained on the distribution profile corpus, each stroke distribution profile numerically characterizes the spatial distribution of multiple strokes in the handwriting sample for each character of the output character set, and in the handwriting recognition model, Train separately, combining a set of spatially derived features and a set of temporally derived features DOO DOO, and a possible use that handwriting recognition model, provides a real-time handwriting recognition Handwritten input from the user.

  The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the present subject matter will be apparent from the description, drawings, and claims.

FIG. 3 is a block diagram illustrating a portable multifunction device with a touch-sensitive display, according to some embodiments.

1 illustrates a portable multifunction device having a touch-sensitive display, according to some embodiments.

FIG. 2 is a block diagram of an exemplary multifunction device comprising a display and a touch sensitive surface, according to some embodiments.

6 illustrates an exemplary user interface for a multifunction device with a touch-sensitive surface separate from a display, according to some embodiments.

FIG. 3 is a block diagram of an operating environment of a handwriting input system, according to some implementations.

FIG. 3 is a block diagram of a multi-script handwriting recognition model according to some embodiments.

2 is a flowchart of an exemplary process for training a multi-script handwriting recognition model, according to some embodiments.

6 illustrates an exemplary user interface showing real-time multi-script handwriting recognition and input on a portable multifunction device, according to some embodiments. 6 illustrates an exemplary user interface showing real-time multi-script handwriting recognition and input on a portable multifunction device, according to some embodiments.

2 is a flow diagram of an example process for providing real-time multiscript handwriting recognition and input on a portable multifunction device. 2 is a flow diagram of an example process for providing real-time multiscript handwriting recognition and input on a portable multifunction device.

2 is a flow diagram of an example process for providing real-time, stroke order-independent handwriting recognition and input on a portable multifunction device, according to some embodiments. 2 is a flow diagram of an example process for providing real-time, stroke order-independent handwriting recognition and input on a portable multifunction device, according to some embodiments. FIG. 6 is a flow diagram of an example process for providing real-time, stroke order-independent handwriting recognition and input on a portable multifunction device, according to some embodiments.

According to some embodiments, one category of recognition results is selectively displayed in the normal view of the candidate display area, and another category of recognition results is selectively displayed in the expanded view of the candidate display area. 2 illustrates an exemplary user interface for doing so. According to some embodiments, one category of recognition results is selectively displayed in the normal view of the candidate display area, and another category of recognition results is selectively displayed in the expanded view of the candidate display area. 2 illustrates an exemplary user interface for doing so. According to some embodiments, one category of recognition results is selectively displayed in the normal view of the candidate display area, and another category of recognition results is selectively displayed in the expanded view of the candidate display area. 2 illustrates an exemplary user interface for doing so. According to some embodiments, one category of recognition results is selectively displayed in the normal view of the candidate display area, and another category of recognition results is selectively displayed in the expanded view of the candidate display area. 2 illustrates an exemplary user interface for doing so. According to some embodiments, one category of recognition results is selectively displayed in the normal view of the candidate display area, and another category of recognition results is selectively displayed in the expanded view of the candidate display area. 2 illustrates an exemplary user interface for doing so. According to some embodiments, one category of recognition results is selectively displayed in the normal view of the candidate display area, and another category of recognition results is selectively displayed in the expanded view of the candidate display area. 2 illustrates an exemplary user interface for doing so. According to some embodiments, one category of recognition results is selectively displayed in the normal view of the candidate display area, and another category of recognition results is selectively displayed in the expanded view of the candidate display area. 2 illustrates an exemplary user interface for doing so. According to some embodiments, one category of recognition results is selectively displayed in the normal view of the candidate display area, and another category of recognition results is selectively displayed in the expanded view of the candidate display area. 2 illustrates an exemplary user interface for doing so. According to some embodiments, one category of recognition results is selectively displayed in the normal view of the candidate display area, and another category of recognition results is selectively displayed in the expanded view of the candidate display area. 2 illustrates an exemplary user interface for doing so. According to some embodiments, one category of recognition results is selectively displayed in the normal view of the candidate display area, and another category of recognition results is selectively displayed in the expanded view of the candidate display area. 2 illustrates an exemplary user interface for doing so. According to some embodiments, one category of recognition results is selectively displayed in the normal view of the candidate display area, and another category of recognition results is selectively displayed in the expanded view of the candidate display area. 2 illustrates an exemplary user interface for doing so.

According to some embodiments, one category of recognition results is selectively displayed in the normal view of the candidate display area, and another category of recognition results is selectively displayed in the expanded view of the candidate display area. 2 is a flowchart of an exemplary process for doing so. According to some embodiments, one category of recognition results is selectively displayed in the normal view of the candidate display area, and another category of recognition results is selectively displayed in the expanded view of the candidate display area. 2 is a flowchart of an exemplary process for doing so.

6 illustrates an exemplary user interface for inputting pictograms through handwriting input, according to some embodiments. 6 illustrates an exemplary user interface for inputting pictograms through handwriting input, according to some embodiments. 6 illustrates an exemplary user interface for inputting pictograms through handwriting input, according to some embodiments. 6 illustrates an exemplary user interface for inputting pictograms through handwriting input, according to some embodiments. 6 illustrates an exemplary user interface for inputting pictograms through handwriting input, according to some embodiments.

3 is a flow diagram of an example process for inputting pictograms through handwriting input, according to some embodiments.

Notifying the handwriting input module how to separate the currently stored handwriting input into one or more recognition units using a pinch gesture or an enlarged gesture, according to some embodiments. 2 shows an exemplary user interface for Notifying the handwriting input module how to separate the currently stored handwriting input into one or more recognition units using a pinch gesture or an enlarged gesture, according to some embodiments. 2 shows an exemplary user interface for Notifying the handwriting input module how to separate the currently stored handwriting input into one or more recognition units using a pinch gesture or an enlarged gesture, according to some embodiments. 2 shows an exemplary user interface for Notifying the handwriting input module how to separate the currently stored handwriting input into one or more recognition units using a pinch gesture or an enlarged gesture, according to some embodiments. 2 shows an exemplary user interface for Notifying the handwriting input module how to separate the currently stored handwriting input into one or more recognition units using a pinch gesture or an enlarged gesture, according to some embodiments. 2 shows an exemplary user interface for Notifying the handwriting input module how to separate the currently stored handwriting input into one or more recognition units using a pinch gesture or an enlarged gesture, according to some embodiments. 2 shows an exemplary user interface for Notifying the handwriting input module how to separate the currently stored handwriting input into one or more recognition units using a pinch gesture or an enlarged gesture, according to some embodiments. 2 shows an exemplary user interface for Notifying the handwriting input module how to separate the currently stored handwriting input into one or more recognition units using a pinch gesture or an enlarged gesture, according to some embodiments. 2 shows an exemplary user interface for Notifying the handwriting input module how to separate the currently stored handwriting input into one or more recognition units using a pinch gesture or an enlarged gesture, according to some embodiments. 2 shows an exemplary user interface for Notifying the handwriting input module how to separate the currently stored handwriting input into one or more recognition units using a pinch gesture or an enlarged gesture, according to some embodiments. 2 shows an exemplary user interface for Notifying the handwriting input module how to separate the currently stored handwriting input into one or more recognition units using a pinch gesture or an enlarged gesture, according to some embodiments. 2 shows an exemplary user interface for

Notifying the handwriting input module how to separate the currently stored handwriting input into one or more recognition units using a pinch gesture or an enlarged gesture, according to some embodiments. 3 is a flow diagram of an exemplary process for. Notifying the handwriting input module how to separate the currently stored handwriting input into one or more recognition units using a pinch gesture or an enlarged gesture, according to some embodiments. 3 is a flow diagram of an exemplary process for.

6 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input according to some embodiments. 6 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input according to some embodiments. 6 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input according to some embodiments. 6 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input according to some embodiments. 6 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input according to some embodiments. 6 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input according to some embodiments. 6 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input according to some embodiments. 6 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwriting input according to some embodiments.

6 is a flow diagram of an exemplary process for providing character-by-character deletion of a user's handwriting input, according to some embodiments. 6 is a flow diagram of an exemplary process for providing character-by-character deletion of a user's handwriting input, according to some embodiments.

6 illustrates an exemplary user interface for switching between a vertical write mode and a horizontal write mode, according to some embodiments. 6 illustrates an exemplary user interface for switching between a vertical write mode and a horizontal write mode, according to some embodiments. 6 illustrates an exemplary user interface for switching between a vertical write mode and a horizontal write mode, according to some embodiments. 6 illustrates an exemplary user interface for switching between a vertical write mode and a horizontal write mode, according to some embodiments. 6 illustrates an exemplary user interface for switching between a vertical write mode and a horizontal write mode, according to some embodiments. 6 illustrates an exemplary user interface for switching between a vertical write mode and a horizontal write mode, according to some embodiments.

6 is a flow diagram of an example process for switching between a vertical write mode and a horizontal write mode, according to some embodiments. 6 is a flow diagram of an example process for switching between a vertical write mode and a horizontal write mode, according to some embodiments. 6 is a flow diagram of an example process for switching between a vertical write mode and a horizontal write mode, according to some embodiments.

FIG. 4 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified in a user's handwriting input, according to some embodiments. FIG. 4 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified in a user's handwriting input, according to some embodiments. FIG. 4 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified in a user's handwriting input, according to some embodiments. FIG. 4 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified in a user's handwriting input, according to some embodiments. FIG. 4 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified in a user's handwriting input, according to some embodiments. FIG. 4 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified in a user's handwriting input, according to some embodiments. FIG. 4 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified in a user's handwriting input, according to some embodiments. FIG. 4 illustrates a user interface for providing a means for displaying and selectively deleting individual recognition units identified in a user's handwriting input, according to some embodiments.

2 is a flow diagram of an example process for providing a means for displaying and selectively deleting individual recognition units identified in a user's handwriting input, according to some embodiments. 2 is a flow diagram of an example process for providing a means for displaying and selectively deleting individual recognition units identified in a user's handwriting input, according to some embodiments.

Implicit confirmation input for entering a new handwritten input provided on top of an existing handwritten input within the handwritten input area and a recognition result displayed for that existing handwritten input, according to some embodiments 2 illustrates an exemplary user interface for use as Implicit confirmation input for entering a new handwritten input provided on top of an existing handwritten input within the handwritten input area and a recognition result displayed for that existing handwritten input, according to some embodiments 2 illustrates an exemplary user interface for use as Implicit confirmation input for entering a new handwritten input provided on top of an existing handwritten input within the handwritten input area and a recognition result displayed for that existing handwritten input, according to some embodiments 2 illustrates an exemplary user interface for use as Implicit confirmation input for entering a new handwritten input provided on top of an existing handwritten input within the handwritten input area and a recognition result displayed for that existing handwritten input, according to some embodiments 2 illustrates an exemplary user interface for use as Implicit confirmation input for entering a new handwritten input provided on top of an existing handwritten input within the handwritten input area and a recognition result displayed for that existing handwritten input, according to some embodiments 2 illustrates an exemplary user interface for use as Implicit confirmation input for entering a new handwritten input provided on top of an existing handwritten input within the handwritten input area and a recognition result displayed for that existing handwritten input, according to some embodiments 2 illustrates an exemplary user interface for use as Implicit confirmation input for entering a new handwritten input provided on top of an existing handwritten input within the handwritten input area and a recognition result displayed for that existing handwritten input, according to some embodiments 2 illustrates an exemplary user interface for use as Implicit confirmation input for entering a new handwritten input provided on top of an existing handwritten input within the handwritten input area and a recognition result displayed for that existing handwritten input, according to some embodiments 2 illustrates an exemplary user interface for use as Implicit confirmation input for entering a new handwritten input provided on top of an existing handwritten input within the handwritten input area and a recognition result displayed for that existing handwritten input, according to some embodiments 2 illustrates an exemplary user interface for use as Implicit confirmation input for entering a new handwritten input provided on top of an existing handwritten input within the handwritten input area and a recognition result displayed for that existing handwritten input, according to some embodiments 2 illustrates an exemplary user interface for use as Implicit confirmation input for entering a new handwritten input provided on top of an existing handwritten input within the handwritten input area and a recognition result displayed for that existing handwritten input, according to some embodiments 2 illustrates an exemplary user interface for use as Implicit confirmation input for entering a new handwritten input provided on top of an existing handwritten input within the handwritten input area and a recognition result displayed for that existing handwritten input, according to some embodiments 2 illustrates an exemplary user interface for use as

Implicit confirmation input for entering a new handwritten input provided on top of an existing handwritten input within the handwritten input area and a recognition result displayed for that existing handwritten input, according to some embodiments 2 is a flow diagram of an exemplary process for use as. Implicit confirmation input for entering a new handwritten input provided on top of an existing handwritten input within the handwritten input area and a recognition result displayed for that existing handwritten input, according to some embodiments 2 is a flow diagram of an exemplary process for use as.

Integrating temporally derived stroke distribution information into a handwritten recognition model based on spatially derived features without compromising the stroke order and stroke direction dependence of the handwritten recognition model according to some embodiments 2 is a flowchart of an exemplary process for doing so. Integrating temporally derived stroke distribution information into a handwritten recognition model based on spatially derived features without compromising the stroke order and stroke direction dependence of the handwritten recognition model according to some embodiments 2 is a flowchart of an exemplary process for doing so.

FIG. 2 is a block diagram illustrating a separate training of spatially and temporally derived features and subsequent integration of an exemplary handwriting recognition system, according to some embodiments.

FIG. 6 is a block diagram illustrating an exemplary method for calculating a stroke distribution profile for a character.

  Like reference numerals refer to corresponding parts throughout the drawings.

Many electronic devices have a graphical user interface with a soft keyboard for character input. On some electronic devices, the user may also be able to install or enable a handwriting input interface that allows the user to touch-sensitive display coupled to the device. Characters can be input via handwriting on the screen or touch sensitive surface. Conventional handwriting recognition input methods and user interfaces have several problems and disadvantages. For example,
In general, the functionality of conventional handwriting input is enabled for each language or for each script. Each additional input language requires the installation of a separate handwriting recognition model that consumes separate storage space and memory. The synergies brought about by combining handwriting recognition models for different languages are subtle, and mixed language or mixed script handwriting recognition has traditionally been extremely time consuming due to complex disambiguation processes It was a thing.
Furthermore, the conventional handwriting recognition system largely depends on language-specific or script-specific characteristics for character recognition. Recognition of mixed language handwriting input has low accuracy. Furthermore, the available combinations of recognized languages are very limited. Most systems required the user to manually specify the desired language specific handwriting recognizer before providing handwriting input in each non-default language or script.
Many existing real-time handwriting recognition models require temporal or sequence information at the stroke unit level, which means high variability in how characters can be written (eg typeface or personal Inaccurate recognition results when dealing with stroke variability in stroke shape, length, steepness, division, order, and direction). Some systems also provide users with strict spatial and temporal criteria (eg, with built-in assumptions for the size, arrangement, and timeframe of each character input) when providing handwritten input. You also need to comply. Any deviation from these standards caused inaccurate recognition results that were difficult to correct.
At present, most real-time handwriting input interfaces only allow the user to input a few characters at a time. Long phrases or sentence inputs are broken down into short segments and entered separately. This formal input not only imposes a cognitive burden on the user to maintain the flow of composition, but also makes it difficult for the user to correct or correct previously entered characters or phrases.

  The embodiments described below address these issues and related issues.

The following FIGS. 1-4 provide a description of exemplary devices. 5, 6, and 26, 27 illustrate an exemplary handwriting recognition and input system. 8A, 8B, 11A-11K, 13A-13E, 15A-15K, 17A-17F, 19A-19F, 21A-21H, and 23A-12L are handwritten. 2 illustrates an exemplary user interface for recognition and input. 7, 9A, 9B, 10A-10C, 12A, 12B, 14, 16, 16A, 16B, 18A, 18B, 20A-20C, 22A, 22B, 24A 24B and 25, train a handwriting recognition model, provide real-time handwriting recognition results, provide means to input and correct handwriting input, and input recognition results as text input. 3 is a flow diagram illustrating a method for enabling handwriting recognition and input including providing a means. Users in FIGS. 8A, 8B, 11A-11K, 13A-13E, 15A-15K, 17A-17F, 19A-19F, 21A-21H, and 23A-12L The interface is shown in FIGS. 7, 9A, 9B, 10A to 10C, 12A, 12B, 14, 16A, 16B, 18A, 18B, 20A to 20C, 22A, and 22B. 24A, 24B, and 25 are used to illustrate the process.
Exemplary device

  Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

  In this specification, terms such as first, second, etc. may be used to describe various elements, but it should also be understood that these elements should not be limited by these terms. Will be done. These terms are only used to distinguish one element from another. For example, a first contact can be referred to as a second contact, and, similarly, a second contact can be referred to as a first contact without departing from the scope of the present invention. The first contact and the second contact are both contacts, but they are not the same contact.

  The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the description of the present invention and the appended claims, the singular forms “a”, “an”, and “the” also refer to the plural unless the context clearly indicates otherwise. It is intended to include. As used herein, the term “and / or” refers to and includes any and all possible combinations of one or more of the associated listed items. Will be understood. The terms “includes”, “including”, “comprises”, and / or “comprising” as used herein are described features, integers, steps Specifying the presence of an action, element, and / or component, but excluding the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof It will be further understood that this is not the case.

  As used herein, the term “in the case of” refers to “when” or “when” or “depending on the determination of” or “to detect” depending on the context. Accordingly. Similarly, the phrase “when determined to be” or “when (described condition or event) is detected” means “when determined to be” or “to” depending on the context. Can be interpreted to mean "when a (description condition or event) is detected" or "in response to a detection (description condition or event)". .

  Embodiments of electronic devices, user interfaces for such devices, and related processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions such as PDA functions and / or music player functions. Exemplary embodiments of portable multifunction devices include Apple Inc. (But not limited to) iPhone (R), iPod Touch (R), and iPad (R) devices by (Cupertino, California). Other portable electronic devices such as laptop computers or tablet computers with touch-sensitive surfaces (eg, touch screen displays and / or touchpads) can also be used. It should also be understood that in some embodiments, the device is not a portable communications device but a desktop computer with a touch-sensitive surface (eg, a touch screen display and / or touchpad).

  The following discussion describes an electronic device that includes a display and a touch-sensitive surface. However, it should be understood that the electronic device may include one or more other physical user interface devices such as a physical keyboard, mouse, and / or joystick.

  This device is typically a drawing application, presentation application, word processing application, website creation application, disc authoring application, spreadsheet application, gaming application, telephone application, video conferencing application, email application, instant messaging application, Various applications such as one or more of a training support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application To support the application.

  Various applications that can run on the device can use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface, as well as corresponding information displayed on the device, can be adjusted and / or changed from application to application and / or within each application. In this manner, the device's common physical architecture (such as a touch-sensitive surface) can support a variety of applications using a user interface that is intuitive and transparent to the user.

  Attention is now directed to embodiments of portable devices comprising a touch sensitive display. FIG. 1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display 112, according to some embodiments. Touch-sensitive display 112 may be referred to as a “touch screen” for convenience, and is also known as or sometimes referred to as a touch-sensitive display system. The device 100 includes a memory 102 (which may include one or more computer readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuitry 108, an audio circuitry 110, Speaker 111, microphone 113, input / output (I / O) subsystem 106, other input or control device 116, and external port 124 may be included. Device 100 may include one or more light sensors 164. These components can communicate via one or more communication buses or signal lines 103.

  Device 100 is only one example of a portable multifunction device, and device 100 may have more or fewer components than those shown, and may include two or more components. It should be understood that they can be combined or have different configurations or arrangements of their components. The various components shown in FIG. 1 can be implemented as hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and / or application specific integrated circuits. .

  The memory 102 can include high-speed random access memory and can also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile semiconductor memory devices. Access to the memory 102 by other components of the device 100, such as the CPU 120 and peripheral device interface 118, can be controlled by the memory controller 122.

  Peripheral interface 118 can be used to couple the input and output peripherals of this device to CPU 120 and memory 102. One or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions related to device 100 and process data.

  In some embodiments, the peripheral interface 118, CPU 120, and memory controller 122 can be implemented on a single chip, such as the chip 104. In some other embodiments, they can be implemented on separate chips.

  The RF (radio frequency) circuit mechanism 108 transmits and receives RF signals, also called electromagnetic signals. The RF circuitry 108 converts electrical signals into electromagnetic signals / converts electromagnetic signals into electrical signals and communicates with communication networks and other communication devices via the electromagnetic signals.

  Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between the user and device 100. The audio circuit mechanism 110 receives audio data from the peripheral device interface 118, converts the audio data into an electric signal, and transmits the electric signal to the speaker 111. The speaker 111 converts the electric signal into a human audible sound wave. The audio circuitry 110 also receives an electrical signal converted from sound waves by the microphone 113. Audio circuitry 110 converts the electrical signal into audio data and transmits the audio data to peripheral device interface 118 for processing. Audio data can be retrieved from memory 102 and / or RF circuitry 108 by peripheral device 118 and / or transmitted to memory 102 and / or RF circuitry 108. In some embodiments, audio circuitry 110 also includes a headset jack (eg, 212 in FIG. 2).

  The I / O subsystem 106 couples input / output peripherals on the device 100, such as the touch screen 112 and other input control devices 116, to the peripheral interface 118. The I / O subsystem 106 may include a display controller 156, as well as one or more input controllers 160 for other inputs or control devices. One or more input controllers 160 receive / send electrical signals to / from other inputs or control device 116. Other input control devices 116 may include physical buttons (eg, push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, the input controller 160 can be coupled to any of pointer devices such as a keyboard, infrared port, USB port, and mouse (or may not be coupled to either). ). One or more buttons (eg, 208 in FIG. 2) may include up / down buttons for volume adjustment of speaker 111 and / or microphone 113. The one or more buttons may include a push button (eg, 206 in FIG. 2).

  Touch sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives and / or transmits electrical signals to and from the touch screen 112. Touch screen 112 displays visual output to the user. This visual output may include graphics, text, icons, video, and any combination thereof (collectively referred to as “graphics”). In some embodiments, some or all of these visual outputs may correspond to user interface objects.

  The touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on tactile and / or tactile contact. Touch screen 112 and display controller 156 (with any associated modules and / or instruction set in memory 102) detect and detect touches on touch screen 112 (and any movement or interruption of that touch). The contact is converted into an interaction with a user interface object (eg, one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In the exemplary embodiment, the point of contact between touch screen 112 and the user corresponds to the user's finger.

  The touch screen 112 may use LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, although other display technologies may be used in other embodiments. it can. The touch screen 112 and display controller 156 can detect a touch and any movement or interruption of the touch using any of a plurality of touch sensing technologies now known or later developed, These techniques include capacitive, resistive, infrared, and surface ultrasound techniques, as well as other proximity sensor arrays or other for determining one or more points of contact with the touch screen 112. Elements, but not limited to. In an exemplary embodiment, Apple Inc. Projected mutual capacitance sensing techniques are used, such as those found in iPhon®, iPod Touch®, and iPad® by (Cupertino, California).

  Touch screen 112 may have a video resolution in excess of 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user can contact touch screen 112 using any suitable object or accessory, such as a stylus, finger, and the like. In some embodiments, the user interface is designed to work primarily with finger-based touches and gestures, but these are based on stylus-based input due to the greater finger contact area on the touch screen. May be less accurate. In some embodiments, the device converts the finger-based rough input into a precise pointer / cursor position or command to perform the action desired by the user. Handwritten input can be provided on the touch screen 112 via finger-based or stylus-based contact location and movement. In some embodiments, touch screen 112 renders finger-based input or stylus-based input as immediate visual feedback for its current handwriting input and uses a writing instrument (eg, a pen) Provides visual effects of actual writing on (eg, a piece of paper).

  In some embodiments, in addition to the touch screen, device 100 may include a touch pad (not shown) for activating or deactivating certain functions. In some embodiments, the touchpad is the touch sensitive area of the device that, unlike a touch screen, does not display visual output. The touchpad can be a touch-sensitive surface that is separate from the touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

  The device 100 also includes a power system 162 for supplying power to various components. The power system 162 is a power management system, one or more power sources (eg, battery, alternating current (AC)), recharging system, power failure detection circuit, power converter or inverter, power status indicator (eg, light emitting diode (LED)). As well as any other component associated with the generation, management, and distribution of power within the portable device.

  Device 100 may also include one or more light sensors 164. FIG. 1 shows an optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106. The photosensor 164 may include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light from the environment projected through one or more lenses and converts the light into data representing an image. In conjunction with the imaging module 143 (also referred to as a camera module), the optical sensor 164 can capture still images or video.

  Device 100 may also include one or more proximity sensors 166. FIG. 1 shows a proximity sensor 166 coupled to the peripheral device interface 118. Alternatively, proximity sensor 166 can be coupled to input controller 160 in I / O subsystem 106. In some embodiments, when the multifunction device is placed near the user's ear (eg, when the user is making a phone call), the proximity sensor turns off the touch screen 112 and disables it. To do.

  Device 100 may also include one or more accelerometers 168. FIG. 1 shows an accelerometer 168 coupled to a peripheral interface 118. Alternatively, accelerometer 168 can be coupled to input controller 160 within I / O subsystem 106. In some embodiments, the information is displayed on the touch screen display in a vertical or horizontal display based on an analysis of data received from one or more accelerometers. The device 100 optionally includes a magnetometer (not shown) and GPS (or not shown) in addition to the accelerometer 168 to obtain information about the position and orientation (eg, vertical or horizontal display) of the device 100. GLONASS or other global navigation system) receiver (not shown).

  In some embodiments, the software components stored in memory 102 include operating system 126, communication module (or instruction set) 128, touch / motion module (or instruction set) 130, graphics module (or instruction set). ) 132, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. Furthermore, in some embodiments, the memory 102 stores a handwriting input module 157, as shown in FIGS. The handwriting input module 157 includes a handwriting recognition model and provides handwriting recognition and input functions to the user of the device 100 (or the device 300). Further details of the handwriting input module 157 are provided in connection with FIGS. 5-27 and their accompanying descriptions.

  Operating system 126 (for example, Darwin (registered trademark), RTXC (registered trademark), LINUX (registered trademark), UNIX (registered trademark), OS X (registered trademark), WINDOWS (registered trademark), VxWorks (registered trademark), etc. Embedded operating system) includes various software components and / or drivers and various hardware for controlling and managing common system tasks (eg, memory management, storage device control, power management, etc.). Facilitates communication between components and software components.

  The communication module 128 facilitates communication with other devices via one or more external ports 124, and various software for processing data received by the RF circuitry 108 and / or the external ports 124. Also includes components. External port 124 (eg, Universal Serial Bus (USB), FIREWIRE®, etc.) is coupled directly to other devices or indirectly via a network (eg, Internet, wireless LAN, etc.) Adapted to do.

  Contact / motion module 130 may detect contact with touch screen 112 (in conjunction with display controller 156) and contact with other touch sensitive devices (eg, a touchpad or physical click wheel). The touch / motion module 130 determines whether a touch has occurred (eg, detects a finger down event), determines whether a touch movement exists, and moves across the touch-sensitive surface. Tracking (e.g., detecting one or more events that drag the finger) and determining whether the contact has stopped (e.g., detecting a finger lift event or contact interruption). Various software components for performing various operations associated with contact detection, such as Contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point, represented by a series of contact data, can be a change in the speed (magnitude), velocity (magnitude and direction), and / or acceleration (magnitude and / or direction) of the contact point. ). These operations can be applied to a single contact (eg, a single finger contact) or multiple simultaneous contacts (eg, “multi-touch” / multiple finger contacts). In some embodiments, contact / motion module 130 and display controller 156 detect contact on the touchpad.

  The contact / motion module 130 can detect a gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns. Therefore, a gesture can be detected by detecting a specific contact pattern. For example, detecting a finger tap gesture may detect a finger down event, followed by a finger at the same position (or substantially the same position) as the finger down event (eg, at the icon location). Detecting an event that raises (lifts off). As another example, detecting a finger swipe gesture on the touch-sensitive surface detects a finger down event, followed by an event of dragging one or more fingers, followed by a finger Detecting an event that raises (lifts off).

  The touch / motion module 130 is displayed on the touch-sensitive display screen 112 within the handwriting input area of the handwriting input interface (or corresponding to the handwriting input area displayed on the display 340 in FIG. 3). Optionally used by the handwriting input module 157 to register the input of the handwritten stroke (within the area of the pad 355). In some embodiments, the first finger down event, the last finger up event, the location, path of movement, and intensity associated with any contact between them is recorded as a handwritten stroke. The Based on such information, handwritten strokes can be rendered on the display as feedback regarding the user input. Furthermore, one or more input images can be generated based on handwritten strokes registered by the contact / motion module 130.

  Graphics module 132 includes various known software components for rendering and displaying graphics on touch screen 112 or other display, including components for changing the brightness of displayed graphics. . As used herein, the term “graphic” includes any object that can be displayed to a user, including text, web pages, icons (user interface objects including softkeys). Etc.), digital images, videos, animations, and the like.

  In some embodiments, the graphics module 132 stores data representing the graphics used. Each graphic can be assigned a corresponding code. The graphic module 132 receives one or more codes that specify the graphic to be displayed, along with coordinate data and other graphic property data, as required, from an application, etc., and then outputs it to the display controller 156. Generate screen image data.

  The text input module 134 can be a component of the graphics module 132 and can include various applications (eg, contacts 137, email 140, IM 141, browser 147, and any other application that requires text input). ) To provide a soft keyboard for entering text. In some embodiments, the handwriting input module 157 is optionally invoked through the user interface of the text input module 134, eg, through keyboard selection affordance. In some embodiments, the same or similar keyboard selection affordance is also provided in the handwriting input interface to invoke the text input module 134.

  The GPS module 135 determines the location of the device and uses this information as photo / video metadata to the phone 138 for use in various applications (eg, for use in location-based dialing calls). 143 and to applications that provide location-based services such as weather widgets, local yellow page widgets, and map / navigation widgets).

  Application 136 may include the following modules (or instruction set), or a subset or superset thereof: contacts module 137 (sometimes called an address book or contact list), telephone module 138; video conferencing module 139; e-mail client module 140; instant messaging (IM) module 141; training support module 142; camera module 143 for still and / or video images; image management module 144; browser module 147; calendar module 148; -1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and user A widget module 149 that may include one or more of the other widgets acquired by the user, as well as a user created widget 149-6; a widget creation module 150 for creating a user created widget 149-6; a search module 151; A video and music player module 152; a memo module 153; a map module 154; and / or an online video module 155, which may comprise a video player module and a music player module.

  Examples of other applications 136 that can be stored in the memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA (registered trademark) compatible applications, encryption, digital works Rights management, voice recognition, and voice replication.

  In connection with the touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, the contact module 137 (e.g., contact module in the memory 102 or memory 370). 137 can be used to manage the address book or contact list (stored in the application internal state 192), including adding names to the address book and deleting names from the address book A telephone number, an email address, an actual address, or other information associated with a name, an image associated with a name, a classification and sorting of names, a telephone 138, a video conference 139, an email 140, Or IM 141 etc. To initiate and / or facilitate communications that include such to provide a telephone number or e-mail address.

  In conjunction with the RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, the telephone module 138 Enter a series of characters corresponding to a phone number, access one or more phone numbers in the address book 137, modify the phone numbers entered, dial each phone number, and conduct a conversation And can be used to disconnect or hang up when the conversation is complete. As described above, this wireless communication can use any of a plurality of communication standards, communication protocols, and communication technologies.

  RF circuit mechanism 108, audio circuit mechanism 110, speaker 111, microphone 113, touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact module 130, graphic module 132, handwriting input module 157, text input module 134 In connection with the contact list 137 and the telephone module 138, the video conference module 139 initiates, executes, and ends a video conference between the user and one or more other participants according to the user's instructions. Contains executable instructions for

  In conjunction with the RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, the email client module 140 is responsive to user instructions. Contains executable instructions for creating, sending, receiving and managing emails. In conjunction with the image management module 144, the email client module 140 makes it very easy to create and send emails with still or video images taken with the camera module 143.

  In conjunction with the RF circuitry 108, the touch screen 112, the display controller 156, the contact module 130, the graphics module 132, the handwriting input module 157, and the text input module 134, the instant messaging module 141 is a series of characters corresponding to an instant message. , Modify previously entered characters (for example, using a short message service (SMS) or multimedia message service (MMS) protocol for telephone-based instant messaging, or Internet-based instant messaging Send and receive instant messages (using XMPP, SIMPLE, or IMPS) For viewing instant messages, including executable instructions. In some embodiments, sent and / or received instant messages may be graphics, photos, audio files, video files, and / or other, as supported by MMS and / or Enhanced Messaging Service (EMS). Can include attachments. As used herein, “instant messaging” refers to phone-based messages (eg, messages sent using SMS or MMS) and Internet-based messages (eg, XMPP, SIMPLE, or IMPS) Message) to be transmitted.

  Training in conjunction with the RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, GPS module 135, map module 154, and music player module 146 The support module 142 is used to create a workout (eg, having time, distance, and / or calorie consumption goals), communicate with a workout sensor (sport device), receive workout sensor data, and monitor workouts Includes executable instructions for calibrating the sensors to be selected, selecting and playing music for training, and displaying, storing and transmitting training data.

  In conjunction with the touch screen 112, display controller 156, light sensor 164, light sensor controller 158, contact module 130, graphics module 132, and image management module 144, the camera module 143 may be a still image or a video (including a video stream). And executable instructions for modifying the still image or video characteristics or deleting the still image or video from the memory 102.

  In conjunction with the touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, and camera module 143, the image management module 144 arranges still images and / or video images. And include executable instructions to modify (eg, edit) or otherwise manipulate, label, delete, present (eg, in a digital slide show or album), and store.

  In conjunction with the RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, the browser module 147 is a web page or portions thereof, and Includes executable instructions for browsing the Internet according to user instructions, including searching, linking, receiving, and displaying attachments and other files linked to web pages.

  In conjunction with the RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, email client module 140, and browser module 147, the calendar module 148 Includes executable instructions for creating, displaying, modifying, and storing a calendar and data associated with the calendar (eg, calendar items, to-do lists, etc.) according to user instructions.

  In conjunction with the RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, and browser module 147, the widget module 149 is downloaded by the user. Mini-applications that can be used (eg, weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5), or created by the user A mini-application (e.g., user-created widget 149-6) that can. In some embodiments, the widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheet) file, and a JavaScript® file. In some embodiments, the widget includes an XML (Extensible Markup Language) file and a JavaScript file (eg, a Yahoo! ® widget).

  In conjunction with the RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, and browser module 147, the widget creation module 150 creates a widget. Can be used by the user to do (e.g., turn a user-specified portion of a web page into a widget).

  In conjunction with the touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, the search module 151 may include one or more search criteria (eg, according to user instructions). Executable instructions for searching text, music, sound, images, videos, and / or other files in memory 102 that match one or more user-specified search criteria).

  In conjunction with the touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, the video and music player module 152 can be MP3 or AAC. Executable instructions and video (eg, touch screen 112) that allow the user to download and play recorded music and other audio files stored in one or more file formats, such as files. Executable instructions for display, presentation or playback in other ways (on or on a display connected externally via external port 124). In some embodiments, the device 100 may include MP3 player functionality, such as iPod (a trademark of Apple Inc.).

  In connection with the touch screen 112, the display controller 156, the contact module 130, the graphics module 132, the handwriting input module 157, and the text input module 134, the memo module 153 creates memos, to-do lists, etc. according to user instructions. Contains executable instructions for management.

  In conjunction with the RF circuitry 108, the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132, the handwriting input module 157, the text input module 134, the GPS module 135, and the browser module 147, the map module 154 Receives maps and data associated with the maps (eg, driving directions, stores at or near specific locations and other points of interest, and other data based on location) according to user instructions Can be used to display, modify and store.

  Touch screen 112, display system controller 156, contact module 130, graphic module 132, audio circuit mechanism 110, speaker 111, RF circuit mechanism 108, handwriting input module 157, text input module 134, e-mail client module 140, and browser module 147 In connection with the online video module 155, the user can access, view, receive (eg, by streaming and / or download) certain online videos, (eg, on a touch screen or external port 124). (On an external display connected via the Internet), send an email with a link to that online video, and in other ways, Includes instructions that allow for managing online video in one or more file formats, such as H.264. In some embodiments, the instant messaging module 141 is used instead of the email client module 140 to send a link to a particular online video.

  Each of the modules and applications identified above may include one or more of the functions described above and methods described in this application (eg, computer-implemented methods and other information processing methods described herein). ) Corresponds to a set of executable instructions. These modules (ie, instruction sets) need not be implemented as separate software programs, procedures, or modules, and thus various embodiments combine various subsets of these modules or otherwise. It can be reconfigured by In some embodiments, the memory 102 may store a subset of the modules and data structures identified above. Furthermore, the memory 102 can store additional modules and data structures not described above.

  In some embodiments, the device 100 is a device in which operation of a predetermined set of functions on the device is performed exclusively via a touch screen and / or touch pad. By using a touch screen and / or touchpad as the primary input control device for the operation of the device 100, the number of physical input control devices (push buttons, dials, etc.) on the device 100 can be reduced.

  FIG. 2 illustrates a portable multifunction device 100 having a touch screen 112, according to some embodiments. The touch screen can display one or more graphics within a user interface (UI) 200. In this embodiment, as well as other embodiments described below, a user may, for example, have one or more fingers 202 (not drawn to scale in the figure) or one or more stylus 203 (see FIG. One or more of the graphics can be selected by performing a gesture on the graphics using (not drawn to exact scale). In some embodiments, selection of one or more graphics occurs when the user interrupts contact with one or more graphics. In some embodiments, this gesture is one or more taps, one or more (left-to-right, right-to-left, upward, and / or downward) swipes, and / or in contact with device 100 ( It may include finger rolling (right to left, left to right, upward and / or downward). In some embodiments, accidental contact with a graphic may not select that graphic. For example, if the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over the application icon cannot select the corresponding application.

  Device 100 may also include one or more physical buttons, such as “Home” or menu button 204. As described above, the menu button 204 can be used to navigate to any application 136 in the set of applications that can be executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 112.

  In one embodiment, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for turning the device on and off, and locking the device, a volume control button 208, a subscriber identity module (SIM) card slot. 210, a headset jack 212, and an external port 124 for docking / charging. Push button 206 turns the power on and off on the device by holding the button down and holding it down for a predefined time interval, before the predefined time interval elapses. Can be used to lock the device by releasing the button and / or unlock the device or initiate the unlocking process. In an alternative embodiment, the device 100 can also accept verbal input for activation or deactivation of some functions via the microphone 113.

  FIG. 3 is a block diagram of an exemplary multifunction device comprising a display and a touch-sensitive surface, according to some embodiments. Device 300 need not be portable. In some embodiments, the device 300 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, telephone device, or control device ( For example, a home or business controller). The device 300 typically has one or more processing units (CPUs) 310, one or more networks or other communication interfaces 360, memory 370, and one or more for interconnecting these components. A communication bus 320 is included. Communication bus 320 may include circuitry (sometimes referred to as a chipset) that interconnects and controls communication between system components. The device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. The I / O interface 330 may also include a keyboard and / or mouse (or other pointing device) 350 and a touch pad 355. Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access semiconductor memory devices, and also includes one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or others. Non-volatile memory, such as non-volatile semiconductor storage devices. Memory 370 may optionally include one or more storage devices that are located remotely from CPU 310. In some embodiments, the memory 370 is similar to programs, modules, and data structures stored in the memory 102 of the portable multifunction device 100 (FIG. 1), or modules, or data structures thereof. Remember the subset. Furthermore, the memory 370 can store additional programs, modules, and data structures that are not present in the memory 102 of the portable multifunction device 100. For example, the memory 370 of the device 300 can store a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disk authoring module 388, and / or a spreadsheet module 390, one of which Thus, the memory 102 of the portable multifunction device 100 (FIG. 1) may not store these modules.

  Each of the elements of FIG. 3 identified above can be stored in one or more of the memory devices described above. Each of the modules identified above corresponds to an instruction set that performs the functions described above. The modules or programs identified above (ie, the instruction set) need not be implemented as separate software programs, procedures, or modules, and thus various embodiments represent different subsets of these modules. They can be combined or reconfigured in other ways. In some embodiments, memory 370 may store a subset of the modules and data structures identified above. Furthermore, the memory 370 can store additional modules and data structures not described above.

  FIG. 4 illustrates an example on a device (eg, device 300 of FIG. 3) that includes a touch-sensitive surface 451 (eg, tablet or touchpad 355 of FIG. 3) separate from display 450 (eg, touch screen display 112). A typical user interface is shown. Many of the following examples are described with reference to input on the touch screen display 112 (when the touch-sensitive surface and display are combined), but in some embodiments, the device is As shown in FIG. 4, an input on a touch-sensitive surface separate from the display is detected. In some embodiments, this touch-sensitive surface (eg, 451 in FIG. 4) is a major axis (eg, in FIG. 4) corresponding to a major axis (eg, 453 in FIG. 4) on the display (eg, 450). 452). According to these embodiments, the device is configured with touch-sensitive surfaces 451 at locations corresponding to respective locations on the display (eg, in FIG. 4, 460 corresponds to 468 and 462 corresponds to 470). Are detected (eg, 460 and 462 in FIG. 4). In this manner, if the touch-sensitive surface is separate from the display, user input (eg, contacts 460 and 462, and theirs) detected by the device on the touch-sensitive surface (eg, 451 in FIG. 4) Move) is used by the device to operate the user interface on the display of the multifunction device (eg, 450 in FIG. 4). It should be understood that similar methods can be used for other user interfaces described herein.

  Attention is now directed to embodiments of handwriting input methods and user interfaces (“UI”) that can be implemented on a multifunction device (eg, device 100).

  FIG. 5 illustrates an interaction with an I / O interface module 500 (eg, I / O interface 330 in FIG. 3 or I / O subsystem 106 in FIG. 1) on a device, according to some implementations. FIG. 6 is a block diagram illustrating an example handwriting input module 157 that provides handwriting input capability to As shown in FIG. 5, the handwriting input module 157 includes an input processing module 502, a handwriting recognition module 504, and a result generation module 506. In some embodiments, the input processing module 502 includes a split module 508 and a normalization module 510. In some embodiments, the result generation module 506 includes an radical clustering module 512 and one or more language models 514.

  In some embodiments, the input processing module 502 communicates with the I / O interface module 500 (eg, the I / O interface 330 in FIG. 3 or the I / O subsystem 106 in FIG. 1) for user Receive handwritten input from. The handwriting is input via any suitable means, such as touch-sensitive display system 112 in FIG. 1 and / or touchpad 355 in FIG. The handwriting input includes data representing each stroke provided by the user in a predefined handwriting input area within the handwriting input UI. In some embodiments, the data representing each stroke of handwriting input is the beginning and end of continuous contact within the handwriting input area (eg, contact between the user's finger or stylus and the touch-sensitive surface of the device). Includes data such as location, intensity profile, and path of motion. In some embodiments, the I / O interface module 500 passes an array of handwritten strokes 516 with associated temporal and spatial information to the input processing module 502 in real time. At the same time, the I / O interface module also provides real-time rendering 518 of those handwritten strokes in the handwriting input area of the handwriting input user interface as visual feedback for this user input.

  In some embodiments, as data representing each handwritten stroke is received by the input processing module 502, temporal and spatial information associated with a plurality of successive strokes is also recorded. For example, the data may optionally be relative to their strokes along the shape, size, spatial saturation, and writing direction of the entire handwriting input of individual strokes having respective stroke array numbers. Contains a stack, showing the spatial location etc. In some embodiments, the input processing module 502 renders the received strokes on the device display 518 (eg, the display 340 in FIG. 3 or the touch-sensitive display 112 in FIG. 1) I An instruction is returned to the / O interface module 500. In some embodiments, the received stroke rendering is animated to mimic the actual progression of writing on a writing surface (eg, a piece of paper) using a writing instrument (eg, a pen). Visual effects are provided. In some embodiments, the user can optionally specify a nib style, color, texture, etc. of the stroke to be rendered.

In some embodiments, the input processing module 502 processes the strokes currently stored in the handwriting input area and assigns those strokes to one or more recognition units. In some embodiments, each recognition unit corresponds to a particular character that is recognized by the handwriting recognition model 504. In some embodiments, each recognition unit corresponds to a particular output character or radical recognized by the handwriting recognition model 504. The radical is a reproducible component found in multiple compound alphabetic characters. A compound table word character may include two or more radicals arranged according to a general layout (eg, left and right layout, upper and lower layout, etc.). In one embodiment, a single Chinese character “
"Means two radicals, the left radical"
"And right radical"
Is built using.

In some embodiments, the input processing module 502 relies on a splitting module to assign or separate the currently stored handwritten strokes to one or more recognition units. For example, the handwritten character “
, The split module 508 optionally selects a stroke that is clustered to the left of the handwriting input as a single (ie, left radical “
”To the recognition unit and the strokes clustered on the right side of the handwriting input are separated by another (ie right radical“
Assigned to the recognition unit. Alternatively, the split module 508 also converts all of these strokes into a single (ie, the character “
Can also be assigned to recognition units.

  In some embodiments, the split module 508 splits the currently stored handwriting input (eg, one or more handwritten strokes) into groups of recognition units in several different ways to split the bundle 520. To produce. For example, it is assumed that a total of nine strokes have been accumulated in the handwriting input area so far. According to the first split chain of split bundle 520, strokes 1, 2, 3 are grouped into first recognition units 522, and strokes 4, 5, 6 are grouped into second recognition units 526. The All of the strokes 1-9 are grouped into one recognition unit 526 according to the second split chain of the split bundle 520.

  In some embodiments, each split chain is given a split score to evaluate the likelihood that a particular split chain is the correct split for the current handwritten input. In some embodiments, factors that are optionally used to calculate the split score for each split chain include the absolute and / or relative size of the stroke, various directions (eg, x-direction, relative and / or absolute span of stroke in y direction, z direction), average and / or variation of stroke saturation level, absolute and / or relative distance to adjacent stroke, absolute and / or stroke Or the relative location, the order or arrangement of strokes entered, the duration of each stroke, the average and / or variation of the speed (or slowness) at which each stroke was entered, the strength of each stroke along the length of the stroke A profile etc. are mentioned. In some embodiments, one or more functions or transformations are optionally applied to one or more of these factors to generate split scores for the various split chains in split bundle 520. The

  In some embodiments, after the split module 508 splits the current handwriting input 516 received from the user, the split module 508 passes the split bundle 520 to the normalization module 510. In some embodiments, the normalization module 510 generates an input image (eg, input image 528) for each recognition unit (eg, recognition units 522, 524, and 526) specified in the split bundle 520. In some embodiments, the normalization module may provide required or desired normalization (eg, stretching, trimming) to the input image so that the handwriting recognition model 504 can be provided with the input image as input. , Downsampling, or upsampling). In some embodiments, each input image 528 includes a stroke assigned to a respective recognition unit and corresponds to a single character or radical recognized by the handwriting recognition module 504.

  In some embodiments, the input image generated by the input processing module 502 does not include any temporal information associated with individual strokes, and spatial information (eg, pixel location and density within the input image). Only information) is stored in the input image. A handwritten recognition model, trained on the spatial information of a purely training written sample, is capable of handwriting recognition based solely on spatial information. As a result, the handwriting recognition model is stroke order and stroke direction independent, and during training, the stroke order and stroke for all characters (ie, all output classes) in the vocabulary of the handwriting recognition model. There is no exhaustive list of all possible permutations of directions. Indeed, in some embodiments, the handwriting recognition module 502 does not distinguish between pixels belonging to one stroke and pixels belonging to another stroke in the input image.

  As will be described in more detail later (eg, in connection with FIGS. 25A-27), in some embodiments, some temporally derived stroke distribution information is purely spatial handwriting. Re-introduced into the recognition model, the recognition accuracy is improved without impairing the stroke order and stroke direction independence of the recognition model.

  In some embodiments, for one recognition unit, the input image generated by the input processing module 502 does not overlap with any other recognition unit input image in the same split chain. In some embodiments, input images generated for different recognition units may have some overlap. In some embodiments, some overlap between input images includes handwritten input written in cursive typeface and / or continued characters (eg, one stroke connects two adjacent characters). Regarding recognition of handwritten input, it is allowed.

  In some embodiments, some normalization is performed prior to partitioning. In some embodiments, the functionality of split module 508 and normalization module 510 may be performed by the same module or two or more other modules.

  In some embodiments, when an input image 528 for each recognition unit is provided as input to the handwriting recognition model 504, the handwriting recognition model 504 may have a recognition unit whose repertoire or vocabulary (ie, A list of all characters and radicals that are recognizable by the handwriting recognition module 504, each of which is an output character of various likelihoods. As will be described in more detail later, the handwriting recognition model 504 is trained to recognize multiple characters in multiple scripts (eg, at least three non-overlapping scripts encoded by the Unicode standard). Has been. Examples of non-overlapping scripts include Latin scripts, Chinese characters, Arabic characters, Persian scripts, Cyrillic scripts, and artificial characters such as pictograms. In some embodiments, the handwriting recognition model 504 generates one or more output characters for each input image (ie, for each recognition unit) and each output character based on a confidence level associated with that character recognition. Assign a recognition score for each.

  In some embodiments, the handwriting recognition model 504 generates a candidate bundle 530 according to the split bundle 520 and corresponds to each arc in the split chain in the split bundle 520 (eg, corresponding to each recognition unit 522, 524, 526). Are expanded within the candidate bundle 530 into one or more candidate arcs (eg, arcs 532, 534, 536, 538, 540, each corresponding to a respective output character). Each candidate chain in the candidate bundle 530 is scored according to the respective split score of the split chain that is the basis of the candidate chain and the recognition score associated with the output character in the candidate chain.

  In some embodiments, after the handwriting recognition model 504 generates output characters from the input image 528 of the recognition unit, the candidate bundle 530 is passed to the result generation module 506 for the currently stored handwriting input 516. One or more recognition results are generated.

  In some embodiments, the result generation module 506 utilizes the radical clustering module 512 to combine one or more radicals in the candidate chain into a composite character. In some embodiments, the result generation module 506 uses one or more language models 514 so that the character chains in the candidate bundle 530 are in an appropriate sequence in the particular language represented by that language model. It is determined whether or not there is. In some embodiments, the result generation module 506 generates a correction candidate bundle 542 by deleting particular arcs in the candidate bundle 530 or combining two or more arcs.

  In some embodiments, the result generation module 506 generates an integrated recognition score for each character array (eg, character arrays 544 and 546) that is still maintained in the correction candidate bundle 542, the radical clustering module 512 and the language. Generate based on the recognition scores of the output characters within those character sequences modified (eg, increased or decreased) by the model 514. In some embodiments, the result generation module 506 ranks the various character sequences maintained in the correction candidate bundle 542 based on their integrated recognition scores.

In some embodiments, the result generation module 506 sends the highest rank character sequence to the I / O interface module 500 as a ranking recognition result 548 for display to the user. In some embodiments, the I / O interface module 500 may receive the received recognition result 548 (eg, “
"as well as"
”) In the candidate display area of the handwriting input interface. In some embodiments, the I / O interface module may have multiple recognition results (eg, “
"as well as"
”) To the user, allowing the user to select a recognition result and enter it as a text input for the associated application. In some embodiments, the I / O interface module may receive the highest rank recognition result (eg, “
)) Is automatically input in response to other input or instruction of the user confirmation of the recognition result. Effective automatic input of the highest ranking results can improve the efficiency of the input interface and provide a better user experience.

  In some embodiments, the result generation module 506 uses other factors to modify the candidate chain's integrated recognition score. For example, in some embodiments, the result generation module 506 optionally maintains a log of the most frequently used characters for a particular user or multiple users. The result generation module 506, if a particular candidate character or character sequence is found in a list of the most frequently used characters or character sequences, the integrated recognition score for that particular candidate character or character sequence is Optionally lift.

  In some embodiments, the handwriting input module 157 provides real-time updates regarding recognition results displayed to the user. For example, in some embodiments, for each additional stroke by the user, the input processing module 502 optionally subdivides the currently stored handwriting input and provides a split bundle provided to the handwriting recognition model 504. And correct the input image. Similarly, handwriting recognition model 504 optionally corrects the candidate bundle provided to result generation module 506. As a result, the result generation module 506 optionally updates the recognition result presented to the user. As used herein, real-time handwriting recognition refers to handwriting recognition in which handwriting recognition results are presented to the user immediately or within a short period of time (eg, within tens of milliseconds to seconds). Real-time handwriting recognition is not performed at once after the current user session, but from recording images saved for later retrieval, recognition starts immediately and is performed substantially simultaneously with receipt of handwriting input In that it differs from offline recognition (eg, in an offline optical character recognition (OCR) application). Furthermore, since off-line character recognition is performed without using any temporal information regarding individual strokes and stroke arrangements, segmentation is performed without benefiting from such information. Further disambiguation between similar looking candidate characters also does not benefit from such temporal information.

  In some embodiments, the handwriting recognition model 504 is implemented as a convolutional neural network (CNN). FIG. 6 shows an exemplary convolutional neural network 602 that is trained on a multiscript training corpus 604 that includes written samples for characters in multiple non-overlapping scripts.

  As shown in FIG. 6, the convolutional neural network 602 includes an input surface 606 and an output surface 608. Between the input surface 606 and the output surface 608 are a plurality of convolution layers 610 (eg, including an initial convolution layer 610a, zero or more intermediate convolution layers (not shown), and a final convolution layer 610n). Each convolution layer 610 is followed by a respective sub-sampling layer 612 (eg, an initial sub-sampling layer 612a, zero or more intermediate sub-sampling layers (not shown), and a final sub-sampling layer 612n). After these convolutional layers and subsampling layers, a hidden layer 614 exists just before the output surface 608. The hidden layer 614 is the final layer before the output surface 608. In some embodiments, a kernel layer 616 (eg, including an initial kernel layer 616a, zero or more intermediate kernel layers (not shown), and a final kernel layer 612n) is inserted before each convolution layer 610. Thus, the calculation efficiency is improved.

  As shown in FIG. 6, the input surface 606 receives an input image 614 of a handwritten recognition unit (for example, a handwritten character or a radical), and the output surface 608 has an input class corresponding to the output class ( For example, a set of probabilities is output that indicates the likelihood of belonging to a particular character in the set of output characters configured to be recognized by the neural network. The entire neural network output class (or set of neural network output characters) is also referred to as the repertoire or vocabulary of the handwriting recognition model. The convolutional neural network described herein can be trained to have a repertoire of tens of thousands of characters.

  As the input image 614 is processed through the various layers of the neural network, the various spatial features embedded in the input image 614 are extracted by the convolution layer 610. Each convolution layer 610 is also referred to as a set of feature maps, and serves as a filter to find specific features in the input image 614 to distinguish images corresponding to different characters. The sub-sampling layer 612 ensures that features are captured from the input image 614 on a gradually increasing scale. In some embodiments, the subsampling layer 612 is implemented using a maximum value pooling technique. This maximum pooling layer causes position invariance over a larger local area, and the output image of the preceding convolutional layer is downsampled by Kx and Ky times along each direction (Kx and Ky). Is the size of the maximum pooling rectangle). Maximum value pooling results in a faster convergence rate and improved generalization ability by selecting the top invariant features. In some embodiments, subsampling is achieved using other methods.

  In some embodiments, there is a fully connected or hidden layer 614 after the final set of convolutional layers 610n and subsampling 612n and before the output plane 608. Fully connected hidden layer 614 is a multi-layer perceptron that fully connects the nodes in final subsampling layer 612n and the nodes in output plane 608. Hidden layer 614 obtains the output image received from the previous layer and reaches one of the output characters in output layer 608 through logistic regression.

  During training of the convolutional neural network 602, the features in the convolutional layer 610, and the respective weights associated with those features, and the weights associated with the parameters in the hidden layer 614, are known output classes in the training corpus 604. Is adjusted so that classification errors are minimized. Once the convolutional neural network 602 has been trained and an optimal set of parameters and associated weights has been established for the various layers in the network, the convolutional neural network 602 is used to receive real-time handwritten input received from the user. New written samples 618 that are not part of the training corpus 604, such as input images generated based on

  As described herein, the convolutional neural network of the handwriting input interface is trained using a multiscript training corpus to enable multiscript or mixed script handwriting recognition. In some embodiments, the convolutional neural network is trained to recognize a wide repertoire of more than 30,000 to 60,000 characters (eg, all characters are encoded by the Unicode standard). Most modern handwriting recognition systems are based on a stroke order dependent hidden Markov method (HMM). Furthermore, most existing handwriting recognition models are language-specific, from a small repertoire of dozens of characters (eg, English alphabet, Greek alphabet, all 10 numbers, etc.) Contains up to several thousand characters (eg, the most commonly used Chinese character set). Therefore, the universal recognizer described herein can handle characters that are orders of magnitude more than most existing systems.

  Some conventional handwriting systems may include several individually trained handwriting recognition models, each adapted for a specific language or a small set of characters. Written samples are propagated through various recognition models until they can be classified. For example, handwritten samples can be provided in a series of linked language-specific or script-specific character recognition models, and if the handwritten samples cannot be classified reliably by the initial recognition model, This subsequent recognition model attempts to classify the handwritten sample within its own repertoire. This classification approach is time consuming and the memory requirements increase rapidly with each additional recognition model that needs to be adopted.

  Other state-of-the-art models require the user to specify a preferred language and use the selected handwriting recognition model to classify the current input. Such an implementation is cumbersome to use and not only consumes significant memory, but also cannot be used to recognize mixed language input. It is not practical that the user needs to switch language settings in the middle of inputting a mixed language or mixed script.

  The multi-script or general-purpose recognizer described herein addresses at least some of the above problems associated with conventional recognition systems. FIG. 7 uses a large multi-script training corpus to train a handwriting recognition module (e.g., a convolutional neural network) that can be used for real-time multi-language and multi-script handwriting with respect to user handwriting input. 7 is a flow diagram of an example process 700 for subsequent use to provide recognition.

  In some embodiments, training of the handwriting recognition model is performed on the server device, and then the trained handwriting recognition model is provided to the user device. The handwriting recognition model optionally performs real-time handwriting recognition locally on the user device without requiring further assistance from the server. In some embodiments, both training and recognition are provided on the same device. For example, the server device can receive the user's handwriting input from the user device, execute handwriting recognition, and transmit the recognition result to the user device in real time.

  In the exemplary process 700, a device having one or more processors and memory, wherein the device is multiscript handwriting based on spatially derived features (eg, stroke order independent features) of the multiscript training corpus. Train the recognition model (702). In some embodiments, the spatially derived features of the multiscript training corpus are stroke order independent and stroke direction independent (704). In some embodiments, training of the multi-script handwriting recognition model is independent of temporal information associated with each stroke in the handwritten sample (706). Specifically, the handwritten sample images are normalized to a predetermined size, and the images do not contain any information about the order in which individual strokes are entered to form the image. Furthermore, the images also do not contain any information about the direction in which individual strokes are entered to form the image. In fact, during training, features are extracted from these handwritten images regardless of how the images are formed in time by individual strokes. Therefore, no time information associated with individual strokes is required during recognition. As a result, the recognition robustly provides consistent recognition results despite delayed out-of-order strokes and arbitrary stroke directions within handwriting input.

  In some embodiments, the multi-script training corpus includes handwritten samples that correspond to at least three non-overlapping script characters. As shown in FIG. 6, the multiscript training corpus includes handwritten samples collected from many users. Each handwritten sample corresponds to one character of each script represented in the handwriting recognition model. In order to properly train the handwriting recognition model, the training corpus includes a number of written samples for each character of the script represented in the handwriting recognition model.

  In some embodiments, the at least three non-overlapping scripts include Chinese characters, pictograms, and Latin scripts (708). In some embodiments, the multi-script handwriting recognition model has at least 30,000 output classes that represent 30,000 characters across at least three non-overlapping scripts (710).

  In some embodiments, the multi-script training corpus includes all or substantially all Chinese characters (eg, all CJK (Chinese-Japanese-Korean) integrated kanji characters) encoded in the Unicode standard. (Part) includes each written sample for each character. The Unicode standard defines about 74,000 CJK integrated kanji in total. The basic block of CJK integrated kanji (4E00-9FFF) contains 20,941 basic Chinese characters used in Chinese and Japanese, Korean, and Vietnamese. In some embodiments, the multi-script training corpus includes written samples for all characters in a basic block of CJK integrated kanji. In some embodiments, the multi-script training corpus further includes written samples for CJK radicals that can be used to structurally construct one or more composite Chinese characters. In some embodiments, the multi-script training corpus further includes written samples for less frequently used Chinese characters, such as Chinese characters encoded in one or more of the CJK integrated Kanji extensions.

  In some embodiments, the multi-script training corpus further includes a respective written sample for each character of all characters in the Latin script encoded by the Unicode code. Characters in basic Latin scripts include Latin capital letters and Latin small letters, as well as basic symbols and numbers commonly used on standard Latin keyboards. In some embodiments, the multi-script training corpus further includes characters in the extended Latin script (eg, various emphasis forms of basic Latin characters).

In some embodiments, the multi-script training corpus includes written samples that correspond to each character of the artificial script that is not associated with any natural human language. For example, in some embodiments, a set of pictograms is optionally defined in the pictogram script, and written samples corresponding to each of those pictograms are included in the multiscript training corpus. For example, the hand-drawn heart-shaped symbol is the pictogram “
Is a handwritten sample. Similarly, a hand-drawn smile (e.g., two dots above the upward-facing arc)
Is a handwritten sample. Other pictograms include various emotions (eg, joy, sadness, anger, embarrassment, surprise, laughter, crying, irritation, etc.), various objects and characters (eg, cats, dogs, rabbits, hearts, fruits, eyes, Lips, gifts, flowers, candles, moon, stars, etc.) and icons indicating various actions (eg handshake, kiss, run, dance, jump, sleep, eat, meet, love, like, vote, etc.) Categories. In some embodiments, the stroke in the handwritten sample corresponding to the pictogram is a simplified and / or stylized line of the actual line that forms the corresponding pictogram. In some embodiments, each device or application can use a different design for the same pictogram. For example, a smiley emoticon presented to a female user is similar to a smiley emoticon presented to a male user, even if the handwriting input received from those two users is substantially the same, Can be different.

  In some embodiments, the multi-script training corpus also includes Greek scripts (eg, including Greek letters and symbols), Cyrillic scripts, Hebrew scripts, and one or more other encoded according to the Unicode standard. Also includes writing samples about characters in other scripts, such as In some embodiments, the at least three non-overlapping scripts included in the multi-script training corpus include Chinese characters, pictograms, and characters in Latin scripts. Chinese characters, pictograms, and characters in Latin scripts are inherently non-overlapping scripts. Many other scripts may overlap each other for at least some characters. For example, some letters (eg, A, Z) in a Latin script can be found in many other scripts (eg, Greek and Cyrillic). In some embodiments, the multi-script training corpus includes Chinese characters, Arabic scripts, and Latin scripts. In some embodiments, the multi-script training corpus includes other combinations of duplicate scripts and / or non-duplicate scripts. In some embodiments, the multiscript training corpus includes written samples for all characters encoded by the Unicode standard.

  As shown in FIG. 7, in some embodiments, in order to train a multi-script handwriting recognition model, the device can be a single convolutional neural network with a single input surface and a single output surface. A handwritten sample of the multiscript training corpus is provided (712). The device spatially derived features of those handwritten samples (eg, stroke order independent features) to distinguish at least three non-overlapping script characters represented in the multiscript training corpus, And their respective weights for those spatially derived features are determined 714 using a convolutional neural network. This multi-script handwriting recognition model has a single input surface and a single output surface, in that a single handwriting recognition model is trained using all samples in the multi-script training corpus. It is different from the conventional multi-script handwriting recognition model. A single convolutional neural network is a separate subnetwork, each handling a small subset of the training corpus (eg, a subnetwork, each trained on a specific script or character used within a specific language). Independently trained to identify all characters represented in the multi-script training corpus. In addition, this single convolutional neural network includes multiple non-overlapping scripts rather than a small number of overlapping script characters (eg, having overlapping characters A, B, E, Z, etc.) such as Latin scripts and Greek scripts. Trained to identify multiple characters across.

  In some embodiments, the device provides real-time handwriting recognition for a user's handwriting input using a multiscript handwriting recognition model that is trained on spatially derived features of a multiscript training corpus ( 716). In some embodiments, providing real-time handwriting recognition for a user's handwriting input continues to provide recognition output for the user's handwriting input when the user continues to provide additions and corrections for handwriting input. Corrective correction. In some embodiments, providing real-time handwriting recognition for a user's handwriting input further includes providing 718 a multi-script handwriting recognition model to the user device, the user device receiving handwriting input from the user. Based on the received multi-script handwriting recognition model, handwriting recognition is locally executed for handwriting input.

  In some embodiments, the device provides a multi-script handwriting recognition model to a plurality of devices that do not have duplicates in their respective input languages, and the multi-script handwriting recognition model is associated with each of those user devices. With respect to handwriting recognition in different languages, it is used in each of those devices. For example, if a multi-script handwriting recognition model is trained to recognize characters in many different scripts and languages, the same handwriting recognition model can be used worldwide to identify any of those input languages. Can provide handwriting input for The first device for users who only want to input in English and Hebrew uses the same handwriting recognition model as the second device for users who only want to input in Chinese and pictograms. Functionality can be provided. A first device user implements an English handwriting input keyboard (eg, implemented with an English-specific handwriting recognition model) and a separate Hebrew handwriting input keyboard (eg, implemented with a Hebrew-specific handwriting recognition model) Once the same generic multi-script handwriting recognition model is installed on the first device, instead of requiring separate installation), both the English and Hebrew handwriting input functions, and both Can be used to provide mixed input in any language. Furthermore, the second user implements a Chinese handwriting input keyboard (for example, implemented with a Chinese-specific handwriting recognition model) and a separate pictogram handwriting input keyboard (for example, a pictogram handwriting recognition model). Once the same generic multi-script handwriting recognition model is installed on the second device, instead of needing to install the handwritten input function for both Chinese and pictograms, as well as both scripts Can be used to provide a mixed input. By using the same multi-script handwriting model to handle a wide repertoire across multiple scripts (eg, a substantial portion or all of the characters encoded in nearly 100 different scripts), device suppliers and users Thus, the usefulness of the recognition unit is improved without imposing a substantial burden.

  Training multi-script handwriting recognition models using large multi-script training corpora is different from traditional HMM-based handwriting recognition systems and does not rely on temporal information associated with individual strokes of characters . Furthermore, the resource and memory requirements for this multi-script recognition system do not increase in proportion to the increase in symbols and languages that are the scope of the multi-script recognition system. For example, in a traditional handwriting system, increasing the number of languages means adding another model trained independently, and memory requirements to adapt to the increasing ability of the handwriting recognition system, It will be at least doubled. In contrast, when a multiscript model is trained by a multiscript training corpus, increasing the language coverage requires retraining the handwriting recognition model using additional handwriting samples. Increase the size of the output surface, but the amount of increase is very modest. Multi-script training corpus includes handwritten samples corresponding to n different languages, multi-script handwriting recognition model occupies memory of size m and increases language coverage to N (N> n) languages The device retrains the multi-script handwriting recognition model based on the spatially derived features of the second multi-script training corpus, the second multi-script training corpus includes N different languages. And a second handwritten sample corresponding to. When the N / n change is 1-100, the M / m change remains substantially constant within the range of 1-2. After the multi-script handwriting recognition model is retrained, the device can use the retrained multi-script handwriting recognition model to provide real-time handwriting recognition for the user's handwriting input.

  8A and 8B illustrate an exemplary user interface for providing real-time multiscript handwriting recognition and input on a portable user device (eg, device 100). 8A and 8B, a handwriting input interface 802 is displayed on a touch-sensitive display screen (eg, touch screen 112) of the user device. The handwriting input interface 802 includes a handwriting input area 804, a candidate display area 806, and a text input area 808. In some embodiments, the handwriting input interface 802 can further include a plurality of control elements that can each be invoked to cause the handwriting input interface to perform a predefined function. As shown in FIG. 8A, the handwriting input interface includes a delete button, a space button, a carriage return / enter button, and a keyboard switching button. Other control elements are possible and can optionally be provided within the handwriting input interface to suit different applications that utilize the handwriting input interface 802. The layout of the various components of the handwriting input interface 802 is merely exemplary and can vary for different devices and different applications.

In some embodiments, handwriting input area 804 is a touch sensitive area for receiving handwriting input from a user. The persistent touch on the touch screen in the handwriting input area 804 and its associated movement path are registered as handwriting strokes. In some embodiments, handwritten strokes registered by the device are visually rendered in the handwriting input area 804 at the same location traced by its persistent contact. As shown in FIG. 8A, the user enters some handwritten Chinese characters (eg, “
"), A number of handwritten strokes, including some handwritten English characters (eg," Happy "), and hand-drawn pictograms (eg, smiles). These handwritten characters are distributed in a plurality of lines (for example, two lines) in the handwriting input area 804.

  In some embodiments, the candidate display area 806 displays one or more recognition results (eg, 810 and 812) for the handwriting input currently stored in the handwriting input area 804. Generally, the recognition result of the highest rank (for example, 810) is displayed at the first position in the candidate display area. As shown in FIG. 8A, the handwriting recognition model described herein is capable of recognizing multiple non-overlapping script characters, including Chinese characters, Latin scripts, and pictograms. The recognition result provided by the recognition model (eg, 810) accurately includes Chinese characters, English characters, and pictograms represented by handwriting input. The user is not required to interrupt in the middle of writing the input in order to select or switch the recognition language.

In some embodiments, the text input area 808 is an area that displays text input provided to each application that employs the handwriting input interface. As shown in FIG. 8A, a text entry area 808 is used by the note application and the text currently shown in the text entry area 808 (eg, “America
")" Is a text input already provided to the memo application. In some embodiments, the cursor 813 indicates the current text input position within the text input area 808.

  In some embodiments, the user can, for example, an explicit selection input (eg, a tap gesture on one of the displayed recognition results) or an implicit confirmation input (eg, an “Enter” button). The specific recognition result displayed in the candidate display area 806 can be selected by the above tap gesture or the double tap gesture in the handwriting input area. As shown in FIG. 8B, the user explicitly uses the tap gesture (as shown by contact 814 on recognition result 810 in FIG. 8A) to identify the highest ranking recognition result 810. Selected. In response to this selection input, the text of the recognition result 810 is inserted at the insertion point indicated by the cursor 813 in the text input area 808. As shown in FIG. 8B, when the text of the selected recognition result 810 is input into the text input area 808, both the handwriting input area 804 and the candidate display area 806 are cleared. At this point, the handwriting input area 804 is ready to accept a new handwriting input, and the candidate display area 806 can now be used to display the recognition results for the new handwriting input. . In some embodiments, the implicit confirmation input causes the top ranking recognition result to be entered in the text input area 808 without requiring the user to interrupt and select the top ranking recognition result. A well-designed implicit confirmation input improves text input speed and reduces the cognitive burden imposed on the user during text composition.

  In some embodiments (not shown in FIGS. 8A, 8B), the highest rank recognition result of the current handwriting input is optionally tentatively displayed in the text input area 808. The temporary text input shown in this text input area 808 is visually identified from other text inputs in the text input area, for example by a temporary input box surrounding the temporary text input. The text shown in this provisional input box has not yet been finalized and has not yet been provided to related applications (for example, memo applications), for example, in response to a user correction of the current handwriting input When the recognition result of the highest rank is changed by the handwriting input module, it is automatically updated.

  9A and 9B are a flow diagram of an example process 900 for providing multiscript handwriting recognition on a user device. In some embodiments, as shown in diagram 900, a user device receives a multiscript handwriting recognition model (902), the multiscript recognition model being spatially derived features of a multiscript training corpus. Trained on (eg, stroke order and stroke direction independent features), the multi-script training corpus includes handwritten samples corresponding to at least three non-overlapping script characters. In some embodiments, the multiscript handwriting recognition model is a single convolutional neural network having a single input surface and a single output surface (906) and is represented in the multiscript training corpus. Includes spatially derived features and respective weights for those spatially derived features to distinguish at least three non-overlapping script characters. In some embodiments, the multi-script handwriting recognition model is configured to recognize characters based on respective input images of one or more recognition units identified in handwriting input (908) and recognize Each spatially derived feature used for is independent of the respective stroke order, stroke direction, and stroke continuity within the handwriting input.

  In some embodiments, the user device receives 908 handwriting input from the user, the handwriting input including one or more handwriting strokes provided on a touch-sensitive surface coupled to the user device. . For example, the handwriting input includes respective data about the location and movement of contact between a finger or stylus and a touch-sensitive surface coupled to the user device. In response to receiving this handwriting input, the user device can receive one or more handwriting recognition results in real time based on a multiscript handwriting recognition model that is trained on spatially derived features of the multiscript training corpus. (910) (912).

  In some embodiments, when providing real-time handwriting recognition results to the user, the user device divides the user's handwriting input into one or more recognition units (914), each recognition unit being defined by the user. Includes one or more of the provided handwriting strokes. In some embodiments, the user device divides the user's handwriting input according to the shape, location, and size of individual strokes created by contact of the user's finger or stylus with the touch-sensitive surface of the user device. To do. In some embodiments, this handwriting segmentation further takes into account the relative order and relative position of individual strokes created by contact of the user's finger or stylus with the touch-sensitive surface of the user device. In some embodiments, the user's handwriting input is a cursive typeface and each continuous stroke in the handwriting input may correspond to a plurality of strokes in the recognized character in printed form. In some embodiments, the user's handwriting input may include continuous strokes across multiple recognized characters in printed form. In some embodiments, the handwriting input segmentation generates one or more input images, each corresponding to each recognition unit. In some embodiments, some of the input images optionally include several overlapping pixels. In some embodiments, those input images do not include any overlapping pixels. In some embodiments, the user device generates a split bundle and each split chain of the split bundle represents a respective scheme that splits the current handwritten input. In some embodiments, each arc in the split chain corresponds to a respective group of strokes in the current handwriting input.

As shown in diagram 900, a user device provides a corresponding image of each of one or more recognition units as input to a multiscript recognition model (914). With respect to at least one of the one or more recognition units, the user device may receive from the multi-script handwriting recognition model, at least a first output character from the first script, and a second different from the first script. At least a second output character from the script is obtained (916). For example, the same input image can cause a multi-script recognition model to output two or more similar-looking output characters from different scripts as recognition results for the same input image. For example, the handwriting input related to the letter “a” in the Latin script and the letter “α” in the Greek script is often similar. Furthermore, the letter “J” in Latin script and the Chinese character “
Are often similar. Similarly, the emoji "
The handwritten input related to "
May be similar to the handwritten input related to "". In some embodiments, the multi-script handwriting recognition model often generates multiple candidate recognition results that are likely to correspond to the user's handwriting input, which is the visual appearance of the handwriting input. This is because even human readers are difficult to decipher. In some embodiments, the first script is a CJK basic character block and the second script is a Latin script as encoded by the Unicode standard. In some embodiments, the first script is a CJK basic character block and the second script is a set of pictograms. In some embodiments, the first script is a Latin script and the second script is a pictograph.

In some embodiments, the user device displays both the first output character and the second output character in the candidate display area of the handwriting input interface of the user device (918). In some embodiments, the user device is one of the first script and the second script that is a corresponding script used with a soft keyboard currently installed on the user device. Based on, one of the first output character and the second output character is selectively displayed (920). For example, the handwriting recognition model has the Chinese character “
”And the Greek letter“ λ ”, the user device has either a Chinese soft keyboard (eg, a keyboard using pinyin input method) or a Greek input keyboard on the user device. Determine if is installed. If the user device determines that only the Chinese soft keyboard is installed, the user device optionally selects the Chinese character “
"Is displayed, and the Greek letter" λ "is not displayed.

  In some embodiments, the user device provides real-time handwriting recognition and input. In some embodiments, the user device may be responsive to continuous additions or corrections to handwriting input by the user before the user makes an explicit or implicit selection of recognition results displayed to the user, One or more recognition results for the user's handwriting input are continuously corrected (922). In some embodiments, in response to each correction of the one or more recognition results, the user sends each corrected one or more recognition results to the user within the candidate display area of the handwriting input user interface. Is displayed (924).

  In some embodiments, the multi-script handwriting recognition model recognizes all characters of at least three non-overlapping scripts, including Chinese characters, pictograms, and Latin scripts, encoded according to the Unicode standard. Trained (926). In some embodiments, the at least three non-overlapping scripts include Chinese characters, Arabic scripts, and Latin scripts. In some embodiments, the multi-script handwriting recognition model has at least 30,000 output classes that represent at least 30 characters across at least three non-overlapping scripts (928).

In some embodiments, the user device allows a user to input multi-script handwriting input, such as a phrase that includes two or more script characters. For example, in order to manually switch the recognition language, the user can continuously write without interruption in the middle of writing, and can receive a handwriting recognition result including characters of two or more scripts. For example, the user enters the multi-script sentence “Hello means” in the handwriting input area of the user device.
in Chinese. ", In which case the Chinese character"
It is not necessary to switch the input language from English to Chinese before writing "" or to switch the input language from Chinese back to English when writing the English word "in Chinese".

  As described herein, a multi-script handwriting recognition model is used to provide real-time handwriting recognition for user input. In some embodiments, real-time handwriting recognition is used to provide real-time multiscript handwriting input functionality on the user's device. 10A-10C are a flow diagram of an example process 1000 for providing real-time handwriting recognition and input on a user device. Specifically, the real-time handwriting recognition is a stroke order independent expression at the character level, the phrase level, and the sentence level.

In some embodiments, stroke order-independent handwriting recognition at the character level may be performed by the handwriting recognition model for that particular handwritten character, regardless of the arrangement of individual strokes of the particular character provided by the user. It is necessary to provide the same recognition result. For example, individual strokes of Chinese characters are typically written in a specific order. Chinese native speakers are often trained in school to write each letter in a specific order, but many users have personal styles and deviations from their traditional stroke order. The stroke arrangement is later adopted. In addition, the script typeface is highly individualized, and multiple strokes in the printed form of Chinese characters are often merged into a single stylized stroke, which is distorted. In some cases, the direction is changed, and further writing is continued to the next character. A stroke order independent recognition model is trained based on images of written samples that do not contain temporal information associated with individual strokes. This recognition is therefore independent of the stroke order information. For example, the Chinese character “
”For the same recognition result“ whether the user writes the horizontal stroke first or the vertical stroke first ”
Is given by the handwriting recognition model.

As shown in FIG. 10A, in process 1000, a user device receives a plurality of handwritten strokes from a user (1002), and the plurality of handwritten strokes corresponds to one handwritten character. For example, the character “
The handwriting input for "" typically includes a substantially horizontal handwriting stroke that intersects a substantially vertical handwriting stroke.

In some embodiments, the user device generates an input image based on the plurality of handwritten strokes (1004). In some embodiments, the user device provides an input image to a handwriting recognition model to perform real-time handwriting recognition of the handwritten character (1006), the handwriting recognition model being a stroke order independent handwriting. Provide recognition. The user device then receives the plurality of handwritten strokes received from the user (e.g., horizontal stroke and vertical stroke) in the same real-time receiving the plurality of handwritten strokes regardless of their order. 1 output character (for example, the character “
”) Is displayed (1008).

  In some conventional handwriting recognition systems, minor stroke order variations with a small number of characters are allowed by specifically including such variations in the training of the handwriting recognition system. Such traditional handwriting recognition systems are not scalable to adapt to arbitrary stroke order variations in many complex characters, such as Chinese characters, but this is a reasonably complex character Even so, this is because a large number of variations in the stroke order have already occurred. Furthermore, by simply including more permutations of acceptable stroke order for a particular character, conventional recognition systems combine multiple strokes into a single stroke (eg, a super-stroke script). Handwritten input (such as in writing) or handwritten input in which one stroke is broken down into multiple substrokes (eg, with characters captured with very rough sampling of input strokes) can still be handled. Impossible. Therefore, a multi-script handwriting system trained on spatially derived features, as described herein, offers advantages over conventional recognition systems.

  In some embodiments, stroke order independent handwriting recognition is performed independently of temporal information associated with individual strokes in each handwritten character. In some embodiments, stroke order-independent handwriting recognition is performed in conjunction with stroke distribution information that takes into account the spatial distribution of individual strokes before being merged into a flat input image. The Further details on how to use this temporally derived stroke distribution information to enhance the above-described stroke order independent handwriting recognition will be discussed later (eg, in FIGS. 25A-27). As provided). The technique described in connection with FIGS. 25A to 27 does not impair the stroke order independence of the handwriting recognition system.

In some embodiments, the handwriting recognition model provides stroke direction independent handwriting recognition (1010). In some embodiments, stroke direction independent recognition is performed regardless of the respective stroke direction of the plurality of handwritten strokes provided by the user in response to receipt of the plurality of handwritten inputs by the user device. , It is necessary to display the same first output character. For example, a user places a Chinese character “
The handwriting recognition model outputs the same recognition result regardless of whether the user has drawn a horizontal stroke from left to right or right to left. Similarly, the handwriting recognition model outputs the same recognition result regardless of whether the user has drawn a vertical stroke in a downward direction or an upward direction. In another embodiment, many Chinese characters are structurally created with two or more radicals. Some Chinese characters each include a left radical and a right radical, and people routinely write the left radical first and the right radical second. In some embodiments, the handwriting recognition model is used by the user as long as the resulting handwritten input shows the left radical to the left of the right radical when the user completes the handwritten character. Regardless of whether the left radical is written first or the right radical is written first, the same recognition result is provided. Similarly, some Chinese characters each include an upper radical and a lower radical, and people routinely write the upper radical first and the lower radical last. In some embodiments, the handwriting recognition model is that the user wrote the upper radical first as long as the resulting handwriting input shows the upper radical above the lower radical, or Provides the same recognition result regardless of whether the lower radical is written first. In other words, the handwriting recognition model determines the identity of handwritten characters without depending on the direction of individual strokes of handwritten characters provided by the user.

In some embodiments, the handwriting recognition model provides handwriting recognition based on the image of the recognition unit regardless of the number of recognition unit substrokes provided by the user. In other words, in some embodiments, the handwriting recognition model provides stroke number independent handwriting recognition (1014). In some embodiments, in response to receiving a plurality of handwritten strokes, the user device is identical regardless of how many handwritten strokes are used to form a continuous stroke in the input image. Display the first output character. For example, when the user has written a Chinese character “+” in the handwriting input area, the handwriting recognition model indicates that the user
4 strokes (e.g., two short horizontal strokes and two short vertical strokes to construct a cross-shaped character) or two strokes (e.g., , L-shaped strokes and 7-shaped strokes, or horizontal and vertical strokes), or any other number of strokes (eg, hundreds of extremely short strokes or points) Regardless of whether the same recognition result is output.

  In some embodiments, the handwriting recognition model is not only capable of recognizing the same character regardless of the order in which each single character was written, direction, and number of strokes, but also the handwriting recognition model. It is also possible to recognize multiple characters regardless of the temporal order of the strokes of the multiple characters provided by the user.

In some embodiments, the user device not only receives a first plurality of handwritten strokes from the user, but also receives a second plurality of handwritten strokes (1016), and the second of them. The plurality of handwritten strokes correspond to the second handwritten character. In some embodiments, the user device generates a second input image based on the second plurality of handwriting strokes (1018). In some embodiments, the user device provides the second input image to the handwriting recognition model to perform real-time recognition of the second handwritten character (1020). In some embodiments, the user device displays a second output character corresponding to the second plurality of handwritten strokes in real time upon receiving the second plurality of handwritten strokes (1022). In some embodiments, the second output character and the first output character are independent of the respective order of the first plurality of handwriting strokes and the second plurality of handwriting strokes provided by the user, Displayed simultaneously in spatial arrangement. For example, when a user places two Chinese characters (eg, “
"as well as"
)) Is written, the user device will write the handwritten input currently stored in the handwritten input area with the character “
To the left of the stroke for "
As long as the strokes related to "
”Or the character“
”Regardless of whether the stroke was first written
Is displayed. In fact, the user
"Before a part of the stroke (for example, a vertical stroke)
Even if a part of the stroke related to “(”, for example, a stroke inclined to the left) is written, the image obtained as a result of the handwriting input in the handwriting input area is the character “
To the left of all strokes related to
As long as all strokes related to "are shown, the user device is able to recognize the recognition result"
Is shown.

  In other words, as shown in FIG. 10B, in some embodiments, the spatial arrangement of the first output character and the second output character is the default (eg, left) of the handwriting input interface of the user device. Corresponding to the spatial distribution of the first plurality of handwritten strokes and the second plurality of strokes along the writing direction (from right to right) (1024). In some embodiments, the second plurality of handwriting strokes is received in time later than the first plurality of handwriting strokes (1026), and the second output character is the handwriting input interface of the user device. A spatial arrangement along the default (eg, left to right) writing direction that precedes the first output character.

In some embodiments, the handwriting recognition model provides stroke order independent recognition at the sentence to sentence level. For example, in the handwriting input area, the handwritten character “
"Is present in the first handwritten sentence and the handwritten character"
”Is present in the second handwritten sentence and the two handwritten characters are separated by one or more other handwritten characters and / or words, Spatial arrangement of two characters
The recognition result indicated by "is still provided. The recognition result and the spatial arrangement of the two recognized characters indicate that when the user completes the handwriting input, the recognition units related to the two characters are the array “
Is kept the same regardless of the temporal order of the strokes of those two characters provided by the user. In some embodiments, the first handwritten character (eg, “
") Is the first handwritten sentence (e.g.,"
is a number. ") As part of the second handwritten character (e.g.,"
") Is a second handwritten sentence (e.g.,"
is another number. )), Provided by the user, these first handwritten text and second handwritten text are simultaneously displayed in the handwriting input area of the user device. In some embodiments, this recognition result (eg, “
is a number.
is another number. When the user confirms that “”) is the correct recognition result, those two sentences are entered into the text input area of the user device, which is used by the user to enter another handwriting input. Cleared.

  In some embodiments, the handwriting recognition model is stroke order independent, not only at the character level, but also at the phrase level and sentence level, so the user can modify the previous unfinished characters and replace subsequent characters. Can be implemented after writing. For example, if a user has forgotten to write a particular stroke for a character before moving to writing one or more subsequent characters in the handwriting input area, the user will still be able to The missing stroke can be entered later at the correct place in the character to receive the correct recognition result.

  In traditional stroke order dependent systems (eg, recognition systems based on HMM), once a character is written, the character is fixed and the user can no longer make any changes to the character. . If the user wants to make any changes, he must delete that character and all subsequent characters and start over. In some conventional recognition systems, the user is required to complete a handwritten character within a short default time window, and any stroke entered outside that default time window is It is not included in the same recognition unit as other strokes provided between windows. Such conventional systems are difficult to use and cause significant irritation to the user. A stroke order independent system is immune to these drawbacks, and the user can complete the characters in any order and in any time frame that the user may find appropriate. The user may also perform modifications to previously written characters (eg, adding one or more strokes) after subsequently writing one or more characters in the handwriting input interface. In some embodiments, the user can also delete previously written characters individually (eg, using methods described later in connection with FIGS. 21A, 22B) and within the handwriting input interface. You can also rewrite that character in the same place.

As shown in FIGS. 10B and 10C, the second plurality of handwritten strokes spatially follow the first plurality of handwritten strokes along the default writing direction of the handwriting input interface of the user device ( 1028), the second output character follows the first output character in a spatial arrangement along its default writing direction within the candidate display area of the handwriting input interface. The user device receives a third handwritten stroke from the user (1030) to correct the first handwritten character (ie, a handwritten character formed by the first plurality of handwritten strokes), and the third device The handwritten stroke is received later in time than the first plurality of handwritten strokes and the second plurality of handwritten strokes. For example, a user may have two characters (eg, “
)). The first plurality of strokes is a handwritten character “
”. The user actually has the character "
Note that one stroke was overwritten, although it was intended. The second plurality of strokes is a handwritten character “
”. The user
"not"
And then understands that he wanted to write "
Below the stroke for "", another vertical stroke can be entered, and the user device can use that vertical stroke as a first recognition unit (e.g., "
To the recognition unit). The user device can create a new output character for the first recognition unit (eg, “
”), And this new output character is the previous output character (for example,“
)). As shown in FIG. 10C, in response to receiving the third handwritten stroke, the user device determines the third handwritten stroke based on the relative proximity of the third handwritten stroke to the first plurality of handwritten strokes. The third handwriting stroke is assigned to the same recognition unit as the plurality of one handwriting stroke (1032). In some embodiments, the user device generates a corrected input image based on the first plurality of handwriting strokes and the third handwriting stroke (1034). The user device provides the corrected input image to the handwriting recognition model to perform real-time recognition of the corrected handwritten character (1036). In some embodiments, in response to receiving the third handwritten input, the user device displays a third output character corresponding to the corrected input image (1040), wherein the third output character is Instead of the first output character, it is displayed simultaneously with the second output character in a spatial arrangement along the default writing direction.

  In some embodiments, the handwriting recognition module recognizes handwriting input written with a default writing direction from left to right. For example, the user can write characters from left to right and in one or more lines. In response to this handwriting input, the handwriting input module presents a recognition result, optionally in a spatial arrangement from left to right, and including characters in one or more lines. When the user selects a recognition result, the selected recognition result is input into the text input area of the user device. In some embodiments, the default writing direction is from top to bottom. In some embodiments, the default writing direction is right to left. In some embodiments, the user optionally changes the default writing direction to an alternative writing direction after a recognition result is selected and the handwriting input area is cleared.

  In some embodiments, the handwriting input module allows a user to input multiple-character handwriting input into the handwriting input area, and not all recognition units at once from the handwriting input, A stroke of one recognition unit can be deleted. In some embodiments, the handwriting input module allows deletion of one stroke at a time from handwriting input. In some embodiments, the recognition unit deletion proceeds one by one in the opposite direction from the default writing direction, regardless of the recognition unit or stroke order entered to produce the current handwriting input. To do. In some embodiments, the deletion of a stroke proceeds one by one in the reverse order in which it was entered within each recognition unit, and once all strokes within a recognition unit have been deleted, Deletion proceeds to the next recognition unit in a direction opposite to the default writing direction.

  In some embodiments, the user device may delete input from the user while the third output character and the second output character are simultaneously displayed as candidate recognition results in the candidate display area of the handwriting input interface. Receive. In response to this deletion input, the user device deletes the second output character from the recognition result, while maintaining the third output character in the recognition result displayed in the candidate display area.

In some embodiments, as shown in FIG. 10C, the user device may have a first plurality of handwriting strokes, a second plurality of handwriting strokes, and a third handwriting stroke, each of which is a handwriting stroke. Rendering in real time as provided by the user (1042). In some embodiments, in response to receiving the deletion input from the user, the user device renders each of the second plurality of handwriting inputs (eg, corresponding to the second handwritten character) from the handwriting input area. While rendering each rendering of the first plurality of handwriting strokes and the third handwriting stroke (for example, corresponding to the corrected first handwritten character together) in the handwriting input area. Maintain (1044). For example, if the user
After providing a missing vertical stroke in "
The stroke in the recognition unit for "is removed from the handwriting input area and the character"
”Is the recognition result“
Is removed. After this deletion, the character "
”Is maintained in the handwriting input area, and the recognition result is the character“
] Only.

  In some embodiments, the handwritten character is a multi-stroke Chinese character. In some embodiments, the first plurality of handwriting inputs is provided in a script typeface. In some embodiments, the first plurality of handwriting inputs is provided in a script typeface, and the handwritten characters are multi-stroke Chinese characters. In some embodiments, the handwritten characters are written in cursive Arabic. In some embodiments, the handwritten characters are written in other scripts of the script.

  In some embodiments, the user device establishes a respective default constraint for a set of acceptable dimensions for handwriting input, and the currently accumulated multiple handwritten strokes to each default constraint. Based on the recognition unit, each input image is generated from each of the recognition units, provided to the handwriting recognition model, and recognized as a corresponding output character.

  In some embodiments, the user device receives a further handwriting stroke from the user after dividing the currently stored handwriting strokes. The user device assigns the additional handwriting stroke to each one of the plurality of recognition units based on the spatial position of the additional handwriting stroke relative to the plurality of recognition units.

  Attention is now directed to an exemplary user interface for providing handwriting recognition and input on a user device. In some embodiments, this exemplary user interface is provided on a user device based on a multi-script handwriting recognition model that provides real-time stroke order independent handwriting recognition of the user's handwriting input. . In some embodiments, the exemplary user interface includes a handwriting input area 804, a candidate display area 804, and a text input area 808 (eg, as shown in FIGS. 8A and 8B). It is a user interface of the input interface 802. In some embodiments, the exemplary handwriting input interface 802 also includes a plurality of control elements 1102, such as a delete button, space bar, Enter button, keyboard switch button, and the like. One or more other areas and / or elements can be provided within the handwriting input interface 902 to allow further functionality described below.

  As described herein, a multi-script handwriting recognition model can have a very wide repertoire of tens of thousands of characters in many different scripts and languages. As a result, for handwriting input, this recognition model is very likely to identify a large number of output characters, all of which are reasonably good likelihood that they are characters intended by the user Have On user devices with a limited display area, it is advantageous to initially provide only a subset of recognition results while other results remain available upon user request.

  FIGS. 11A-11G display a subset of the recognition results in the normal view of the candidate display area, along with an affordance for invoking an expanded view of the candidate display area to display the remainder of the recognition result. FIG. 6 shows an exemplary user interface of FIG. Furthermore, in the expanded view of the candidate display area, the recognition results are classified into different categories and displayed on different tabbed pages in the expanded view.

  FIG. 11A shows an exemplary handwriting input interface 802. The handwriting input interface includes a handwriting input area 804, a candidate display area 806, and a text input area 808. One or more control elements 1102 are also included in the handwriting input interface 1002.

  As shown in FIG. 11A, candidate display area 806 optionally includes an area for displaying one or more recognition results and an affordance 1104 for calling an expanded version of candidate display area 806 (eg, an extension Icon).

  FIGS. 11A-11C illustrate that when a user provides one or more handwriting strokes in handwriting input area 804 (eg, strokes 1106, 1108, and 1110), the user device currently accumulates in handwriting input area 804. FIG. It shows that each set of recognition results corresponding to the stroke being identified is specified and displayed. As shown in FIG. 11B, after the user has entered the first stroke 1106, the user device has three recognition results 1112, 1114, and 1116 (eg, the characters “/”, “1”, and “, )) Is specified and displayed. In some embodiments, within the candidate display area 806, a small number of candidate characters are displayed in an order according to the recognition confidence associated with each character.

  In some embodiments, the highest ranked candidate result (eg, “/”) is tentatively displayed in text entry area 808, eg, in box 1118. The user confirms that the top-ranked candidate is the intended input with a simple confirmation input (eg, by pressing the “Enter” key or providing a double-tap gesture in the handwriting input area). Can optionally be confirmed.

  FIG. 11C shows that before the user selects any candidate recognition results, if the user enters two more strokes 1108 and 1110 into the handwriting input area 804, those additional strokes, along with the initial stroke 1106, are entered. Rendered in the handwriting input area 804, indicating that the candidate results are updated to reflect those changes in the recognition units identified from the currently stored handwriting input. As shown in FIG. 11C, based on the three strokes, the user device has identified a single recognition unit. Based on this single recognition unit, the user device identifies and displays several recognition results 1118 to 1124. In some embodiments, one or more of these recognition results (eg, 1118 and 1122) currently displayed in the candidate display area 806 are each a plurality of similarities for the current handwriting input. Each representative candidate character selected from the appearance candidate characters.

  As shown in FIGS. 11C and 11D, when the user selects affordance 1104 (eg, using a tap gesture with contact 1126 on affordance 1104), the candidate display area is It changes from a normal view (shown in 11C) to an extended view (eg shown in FIG. 11D). In some embodiments, this expanded view shows all of the recognition results (eg, candidate characters) that have been identified for the current handwriting input.

  In some embodiments, the initially displayed normal view of the candidate display area 806 shows only the most commonly used characters used in the respective script or language, while expanding The view shows all candidate characters, including characters that are less frequently used in scripts or languages. The extended view of this candidate display area can be designed in various ways. 11D-11G illustrate exemplary designs of extended candidate display areas, according to some embodiments.

  As shown in FIG. 11D, in some embodiments, the expanded candidate display area 1128 includes one or more tabbed pages (eg, pages 1130, 1132, each) that present a respective category of candidate characters. 1134 and 1136). The tabbed design shown in FIG. 11D allows the user to quickly find the desired character category and then find the character that the user intended to enter in its corresponding tabbed page. Become.

In FIG. 11D, the first tabbed page 1130 displays all candidate characters that have been identified for the current handwritten input, including both commonly used characters as well as infrequent characters. indicate. As shown in FIG. 11D, the tabbed page 1130 includes all of the characters that were shown in the initial candidate display area 806 in FIG. 11C and some that were not included in the initial candidate display area 806. Additional characters (for example, "
”,“ Β ”,“
Etc.).

  In some embodiments, the characters displayed in the initial candidate display area 806 are characters from a commonly used set of characters associated with a particular script (eg, encoded according to the Unicode standard). All characters in the basic block of the CJK script. In some embodiments, the characters displayed in the expanded candidate display area 1128 are CJK script extensions, such as those encoded according to the Unicode standard, that are associated with the script. All characters in the block). In some embodiments, the expanded candidate display area 1128 further includes candidate characters from other scripts not commonly used by the user, such as Greek scripts, Arabic scripts, and / or emoji scripts.

  In some embodiments, as shown in FIG. 11D, each expanded candidate display area 1128 includes a respective category of candidate characters (eg, all characters, infrequent characters, characters from a Latin script, respectively). , And characters from the emoji script), respectively, with tabbed pages 1130, 1132, 1134, and 1138. FIGS. 11E-11G illustrate that the user can select each of the various tabbed pages to reveal candidate characters in the corresponding category. FIG. 11E shows only infrequent characters (eg, characters from an extended block of a CJK script) corresponding to the current handwritten input. FIG. 11F shows only Latin and Greek letters corresponding to the current handwriting input. FIG. 11G shows only pictograms corresponding to the current handwriting input.

  In some embodiments, the expanded candidate display area 1128 displays candidate characters in each tabbed page based on respective criteria (eg, based on phonetic spelling, based on the number of strokes, and It further includes one or more affordances for reordering (based on radicals, etc.). The ability to reorder candidate characters within each category according to criteria other than the recognition confidence score provides the user with the additional ability to quickly find the desired candidate character for text entry.

  11H-11K may group similar appearance candidate characters in some embodiments, and only representative characters from each group of similar appearance candidate characters are It shows that it is presented in the candidate display area 806. Because the multi-script recognition model described herein can produce many candidate characters that are approximately equally valid for a given handwriting input, this recognition model can be used to convert one candidate to another similar look. It is not always possible to delete a candidate at the expense. On devices with limited display area, displaying many similar appearance candidates all at once does not help the user to pick the right character, but this makes it easy to make fine distinctions Even if the user cannot view the desired character, selecting the character from a very crowded display using a finger or stylus This is because it may be difficult.

In some embodiments, in order to address the above problems, the user device may select candidate characters that are highly similar to each other (eg, according to a concordance or dictionary of similar looking characters, or some criteria based on images). Identify and group those candidate characters into their respective groups. In some embodiments, one or more groups of similar looking characters can be identified from a set of candidate characters for a given handwriting input. In some embodiments, the user device has identified a representative candidate character from a plurality of similar-looking candidate characters in the same group, and only the representative candidate is displayed in the initial candidate display area 806. Display within. If a commonly used character does not appear sufficiently similar to any other candidate character, the character itself is displayed. In some embodiments, as shown in FIG. 11H, each group of representative candidate characters (eg, candidate characters 1118 “
”And 1122“ T ”) are candidate characters that do not belong to any group (eg, candidate character 1120“ T ”).
"And 1124" J ") are displayed in a different manner (eg, in a thick box). In some embodiments, the criteria for selecting a representative character for a group is based on the relative usage of candidate characters within that group. In some embodiments, other criteria can be used.

  In some embodiments, when representative characters are displayed to the user, the user optionally expands the candidate display area 806 to show candidate characters that look similar in an enlarged view. Can do. In some embodiments, selection of a particular representative character can result in an enlarged view of only candidate characters within the same group as the selected representative character.

  Various designs are possible to provide an enlarged view of similar visual candidates. FIGS. 11H-11K illustrate an embodiment in which a magnified view of a representative candidate character is invoked by a predetermined gesture (eg, an enlarged gesture) detected on a representative candidate character (eg, representative character 1118). Indicates. A default gesture for invoking an enlarged view (eg, an enlarged gesture) is different from a default gesture (eg, a tap gesture) for selecting representative characters for text input.

As shown in FIGS. 11H and 11I, the user makes an enlarged gesture over the first representative character 1118 (eg, as shown by two contacts 1138 and 1140 moving away from each other). When provided, the area displaying the representative character 1118 is enlarged, and three similar appearance candidate characters (e.g., “
, "
",as well as"
”) Other candidate characters that are not in the same expanded group (for example,“
') Is presented in an enlarged view (eg, in enlarged boxes 1142, 1144, and 1146, respectively).

As shown in FIG. 11I, when presented in an enlarged view, three similar appearance candidate characters (eg, “
, "
",as well as"
") Can be visually recognized more easily by the user. If one of these three candidate characters is the intended character input, the user can select the candidate character, for example, by touching the area where the character is displayed. it can. As shown in FIGS. 11J and 11K, the user can select a second character (eg, “
") Is selected (at contact 1148). In response, the selected character (for example, “
") Is entered in the text entry area 808 at the insertion point indicated by the cursor. As shown in FIG. 11K, when a character is selected, the handwritten input in the handwriting input area 804 and the candidate characters in the candidate display area 806 (or an enlarged view of the candidate display area) are To be erased.

  In some embodiments, if the user does not see the desired candidate character in the enlarged view of the first representative candidate character 1142, the user optionally uses the same gesture, Other characters displayed in the candidate display area 806 can be enlarged. In some embodiments, another representative character in candidate display area 806 is magnified, and the currently presented magnified view is automatically restored to the normal view. In some embodiments, the user optionally uses a reduction gesture to restore the currently magnified view to a normal view. In some embodiments, the user can scroll the candidate display area 806 (eg, to the left or right) to display other candidate characters that are not visible in the candidate display area 806.

  12A and 12B show that the first subset of recognition results is presented in the initial candidate display area, while the second subset of recognition results is hidden until specifically invoked by the user. 2 is a flow diagram of an example process 1200 presented within an expanded candidate display area that is not visible. In the exemplary process 1200, the device identifies a subset of recognition results having a visual similarity level that exceeds a predetermined threshold from a plurality of handwriting recognition results for handwriting input. The user device then selects a representative recognition result from the subset of recognition results and displays the selected representative recognition result in a candidate display area of the display. Process 1200 is shown in FIGS. 11A-11K.

  As shown in FIG. 12A, in an exemplary process 1200, a user device receives handwriting input from a user (1202). The handwriting input is one or more handwriting strokes (eg, 1106 in FIG. 11C) provided within a handwriting input area (eg, 806 in FIG. 11C) of a handwriting input interface (eg, 802 in FIG. 11C). 1108, 1110). Based on the handwriting recognition model, the user device identifies a plurality of output characters (for example, characters shown in the tabbed page 1130 in FIG. 11C) regarding the handwriting input (1204). The user device classifies the plurality of output characters into two or more categories based on predetermined categorization criteria (1206). In some embodiments, the default categorization criteria determines whether each character is a commonly used character or a low frequency character (1208).

  In some embodiments, the user device may include a first category (eg, two or more categories) within an initial view of a candidate display area (eg, 806 as shown in FIG. 11C) of the handwriting input interface. , Commonly used characters) to display each output character (1210), and the initial view of this candidate display area calls up an expanded view of the candidate display area (eg, 1128 in FIG. 11D) Offered at the same time (eg, 1104 in FIG. 11C).

  In some embodiments, the user device receives user input (1212) that selects affordances for invoking an enhanced view, eg, as shown in FIG. 11C. In response to this user input, the user device, for example, in the expanded view of the candidate display area, each output character in the first category, and in the initial view of the candidate display area, as shown in FIG. 11D. Each of the output characters in at least the second category of the two or more categories that has not been previously displayed is displayed (1214).

  In some embodiments, each character in the first category is a character found in a commonly used dictionary of characters, and each character in the second category is an infrequent character. Characters found in the dictionary. In some embodiments, commonly used character dictionaries and infrequent character dictionaries are dynamically adjusted or updated based on usage histories associated with user devices.

  In some embodiments, the user device may be based on a predetermined similarity criterion (e.g., based on a dictionary of similar characters or based on some spatially derived feature) from the plurality of output characters. A group of characters visually similar to each other is identified (1216). In some embodiments, the user device selects a representative character from a group of visually similar characters based on predetermined selection criteria (eg, based on historical usage frequency). In some embodiments, this default selection criteria is based on the relative usage of characters within the group. In some embodiments, this default selection criteria is based on the preferred input language associated with the device. In some embodiments, the representative candidates are based on other factors that indicate the likelihood that each candidate is the intended input by the user. These factors include, for example, whether the candidate character belongs to a script for a soft keyboard currently installed on the user's device, or whether the candidate character is associated with the user or user device. Whether it is present in the most commonly used set of characters in any language.

In some embodiments, the user device may include a representative character (eg, “" in the initial view of the candidate display area (eg, 806 in FIG. 11H).
") For other characters in the group of visually similar characters (e.g.,"
, "
")" Is displayed instead of (1220). In some embodiments, in the initial view of the candidate display area to indicate whether each candidate character is a representative character of the group or a normal candidate character that does not exist in any group. Visual instructions (eg, selective visual highlighting, special background) are provided. In some embodiments, the user device may have a default magnified input (eg, a magnified gesture) that targets representative characters displayed in the initial view of the candidate display area, for example, as shown in FIG. 11H. Is received from the user (1222). In some embodiments, in response to receiving this default magnified input, the user device may have a group of characters that are visually similar to the magnified view of the representative character, eg, as shown in FIG. 11I. Each enlarged view of one or more of the other characters is displayed simultaneously (1224).

  In some embodiments, this default magnified input is a magnified gesture that is detected on representative characters displayed in the candidate display area. In some embodiments, this default magnified input is a touch that is detected on a representative character displayed in the candidate display area and lasts longer than a predetermined threshold time. In some embodiments, this continuous contact for expanding a group has a longer threshold duration than a tap gesture that selects representative characters for text entry.

  In some embodiments, each representative character is displayed simultaneously with a corresponding affordance (e.g., a corresponding zoom button) to invoke a zoomed view of the group of candidate characters that look similar. In some embodiments, the default expanded input is a selection of a corresponding affordance associated with the representative character.

  As described herein, in some embodiments, the repertoire of multi-script handwriting recognition models includes emoji scripts. The handwriting input recognition module can recognize a pictograph based on a user's handwriting input. In some embodiments, the handwriting recognition module presents both a pictograph identified directly from handwriting and a character or word in natural human language that represents the identified pictograph. In some embodiments, the handwriting input module recognizes a character or word in a natural human language based on the user's handwriting input and corresponds to the recognized character or word and the recognized character or word Present both the emoji to be. In other words, the handwriting input module provides a method for inputting pictograms without switching from the handwriting input interface to the pictogram keyboard. Furthermore, the handwriting input module also provides a method for inputting regular natural language characters by drawing pictograms by hand. 13A-13E provide an exemplary user interface that illustrates these various ways of entering pictograms and regular natural language characters.

FIG. 13A shows an exemplary handwriting input interface 802 invoked under a chat application. The handwriting input interface 802 includes a handwriting input area 804, a candidate display area 806, and a text input area 808. In some embodiments, once the user is satisfied with the text composition in the text entry area 808, the user can choose to send the text composition to another participant in the current chat session. The interaction history of the chat session is shown in the interaction panel 1302. In this example, the user may enter a chat message 1304 (eg, “Happy Birthdayday” displayed in the interaction panel 1302.
)).

  As shown in FIG. 13B, the user is providing handwriting input 1306 for the English word “Thanks” in handwriting input area 804. In response to the handwriting input 1306, the user device has identified several candidate recognition results (eg, recognition results 1308, 1310, and 1312). The recognition result 1303 of the highest rank is provisionally input in the box 1314 in the text input area 808.

  As shown in FIG. 13C, after the user has entered the handwritten word “Thanks” in the handwriting input area 806, the user then places a stroke 1316 (eg, down a round circle) in the handwriting input area 806. A stylized exclamation mark is drawn with a slender circle. The user device recognizes that this additional stroke 1316 forms a recognition unit that is separate from other recognition units previously recognized from the accumulated handwriting stroke 1306 in the handwriting input area 806. Based on this newly entered recognition unit (ie, the recognition unit formed by stroke 1316), the user device uses a handwriting recognition model to identify a pictograph (eg, stylized “!”). To do. Based on this recognized pictogram, the user device presents a first recognition result 1318 (eg, “Thanks!” With stylized “!”) In the candidate display area 806. Furthermore, the user device also identifies the number “8”, which is also visually similar to the newly entered recognition unit. Based on this recognized number, the user device presents a second recognition result 1322 (eg, “Thanks 8”) in the candidate display area 806. Further, based on the identified pictograph (eg, stylized “!”), The user device also identifies a regular character (eg, the regular character “!”) Corresponding to the pictograph. Based on the indirectly recognized legitimate characters, the user device presents a third recognition result 1320 (eg, “Thanks!” With legitimate “!”) In the candidate display area 806. At this point, the user can select any one of candidate recognition results 1318, 1320, and 1322 and enter the candidate recognition result in text input area 808.

As shown in FIG. 13D, the user continues to provide additional handwriting strokes 1324 within the handwriting input area 806. This time, the user is drawing a heart symbol following the stylized exclamation mark. In response to this new handwriting stroke 1324, the user device recognizes that the newly provided handwriting stroke 1324 forms yet another new recognition unit. Based on this new recognition unit, the user device can use the pictograph “
", And alternatively the number" 0 ". Based on these new candidate characters recognized from the new recognition unit, the user device can generate two updated candidate recognition results 1326 and 1330 (e.g., "Thanks
And “Thanks 80”). In some embodiments, the user device may identify an identified emoji (eg, “
”) Further identifies a regular character or word (eg,“ Love ”). Based on the identified canonical character or word for the recognized pictogram, the user device presents a third recognition result 1328 in which the recognized pictogram has been replaced with its corresponding canonical character or word. To do. As shown in FIG. 13D, in the recognition result 1328, the pictogram “
"Will be replaced with the normal exclamation mark"! "
"Is replaced with the regular character or the word" Love ".

As shown in FIG. 13E, the user can select one of the candidate recognition results (eg, the mixed script text “Thanks
”Is selected, and the selected recognition result text is entered into the text entry area 808 and then sent to other participants in the chat session. Message balloon 1332 shows the text of the message in dialog panel 1302.

  FIG. 14 is a flow diagram of an example process 1400 where a user enters a pictograph using handwriting input. 13A-13E illustrate an exemplary process 1400, according to some embodiments.

In process 1400, the user device receives handwriting input from a user (1402). The handwriting input includes a plurality of handwriting strokes provided in a handwriting input area of the handwriting input interface. In some embodiments, the user device recognizes a plurality of output characters from the handwriting input based on the handwriting recognition model (1404). In some embodiments, the output characters are at least a first glyph (eg, stylized exclamation mark “
"Or emoji"
)) And at least a first character from a script in natural human language (eg, a character from the word “Thanks” in FIG. 13D). In some embodiments, the user device may have their first pictograph (eg, stylized exclamation mark “in FIG. 13D, as shown in FIG. 13D).
"Or emoji"
)) And a first character from a natural human language script (eg, a character from the word “Thanks” in FIG. 13D) (eg, a result 1326 in FIG. 13D) with a handwritten input interface Are displayed in the candidate display area (1406).

In some embodiments, based on the handwriting recognition model, the user device optionally recognizes (1408) at least a first semantic unit (eg, the word “Thanks”) from its handwriting input. This first semantic unit includes each character, word, or phrase that can convey the respective semantic meaning in the respective human language. In some embodiments, the user device identifies a second glyph (eg, “shake hand” glyph) associated with a first semantic unit (eg, the word “Thanks”) recognized from handwriting input. (1410). In some embodiments, the user device includes at least a second pictograph identified from a first semantic unit (eg, the word “Thanks”) in a candidate display area of the handwriting input interface, The recognition result (for example, “
"as well as"
"Recognition result indicating the pictograph" is displayed (1412). In some embodiments, displaying the second recognition result includes a third recognition result (eg, the recognition result “Thanks”) that includes at least a first semantic unit (eg, the word “Thanks”).
")) And simultaneously displaying the second recognition result.

  In some embodiments, the user receives a user input that selects the first recognition result displayed in the candidate display area. In some embodiments, in response to the user input, the user device enters the selected first recognition result text in a text input area of the handwriting input interface, the text being at least a first 1 pictograph and a first character from a natural human language script. In other words, the user can use a single handwriting input (although handwriting input including multiple strokes) in the handwriting input area without switching between a natural language keyboard and an emoji keyboard, It is possible to enter an input.

  In some embodiments, the handwriting recognition model is trained on a multi-script training corpus that includes written samples corresponding to at least three non-overlapping script characters, wherein the three non-overlapping scripts are emoji, Chinese Includes a set of characters and Latin scripts.

In some embodiments, the user device may recognize a first pictograph (eg, “
The second semantic unit (for example, the word “Love”) corresponding to “the pictogram” is identified (1414). In some embodiments, the user device places a first pictograph (e.g., "" in the candidate display area of the handwriting input interface.
A fourth recognition result (eg, 1328 in FIG. 13D) that includes at least the second semantic unit (eg, the word “Love”) identified from the “ In some embodiments, the user device may place a first recognition result (eg, result “Thanks” in the candidate display area, as shown in FIG. 13D.
At the same time, the fourth recognition result (for example, the result 1328 “Thanks! Love”) is displayed.

In some embodiments, the user device allows the user to enter regular text by drawing an emoji. For example, if the user does not know how to spell the word “elephant”, the user optionally draws a stylized pictograph for “elephant” in the handwriting input area, and the user device If the handwriting input can be correctly recognized as a pictograph relating to “elephant”, the user device may also optionally select normal text as one of the recognition results displayed in the candidate display area. Also presents the word “elephant”. In another embodiment, the user places a Chinese character “
Instead of writing ", you can draw a stylized cat. If the user device identifies an emoji related to “Cat” based on the handwriting input provided by this user, the user device may also optionally include an emoji related to “Cat” in the candidate display area. , The Chinese character “cat” in Chinese
". By presenting normal text about the recognized emoji, the user device enters a complex character or word using a small number of stylized strokes that are typically associated with known emoji. Provide an alternative method. In some embodiments, the user device may convert the emoticons into their corresponding normal text (eg, characters, words, phrases, symbols) in one or more preferred scripts or languages (eg, English or Chinese). Etc.), and the dictionary is memorized.

  In some embodiments, the user device recognizes a pictograph based on the visual similarity of the pictograph to an image generated from handwritten input. In some embodiments, the handwriting recognition model used on the user device to enable recognition of pictograms from handwriting input is a handwritten sample corresponding to a script in natural human language, and artificially Trained using a training corpus that includes both handwritten samples corresponding to the set of designed pictograms. In some embodiments, glyphs associated with the same semantic concept may have different appearances when used in mixed input with different natural language text. For example, a pictograph related to the semantic concept of “Love” can be a “heart” pictograph when presented in normal text in one natural language (eg, Japanese) and another natural language (eg, , English or French) can be a “kiss” pictograph.

  As described herein, when performing multi-character handwriting input recognition, the handwriting input module performs splitting of the handwriting input currently stored in the handwriting input area and stores them. Separating the strokes into one or more recognition units. One of the parameters used to determine how to split the handwriting input is how the strokes are clustered within the handwriting area and between clusters of different strokes Distance. People have various typefaces. Some people tend to write at very low density, increasing the distance between strokes, or between different parts of the same character, while others are distances between strokes or different There is a tendency to write at a very high density with a very small distance between characters. Even for the same user, due to imperfect planning, handwritten characters may deviate from a balanced appearance and may be biased, stretched or compressed in various ways. As described herein, the multi-script handwriting recognition model provides stroke order-independent recognition, thus allowing the user to write characters or portions of characters out of order. . As a result, achieving spatial uniformity and balance of handwriting input between characters can be difficult.

  In some embodiments, the handwriting input model described herein allows the user to cause the handwriting input module to fuse two adjacent recognition units into a single recognition unit, or a single A method is provided for notifying whether a recognition unit should be separated into two separate recognition units. With the help of this user, the handwriting input module can correct the initial division and generate the result intended by the user.

  FIGS. 15A-15J illustrate some exemplary user interfaces and processes in which a user provides default pinch and magnification gestures to modify the recognition unit specified by the user device.

As shown in FIGS. 15A and 15B, the user inputs a plurality of handwriting strokes 1502 (for example, three strokes) in the handwriting input area 806 of the handwriting input interface 802. The user device identifies a single recognition unit based on the handwritten stroke 1502 accumulated at the present time, and three candidate characters 1508 (for example, “
, "
",as well as"
)).

  FIG. 15C shows that the user has additionally entered a small number of additional strokes 1510 to the right of the initial handwriting stroke 1502 in the handwriting input area 606. The user device determines that stroke 1502 and stroke 1510 should be considered as two separate recognition units (eg, based on the dimensions and spatial distribution of multiple strokes 1502 and 1510). Based on the separation of the recognition units, the user device provides the input images of the first recognition unit and the second recognition unit to the handwriting recognition model, and obtains two sets of candidate characters. The user device then generates a plurality of recognition results (eg, 1512, 1514, 1516, and 1518) based on the various combinations of the recognized characters. Each recognition result includes a recognized character related to the first recognition unit and a recognized character related to the second recognition unit. As shown in FIG. 15C, the plurality of recognition results 1512, 1514, 1516, and 1518 include two recognition characters, respectively.

In this embodiment, the user actually intended that the handwritten input is recognized as a single character, but the handwritten character (for example, “
”) (For example, the left radical“
") And the right part (for example, the right radical"
)), An excessively large space is inadvertently left. Viewing the results presented in the candidate display area 806 (eg, 1512, 1514, 1516, and 1518), the user mistakenly splits the current handwriting input into two recognition units. Understand that. This division can be based on objective criteria, but for the user it is not possible to delete the current handwriting input and rewrite the entire character with a smaller distance between the left and right parts. , Not desirable.

  Instead, as shown in FIG. 15D, the user uses a pinch gesture on two clusters of handwriting strokes 1502 and 1510 to identify the two recognition units identified by their handwriting input module. Indicates to the handwriting input module that it should be merged as a single recognition unit. This pinch gesture is illustrated by two contacts 1520 and 1522 on the touch sensitive surface that move towards each other.

FIG. 15E shows that in response to the user's pinch gesture, the user device corrects a split of the currently stored handwriting input (eg, strokes 1502 and 1510) and converts those handwritten strokes into a single recognition unit. It shows that it is fused with. As shown in FIG. 15E, the user device provides an input image based on the corrected recognition unit to the handwriting recognition model, and three new candidate characters 1524, 1526, and 1528 (for example, “
, "
",as well as"
)). In some embodiments, as shown in FIG. 15E, the user device optionally renders the handwriting input in the handwriting input area 806, reducing the distance between the left and right clusters of handwriting strokes. Adjust so that. In some embodiments, the user device does not change the rendering of the handwriting input shown in the handwriting input area 608 in response to the pinch gesture. In some embodiments, the user device identifies a pinch gesture from the input stroke based on two simultaneous contacts (as opposed to a single contact) detected in the handwriting input area 806.

As shown in FIG. 15F, the user can enter the handwriting input (ie, the character “
Two more strokes 1530 are input to the right of “stroke”. The user device determines that the newly input stroke 1530 is a new recognition unit, and candidate characters (for example, “
)). The user device can then use the newly identified character (eg, “
”) With candidate characters for previously identified recognition units to present several different recognition results (eg, results 1532 and 1534) in candidate display area 806.

  Following the handwriting stroke 1530, the user continues to write additional strokes 1536 (eg, three additional strokes) to the right of the stroke 1530, as shown in FIG. 15G. Since the horizontal distance between the stroke 1530 and the stroke 1536 is very small, the user device determines that the stroke 1530 and the stroke 1536 belong to the same recognition unit, and is formed by the strokes 1530 and 1536 in the handwriting recognition model. Provide input image. The handwriting recognition model identifies three different candidate characters for the corrected recognition unit and generates two corrected recognition results 1538 and 1540 for the currently stored handwriting input.

In this example, the last two sets of strokes 1530 and 1536 are actually two separate characters (eg, “
”And“ ± ”). After the user sees that the user device mistakenly combines a set of two strokes 1530 and 1536 into a single recognition unit, the user continues to provide an enlargement gesture to the user device. Signals that the set of two strokes 1530 and 1536 should be separated into two separate recognition units. As shown in FIG. 15H, the user performs two contacts 1542 and 1544 around strokes 1530 and 1536, and then places those two contacts in a generally horizontal orientation (ie, default writing). Move in a direction away from each other).

  FIG. 15I shows that in response to this user's expanded gesture, the user device corrects the previous division of the handwritten input currently accumulated and assigns stroke 1530 and stroke 1536 to two consecutive recognition units. It shows that. Based on the input images generated for those two separate recognition units, the user device identifies one or more candidate characters for the first recognition unit based on stroke 1530 and sets the first number based on stroke 1536. One or more candidate characters are identified for two recognition units. The user device then generates two new recognition results 1546 and 1548 based on the various combinations of those recognized characters. In some embodiments, the user device optionally modifies the rendering of strokes 1536 and 1536 to reflect the previously identified recognition unit separation.

  As shown in FIGS. 15J and 15K, the user has selected and selected one of the candidate recognition results displayed in candidate display area 806 (as indicated by contact 1550). The recognition result (eg, result 1548) is entered in the text input area 808 of the user interface. After the selected recognition result is entered into the text input area 808, both the candidate display area 806 and the handwriting input area 804 are cleared and ready for displaying subsequent user input.

  FIGS. 16A and 16B inform the user how to divide or correct an existing division of the current handwriting input using a predefined gesture (eg, pinch gesture and / or magnification gesture). , Is a flow diagram of an exemplary process 1600. 15J and 15K provide a diagram of an exemplary process 1600, according to some embodiments.

  In some embodiments, the user device receives handwriting input from the user (1602). The handwriting input includes a plurality of handwriting strokes provided in a touch sensitive surface coupled to the device. In some embodiments, the user device renders (1604) those handwritten strokes in real time within the handwriting input area of the handwriting input interface (eg, the handwriting input area 806 of FIGS. 15A-15K). The user device receives one of the pinch gesture input and the enlarged gesture input on the plurality of handwritten strokes, for example, as shown in FIGS. 15D and 15H.

  In some embodiments, upon receiving the pinch gesture input, the user device may process their handwriting strokes as a single recognition unit, eg, as shown in FIGS. 15C-15E. A first recognition result based on a plurality of handwritten strokes is generated (1606).

  In some embodiments, upon receipt of the enlarged gesture input, the user device may, for example as shown in FIGS. 15G-15I, have multiple handwritten strokes separated into two separate recognition units separated by the enlarged gesture input. To generate a second recognition result based on the plurality of handwritten strokes (1608).

  In some embodiments, upon generating one of each of the first recognition result and the second recognition result, the user device may generate the generated recognition, eg, as shown in FIGS. 15E and 15I. The result is displayed in the candidate display area of the handwriting input interface.

  In some embodiments, the pinch gesture input includes two simultaneous contacts on the touch-sensitive surface that converge toward each other within an area occupied by multiple handwriting strokes. In some embodiments, the magnified gesture input includes two simultaneous contacts on the touch-sensitive surface that are dispersed from each other within an area occupied by a plurality of handwriting strokes.

  In some embodiments, the user device identifies two adjacent recognition units from these multiple handwritten strokes (eg, 1614). For example, as shown in FIG. 15C, the user device may include an initial recognition result (eg, result 1512 in FIG. 15C, including each character recognized from two adjacent recognition units in the candidate display area. 1514, 1516, and 1518) are displayed (1616). In some embodiments, in response to a pinch gesture, when displaying a first recognition result (eg, results 1524, 1526, or 1528 in FIG. 15E), the user device may have an initial display within the candidate display area. The recognition result is replaced with the first recognition result (1618). In some embodiments, the user device receives a pinch gesture input while an initial recognition result is displayed in the candidate display area, as shown in FIG. 15D (1620). In some embodiments, in response to this pinch gesture input, the user device may reduce the distance between two adjacent recognition units in the handwriting input area, eg, as shown in FIG. 15E. A plurality of handwritten strokes are re-rendered (1622).

In some embodiments, the user device identifies a single recognition unit from these multiple handwritten strokes (1624). The user device may recognize characters (e.g., “
""
”), The initial recognition results (eg, results 1538 or 1540 of FIG. 15G) are displayed (1626). In some embodiments, if a second recognition result (eg, result 1546 or 1548 in FIG. 15I) is displayed in response to the magnification gesture, the user device may be as shown in, eg, FIG. 15H, FIG. 15I. In the candidate display area, the initial recognition result (eg, result 1538 or 1540) is replaced with the second recognition result (eg, result 1546 or 1548) (1628). In some embodiments, the user device receives an expanded gesture input (1630) while the initial recognition result is displayed in the candidate display area, as shown in FIG. 15H. In some embodiments, in response to this expanded gesture input, the user device may use a first stroke assigned to a first recognition unit in the handwriting input area, as shown in FIGS. 15H and 15I. The plurality of handwriting strokes are re-rendered (1632) to increase the distance between the subset of the second handwriting strokes assigned to the second recognition unit.

  In some embodiments, immediately after the user provides strokes and understands that those strokes may be overly spread over the correct split based on the standard split process, The user optionally provides a pinch gesture to notify the user device to treat those multiple strokes as a single recognition unit. The user device can distinguish the pinch gesture from the regular stroke based on two simultaneous contacts present in the pinch gesture. Similarly, in some embodiments, the user has provided strokes and understands that those strokes may be overly crowded for correct splitting based on a standard splitting process. Immediately thereafter, the user optionally provides an enlarged gesture to inform the user device to treat those multiple strokes as two separate recognition units. The user device can identify the enlargement gesture from the regular stroke based on two simultaneous contacts present in the pinch gesture.

In some embodiments, the direction of the motion of the pinch or magnification gesture is optionally used to provide further guidance on how to split the stroke under the gesture. For example, with respect to a handwriting input area, if multiple lines of handwriting input are valid, a pinch gesture with two contacts that move in the vertical direction is sent to the handwriting input module by the two specified in two adjacent lines. The recognition units can be notified to be merged into a single recognition unit (eg, as an upper radical and a lower radical). Similarly, an enlarged gesture with two contacts moving in the vertical direction can inform the handwriting input module to split a single recognition unit into two recognition units in two adjacent rows. . In some embodiments, pinch gestures and magnification gestures can also be used as a splitting guide in sub-parts of character input, such as fusing two sub-components, or compound characters (
Etc.) can also be provided for separating a single component within the various characters of the compound character (eg, the top, bottom, left, or right portion). This is particularly useful for recognizing complex compound Chinese characters as the user tends to lose the correct ratio and balance when writing complex compound characters by hand. After completion of handwriting input, the ability to adjust the ratio and balance of handwriting input, for example by pinch gestures and magnified gestures, allows the user to make several attempts to reach the correct ratio and balance Especially useful for typing the correct characters without having to.

  As described herein, the handwriting input module allows a user to input multi-character handwriting input, within a single character within a handwriting input area, across multiple characters, and even more. Allows out-of-order strokes for handwriting input of multiple characters across multiple words, sentences, and / or lines. In some embodiments, the handwriting input module also provides character-by-character deletion within the handwriting input area, the order of character deletion being the reverse direction of writing, and when the stroke for each character is It is unrelated to whether it was provided in the handwriting input area. In some embodiments, deletion of each recognition unit (eg, character or radical) within the handwriting input area is optionally performed on a stroke-by-stroke basis, and the strokes are within the recognition unit. Deleted in reverse chronological order provided. 17A-17H illustrate an exemplary user interface for providing character-by-character deletion within a multi-character handwriting input in response to a deletion input from the user.

As shown in FIG. 17A, the user provides a plurality of handwriting strokes 1702 in the handwriting input area 804 of the handwriting input user interface 802. Based on these currently accumulated strokes 1702, the user device presents three recognition results (eg, results 1704, 1706, and 1708) in the candidate display area 806. As shown in FIG. 17B, the user is providing additional strokes 1710 in the handwriting input area 806. The user device recognizes three new output characters and replaces the three previous recognition results 1704, 1706, and 1708 with three new recognition results 1712, 1714, and 1716. In some embodiments, as shown in FIG. 17B, even if the user device has identified two separate recognition units from the current handwriting input (eg, stroke 1702 and stroke 1710). The cluster of strokes 1710 does not correspond well with any known characters in the repertoire of handwriting recognition modules. As a result, the candidate characters identified for the recognition unit that includes the stroke 1710 (eg, “
, "
') All have recognition confidence below a predetermined threshold. In some embodiments, the user device may be a candidate character for the first recognition unit (eg, “
') Only, but does not include any candidate characters for the second recognition unit, a partial recognition result (eg, result 1712) is presented in the candidate display area 806. In some embodiments, the user device includes a full recognition result (eg, results 1714 or 1716) that includes candidate characters for both recognition units regardless of whether the recognition confidence has passed a predetermined threshold. Is further displayed. Providing the partial recognition result informs the user which part of the handwritten input needs correction. In addition, the user can also initially choose to enter the correctly recognized portion of the handwriting input and then rewrite the portion that was not correctly recognized.

FIG. 17C shows that the user continues to provide additional handwriting strokes 1718 to the left of stroke 1710. Based on the relative location and distance of stroke 1718, the user device determines that this newly added stroke belongs to the same recognition unit as the cluster of handwritten strokes 1702. Based on this corrected recognition unit, a new character (e.g., “
)) Is recognized, and a set of new recognition results 1720, 1722, and 1724 is generated. Again, the first recognition result 1720 is a partial recognition result because none of the candidate characters identified for the stroke 1710 meets the predetermined confidence threshold.

FIG. 17D shows that the user has entered multiple new strokes 1726 between strokes 1702 and 1710 at this point. The user device assigns this newly input stroke 1726 to the same recognition unit as the stroke 1710. At this point, the user has two Chinese characters (for example, “
)) Has been entered and the correct recognition result 1728 is shown in the candidate display area 806.

  FIG. 27E shows that the user is entering the initial portion of the delete input, for example, by performing a light contact 1730 on the delete button 1732. When the user maintains contact with the delete button 1732, the user can delete the current handwritten input character in character units (or in recognition units). This deletion is not executed simultaneously for all handwriting inputs.

In some embodiments, when the user's finger first touches the delete button 1732 on the touch-sensitive screen, as shown in FIG. 17E, in the default (eg, left to right) writing direction. The last recognition unit (eg the character "
Are visually highlighted with respect to other recognition units that are simultaneously displayed in the handwriting input area 804 (e.g., highlighted with a border 1734 or a light background).

  In some embodiments, when the user device detects that the user has maintained contact 1730 on the delete button 1732 beyond a threshold duration, the user device may be as shown in FIG. 17F. The highlighted recognition unit (eg, in box 1734) is removed from handwriting input area 806. In addition, the user device also has a recognition result shown in the candidate display area 608 to delete any output characters generated based on the deleted recognition unit, as shown in FIG. 17F. correct.

FIG. 17F shows the last recognition unit (for example, the character “
When the user continues to maintain the contact 1730 on the delete button 1732 after the “recognition unit” is deleted, the recognition unit adjacent to the deleted recognition unit (for example, the character “
It is further shown that the “recognition unit” is the next recognition unit to be deleted. As shown in FIG. 17F, this remaining recognition unit is visually highlighted (eg, in box 1736) and is ready to be deleted. In some embodiments, the visual highlighting of the recognition unit provides a preview of the recognition unit that will be deleted if the user continues to maintain contact with the delete button. If the user interrupts contact with the delete button before the threshold duration is reached, the visual highlight is removed from the last recognition unit and the recognition unit is not deleted. As will be appreciated by those skilled in the art, the duration of contact is reset each time the recognition unit is deleted. Further, in some embodiments, the intensity of contact (eg, the pressure that the user is applying to contact 1730 with the touch sensitive screen) is optionally used to determine the currently highlighted recognition unit. Adjust the threshold duration for confirming the intention of the user to be deleted. 17F and 17G show that before the threshold duration is reached, the user has interrupted contact 1730 with the delete button 1732 and the characters “
"Indicates that the recognition unit is stored in the handwriting input area 806. When the user selects a first recognition result (eg, result 1738) for this recognition unit (eg, as indicated by contact 1740), the first recognition as shown in FIGS. 17G and 17H. The text in result 1738 is entered into text entry area 808.

  18A and 18B are a flow diagram of an example process 1800 in which a user device provides character-by-character deletion within a multi-character handwriting input. In some embodiments, the deletion of the handwriting input is performed before characters recognized from the handwriting input are identified and entered into the text input area of the user interface. In some embodiments, deletion of characters in the handwriting input proceeds according to the reverse spatial order of the recognition units identified from the handwriting input and is independent of the temporal arrangement in which those recognition units are formed. It is. 17A-17H illustrate an exemplary process 1800, according to some embodiments.

As shown in FIG. 18A, in an exemplary process 1800, the user device receives handwriting input from a user (1802), and the handwriting input is a handwriting input area (eg, area 804 in FIG. 17D) of the handwriting input interface. ) Includes a plurality of handwritten strokes provided within. The user device identifies a plurality of recognition units from the plurality of handwritten strokes (1804), and each recognition unit includes a corresponding subset of the plurality of handwritten strokes. For example, as shown in FIG. 17D, the first recognition unit includes strokes 1702 and 1718, and the second recognition unit includes strokes 1710 and 1726. The user device generates a multi-character recognition result (eg, result 1728 in FIG. 17D) that includes each character recognized from the plurality of recognition units (1806). In some embodiments, the user device displays the multi-character recognition result (eg, result 1728 in FIG. 17D) within the candidate display area of the handwriting input interface. In some embodiments, while the multi-character recognition result is displayed in the candidate display area, the user device may receive a delete input from the user (eg, a delete button, eg, as shown in FIG. 17E). Contact 1730) on 1732 is received (1810). In some embodiments, in response to receiving this deletion input, the user device may display a plurality of displayed in a candidate display area (eg, candidate display area 806), for example, as shown in FIGS. 17E, 17F. From the character recognition result (eg, result 1728), the last character (eg, spatial array “
"At the end of"
Is removed (1812).

In some embodiments, the user device may include a plurality of the handwriting input areas within the handwriting input area of the handwriting input interface when a plurality of handwriting strokes are provided by the user, for example, as shown in FIGS. 17A-17D. Is rendered in real time (1814). In some embodiments, in response to receiving the deletion input, the user device is formed by a plurality of recognition units in the handwriting input area from the handwriting input area (eg, handwriting input area 804 in FIG. 17E). The corresponding subset of the plurality of handwritten strokes corresponding to the last recognition unit (eg, the recognition unit including strokes 1726 and 1710) in the spatial array is removed (1816). This last recognition unit is the last character (for example, the character “for example” in the result 1728 in FIG. 17E).
)).

  In some embodiments, this last recognition unit does not include the last handwritten stroke in time among the plurality of handwritten strokes provided by the user (1818). For example, if the user is providing stroke 1718 after providing strokes 1726 and 1710, the last recognition unit that includes strokes 1726 and 1710 is still deleted first.

  In some embodiments, in response to receiving the initial portion of the deletion input, the user device may determine the last recognition unit as another recognition unit identified in the handwriting input area, eg, as shown in FIG. 17E. Are visually identified (1820). In some embodiments, the initial portion of the delete input is an initial touch detected on the delete button in the handwriting input interface (1822), and the initial touch exceeds a predetermined threshold amount of time. A delete input is detected if persisted.

  In some embodiments, the last recognition unit corresponds to handwritten Chinese characters. In some embodiments, handwriting input is written in a cursive typeface. In some embodiments, the handwriting input corresponds to a plurality of Chinese characters written in a cursive typeface. In some embodiments, at least one of the handwritten strokes is separated into two adjacent recognition units of the plurality of recognition units. For example, in some cases, the user can use long strokes that are continuously written into multiple characters, in which case the split module of the handwriting input module can optionally Separate the stroke into several recognition units. When deletion of handwriting input is performed in units of characters (or for each recognition unit), only this long stroke segment (eg, segment in the corresponding recognition unit) is deleted at a time.

  In some embodiments, the delete input is a persistent contact on a delete button provided within the handwriting input interface (1824), and removing the corresponding subset of the plurality of handwritten strokes Further including removing the subset of handwritten strokes within the recognition unit from the handwriting input area in a reverse temporal order provided by the user in a stroke unit.

  In some embodiments, the user device generates a partial recognition result including a subset of each character recognized from the plurality of recognition units, eg, as shown in FIGS. 17B and 17C (1826). , Each of the subsets of characters meet a predetermined confidence threshold. In some embodiments, the user device may display a partial recognition result (eg, result 1712 in FIG. 17B and result 1720 in FIG. 17C) within the candidate display area of the handwriting input interface, and a multi-character recognition result. (Eg, results 1714 and 1722) are displayed simultaneously (1828).

  In some embodiments, the partial recognition result does not include at least the last character in the multi-character recognition result. In some embodiments, the partial recognition result does not include at least the first character in the multi-character recognition result. In some embodiments, the partial recognition results do not include at least intermediate characters in the multi-character recognition results.

  In some embodiments, the smallest unit of deletion is a radical, and handwriting input is always the last recognition unit in handwriting input that happens to remain in the handwriting input area by chance, One radical is deleted at a time.

  As described herein, in some embodiments, the user device provides both a horizontal write mode and a vertical write mode. In some embodiments, the user device allows a user to enter text in a horizontal writing mode in one or both of a left-to-right writing direction and a right-to-left direction. . In some embodiments, the user device allows the user to enter text in a vertical writing mode in one or both of a top-to-bottom writing direction and a bottom-to-top direction. . In some embodiments, the user device may display various affordances (eg, write mode or write direction button) on the user interface to invoke the respective write mode and / or write direction for the current handwriting input. provide. In some embodiments, the text input direction within the text input area is by default the same as the handwriting input direction in the handwriting input direction. In some embodiments, the user device allows the user to manually set the input direction within the text input area and the writing direction within the handwriting input area. In some embodiments, the text display direction in the candidate display area is by default the same as the handwriting input direction in the handwriting input area. In some embodiments, the user device allows the user to manually set the text display direction within the text input area regardless of the handwriting input direction within the handwriting input area. In some embodiments, the user device associates the writing mode and / or writing direction of the handwriting input interface with the corresponding device orientation, and changing the orientation of the device causes a change in the writing mode and / or writing direction. Triggered. In some embodiments, a change in the writing direction automatically inputs the highest rank recognition result into the text input area.

  19A-19F illustrate an exemplary user interface of a user device that provides both a horizontal input mode and a vertical input mode.

  FIG. 19A shows a user device in horizontal input mode. In some embodiments, the horizontal input mode is provided when the user device is in a landscape orientation, as shown in FIG. 19A. In some embodiments, a horizontal input mode is optionally associated and provided when the device is operated in a portrait orientation. In different applications, the relevance between device orientation and write mode may be different.

  In the horizontal input mode, the user can provide handwritten characters in a horizontal writing direction (eg, a default writing direction that proceeds from left to right or a default writing direction that proceeds from right to left). In the horizontal input mode, the user device performs division of handwritten input into one or more recognition units along the horizontal writing direction.

  In some embodiments, the user device only allows a single line input within the handwriting input area. In some embodiments, as shown in FIG. 19A, the user device allows multiple lines of input (eg, two lines of input) within the handwriting input area. In FIG. 19A, the user provides a plurality of handwritten strokes in several lines in the handwriting input area 806. Based on the arrangement of a plurality of handwritten strokes provided by the user, the relative location of the plurality of handwritten strokes, and the distance between the strokes, the user device determines that the user is inputting two lines of characters. To do. After splitting this handwritten input into two separate lines, the device determines the recognition units in each line.

  As shown in FIG. 19A, the user device recognizes each character for each recognition unit specified in the current handwriting input 1902 and generates several recognition results 1904 and 1906. As further shown in FIG. 19A, in some embodiments, the output character (eg, the letter “I”) for a particular set of recognition units (eg, the recognition units formed by the initial stroke) is low. The user device optionally generates a partial recognition result (eg, result 1906) that shows only output characters with sufficient recognition confidence. In some embodiments, the user understands from this partial recognition result 1906 that the first stroke can be corrected or deleted individually in order for the recognition model to produce the correct recognition result. Can do. In this particular embodiment, the first recognition unit 1904 indicates the desired recognition result for the first recognition unit, so editing the first recognition unit is not necessary.

  In this example, as shown in FIGS. 19A and 19B, the user has rotated the device in a portrait orientation (eg, as shown in FIG. 19B). In response to the change in the orientation of the device, the handwriting input interface is changed from the horizontal input mode to the vertical input mode as shown in FIG. 19B. In the vertical input mode, the layout of the handwriting input area 804, the candidate display area 806, and the text input area 808 may be different from that shown in the horizontal input mode. The specific layout of the horizontal input mode and vertical input mode can be varied to suit various device shapes and application requirements. In some embodiments, rotation of the device orientation and change of input mode may cause the user device to automatically receive the highest ranked result (eg, result 1904) as text input 1910 in text input area 808. input. The orientation and position of the cursor 1912 also reflects changes in the input mode and writing direction.

  In some embodiments, the input mode change is optionally triggered by a user touching a special input mode selection affordance 1908. In some embodiments, this input mode selection affordance is a graphical user interface element that also indicates the current writing mode, the current writing direction, and / or the current paragraph direction. In some embodiments, the input mode selection affordance can cycle through all available input modes and writing directions provided by the handwriting input interface 802. As shown in FIG. 19A, affordance 1908 indicates that the current input mode is a horizontal input mode with a writing direction from left to right and a paragraph direction from top to bottom. In FIG. 19B, affordance 1908 indicates that the current input mode is a vertical input mode with a writing direction from top to bottom and a paragraph direction from right to left. Other combinations of writing direction and paragraph direction are possible according to various embodiments.

As shown in FIG. 19C, in vertical input mode, the user enters a plurality of new strokes 1914 (eg, two Chinese characters “
"Handwritten stroke)". This handwritten input is written in a vertical writing direction. The user device divides the handwritten input in the vertical direction into two recognition units, and displays two recognition results 1916 and 1918 each including two recognition characters laid out in the vertical direction.

  19C and 19D illustrate that when the user selects a displayed recognition result (eg, result 1916), the selected recognition result is entered in the text input area 808 vertically.

  19E and 19F show that the user is inputting additional handwriting input 1920 rows in a vertical writing direction. These lines go from left to right according to the paragraph direction of traditional Chinese writing. In some embodiments, the candidate display area 806 also shows recognition results (eg, results 1922 and 1924) in the same writing direction and paragraph direction as those for the handwriting input area. In some embodiments, other writing directions and according to the primary language associated with the user device or the language of the soft keyboard installed on the user device (eg, Arabic, Chinese, Japanese, English, etc.) Paragraph direction can be provided by default.

  19E and 19F also illustrate that when the user selects a recognition result (eg, result 1922), the text of the selected recognition result is entered into the text input area 808. As shown in FIG. 19F, the text currently entered in the text input area 808 is therefore text written in horizontal mode and writing from top to bottom with a writing direction from left to right. Includes both text written in vertical mode with direction. The paragraph direction for horizontal text is from top to bottom, while the paragraph direction for vertical text is from right to left.

  In some embodiments, the user device allows the user to establish a preferred writing direction, paragraph direction separately for each of the handwriting input area 804, candidate display area 806, and text input area 808. . In some embodiments, the user device establishes a preferred writing direction and paragraph direction for each of the handwriting input area 804, candidate display area 806, and text input area 808 that is associated with the orientation of the respective device. Make it possible to do.

  20A-20C are a flow diagram of an example process 2000 for changing the text input direction and handwriting input direction of a user interface. 19A-19F illustrate a process 2000, according to some embodiments.

  In some embodiments, the user device determines the orientation of the device (2002). Device orientation and changes in device orientation can be detected by accelerometers and / or other orientation sensing elements within the user device. In some embodiments, the user device provides a handwriting input interface in horizontal input mode on the device according to the device being in the first orientation (2004). Each line of handwritten input input in this horizontal input mode is divided into one or more respective recognition units along the horizontal writing direction. In some embodiments, the device provides a handwriting input interface in vertical input mode on the device according to the device being in the second orientation (2006). Each line of handwritten input input in this vertical input mode is divided into one or more respective recognition units along the vertical writing direction.

  In some embodiments, while operating in horizontal input mode (2008), the device detects a change in device orientation from a first orientation to a second orientation (2010). In some embodiments, in response to a change in the orientation of the device, the device switches from a horizontal input mode to a vertical input mode (2012). This is illustrated, for example, in FIGS. 19A and 19B. In some embodiments, while operating in the vertical input mode (2014), the user device detects a change in device orientation from the second orientation to the first orientation (2016). In some embodiments, in response to a change in the orientation of the device, the user device switches from a vertical input mode to a horizontal input mode (2018). In some embodiments, the relationship between device orientation and input mode can be reversed from that described above.

  In some embodiments, while operating in horizontal input mode (2020), the user device receives a first multi-word handwriting input from the user (2022). In response to the handwriting input of the first plurality of words, the user device presents the recognition result of the first plurality of words according to the horizontal writing direction in the candidate display area of the handwriting input interface (2024). This is illustrated, for example, in FIG. 19A. In some embodiments, while operating in vertical input mode (2026), the user device receives a second multi-word handwriting input from the user (2028). In response to the handwriting input of the second plurality of words, the user device presents the recognition result of the second plurality of words in the candidate display area according to the vertical writing direction (2030). This is illustrated, for example, in FIGS. 19C and 19E.

  In some embodiments, the user device receives a first user input (2032) that selects a first multi-word recognition result, eg, as shown in FIGS. 19A and 19B, and this selection. Is implicitly implemented using an input to change the input direction (eg, device rotation or affordance 1908 selection). The user device receives a second user input (2034) that selects a second multi-word recognition result, for example, as shown in FIG. 19C or FIG. 19E. The user device simultaneously displays the corresponding texts of the recognition result of the first plurality of words and the recognition result of the second plurality of words in the text input area of the handwriting input interface (2036). The text corresponding to the recognition result is displayed according to the horizontal writing direction, and the text corresponding to the recognition result of the second plurality of words is displayed according to the vertical writing direction. This is shown, for example, in the text input area 808 in FIG. 19F.

  In some embodiments, the handwriting input area accepts multiple lines of handwriting input in a horizontal writing direction and has a default top-to-down paragraph direction. In some embodiments, this horizontal writing direction is from left to right. In some embodiments, this horizontal writing direction is from right to left. In some embodiments, the handwriting input area accepts multiple lines of handwriting input in a vertical writing direction and has a default left-to-right paragraph direction. In some embodiments, the handwriting input area accepts multiple lines of handwriting input in a vertical writing direction and has a default right-to-left paragraph direction. In some embodiments, this vertical writing direction is from top to bottom. In some embodiments, the first orientation defaults to landscape orientation and the second orientation defaults to portrait orientation. In some embodiments, the user device provides a corresponding affordance in the handwriting input interface to manually switch between horizontal input mode and vertical input mode regardless of device orientation. In some embodiments, the user device provides a corresponding affordance in the handwriting input interface to manually switch between the two alternative writing directions. In some embodiments, the user device provides a corresponding affordance in the handwriting input interface to manually switch between the two alternative paragraph directions. In some embodiments, this affordance is a toggle button that rotates around each available combination of input and paragraph directions when called more than once in succession.

  In some embodiments, the user device receives handwriting input 2038 from the user. The handwriting input includes a plurality of handwriting strokes provided in a handwriting input area of the handwriting input interface. In response to the handwriting input, the user device displays one or more recognition results in the candidate display area of the handwriting input interface (2040). While the one or more recognition results are displayed in the candidate display area, the user device detects user input to switch from the current handwriting input mode to an alternative handwriting input mode (2042). . In response to this user input (2044), the user device switches from the current handwriting input mode to an alternative handwriting input mode (2046). In some embodiments, the user device erases the handwriting input from within the handwriting input area (2048). In some embodiments, the user device automatically inputs the highest ranking recognition result among the one or more recognition results displayed in the candidate display area in the text input area of the handwriting input interface. (2050). This is illustrated, for example, in FIGS. 19A and 19B, where the current handwriting input mode is the horizontal input mode and the alternative handwriting input mode is the vertical input mode. In some embodiments, the current handwriting input mode is a vertical input mode and the alternative handwriting input mode is a horizontal input mode. In some embodiments, the current handwriting input mode and the alternative handwriting input mode are modes in which any two different handwriting input directions or paragraph directions are provided. In some embodiments, the user input is a rotation of the device from the current orientation to a different orientation (2052). In some embodiments, this user input is an affordance call to manually switch the current handwriting input mode to an alternative handwriting input mode.

  As described herein, the handwriting input module allows a user to input handwritten strokes and / or characters in any temporal order. Therefore, deleting an individual handwritten character in a handwritten input of a plurality of characters and rewriting the same handwritten character or a different handwritten character in the same place as the deleted character This is advantageous because it helps to correct long handwriting input without the need to delete.

  20A-20H are exemplary users for visually highlighting and / or deleting recognition units identified in a plurality of handwriting strokes currently accumulated in a handwriting input area. Indicates the interface. Allowing a user to individually select, view and delete any one of a plurality of recognition units identified in a plurality of inputs is a multi-character handwriting input, or even This is particularly useful when multiple lines of handwriting input are allowed by the user device. By allowing the user to delete a specific recognition unit at the beginning or middle of handwriting input, without requiring the user to delete all recognition units located after the unwanted recognition unit The user can make corrections for long inputs.

  As shown in FIGS. 21A-21C, the user is providing a plurality of handwritten strokes (eg, strokes 2102, 2104, and 2106) within the handwriting input area 804 of the handwriting input user interface 802. While the user continues to provide additional strokes to the handwriting input area 804, the user device updates and updates the recognition units identified from the handwriting input currently stored in the handwriting input area. The recognition result is corrected according to the output character recognized from the recognized recognition unit. As shown in FIG. 20C, the user device identifies two recognition units from the current handwriting input and presents three recognition results (eg, 2108, 2010, and 2112) each containing two Chinese characters. doing.

  In this embodiment, after the user has written two handwritten characters, the user incorrectly writes the first recognition unit, and as a result, the user device identifies the desired recognition result within the candidate display area. Understand what is not presented.

  In some embodiments, when a user provides a tap gesture on a touch-sensitive display (eg, an immediate lift-off at the same location following a touch), the user device may place the tap gesture within the handwriting input area. Interpret as input to cause visual highlighting of individual recognition units currently identified. In some embodiments, another predefined gesture (e.g., multiple finger wipe gesture over the handwriting input area) is used to allow the user device to recognize individual recognition units in the handwriting input area 804. Highlight. Tap gestures may be preferred because they are relatively easy to distinguish from handwriting strokes (usually with longer duration continuous contact and having contact movement within the handwriting input area 804). Multi-tap gestures may be preferred because they are relatively easy to distinguish from handwriting strokes (usually with a single contact in handwriting input area 804). In some embodiments, the user device is by the user (eg, via contact 2114) to visually highlight individual recognition units (eg, as indicated by boxes 2108 and 2110). An affordance 2112 that can be invoked is provided in the user interface. In some embodiments, this affordance is preferred when there is sufficient screen space to accommodate such affordance. In some embodiments, this affordance can be called continuously multiple times by the user, which visually enhances the user device with recognition units identified according to different split chains in the split bundle. Display and turn off when all split chains are shown with respect to the highlighting.

  As shown in FIG. 21D, if the user has provided the necessary gestures to highlight individual recognition units in the handwriting input area 804, the user device will Above, each delete affordance (eg, small delete buttons 2116 and 2118) is further displayed. FIGS. 21E and 21F show that when a user touches (eg, via contact 2120) a deletion affordance for each recognition unit (eg, delete button 2116 for the first recognition unit in box 2118). Indicates that the recognition unit (eg, in box 2118) is removed from handwriting input area 804. In this particular embodiment, the recognition unit to be deleted is not the last recognition unit entered temporally, nor is it the last recognition unit spatially along the writing direction. In other words, the user can delete any recognition unit regardless of the time and place where the recognition unit was provided in the handwriting input area. FIG. 21F shows that the user device also updates the recognition result displayed in the candidate display area 806 in response to the deletion of this first recognition unit in the handwriting input area. As shown in FIG. 21F, the user device also deletes the candidate character corresponding to the deleted recognition unit from the recognition result. As a result, a new recognition result 2120 is shown in the candidate display area 806.

  As shown in FIGS. 21G and 21H, after the first recognition unit is removed from the handwriting input interface 804, the user can enter a plurality of areas within the area previously occupied by the deleted recognition unit. A new handwriting stroke 2122 is provided. The user device subdivides the handwritten input currently stored in the handwritten input area 804. Based on the recognition unit identified from the handwriting input, the user device regenerates the recognition results (eg, results 2124 and 2126) in the candidate display area 806. 21G and 21H show that the user has selected one of their recognition results (eg, result 2124) (eg, via contact 2128) and the text of the selected recognition result is Indicates that the text is input in the text input area 808.

  22A and 22B show that individual recognition units identified in the current handwriting input are visually presented and can be deleted individually regardless of the temporal order in which those recognition units are formed. 3 is a flow diagram for an exemplary process 2200. 21A-21H illustrate a process 2200, according to some embodiments.

  In the example process 2200, the user device receives handwriting input from the user (2202). The handwriting input includes a plurality of handwriting strokes provided on a touch sensitive surface coupled to the device. In some embodiments, the user device renders (2204) those multiple handwritten strokes within a handwriting input area (eg, handwriting input area 804) of the handwriting input interface. In some embodiments, the user device divides the plurality of handwritten strokes into two or more recognition units (2206), each recognition unit being a corresponding subset of the plurality of handwriting strokes. including.

  In some embodiments, the user device receives an edit request from the user (2208). In some embodiments, the edit request is a contact 2210 detected on a predetermined affordance (eg, affordance 2112 in FIG. 21D) provided in the handwriting input interface. In some embodiments, the edit request is a tap gesture that is detected over a predetermined area in the handwriting input interface (2212). In some embodiments, this default area exists within the handwriting input area of the handwriting input interface. In some embodiments, this default area is outside the handwriting input area of the handwriting input interface. In some embodiments, another predefined gesture outside the handwriting input area (eg, cross gesture, horizontal swipe gesture, vertical swipe gesture, tilt swipe gesture) may be used as an edit request. Gestures outside the handwriting input area are provided outside the handwriting input area so that they can be easily identified from the handwriting stroke.

  In some embodiments, in response to this edit request, the user device visually identifies two or more recognition units in the handwriting input area using, for example, boxes 2108 and 2110 in FIG. 21D. (2214). In some embodiments, visually identifying the two or more recognition units further includes highlighting respective boundaries between the two or more recognition units in the handwriting input area (2216). ). In various embodiments, various methods can be used that visually identify the recognition units identified in the current handwriting input.

  In some embodiments, the user device provides a means for individually deleting each of the two or more recognition units from the handwriting input area (2218). In some embodiments, the means for individually deleting each of the two or more recognition units is proximate to each recognition unit, eg, as indicated by delete buttons 2116 and 2118 in FIG. 21D. Each delete button is displayed. In some embodiments, the means for individually deleting each of the two or more recognition units is a means for detecting a default delete gesture input on each recognition unit. In some embodiments, the user device does not visually display individual deletion affordances on the highlighted recognition unit. Instead, in some embodiments, a user can use a delete gesture to delete each recognition unit under the delete gesture. In some embodiments, when the user device is displaying the recognition unit in a visually highlighted manner, the user device does not accept further handwriting strokes within the handwriting input area. Instead, a default gesture or any gesture that is detected on a visually highlighted recognition unit causes the user device to remove the recognition unit from the handwriting input area and display it in the candidate display area. The recognition result is corrected appropriately. In some embodiments, the tap gesture causes the user device to visually highlight individual recognition units identified in the handwriting recognition area, and then the user uses the delete button to write In the reverse direction, individual recognition units can be deleted one by one.

  In some embodiments, the user device may provide a deletion input to individually delete the first recognition unit of the two or more recognition units from the handwriting input area, for example, as shown in FIG. 21E. Received from the user through the provided means (2224). In response to this delete input, the user device removes a corresponding subset of the handwritten strokes in the first recognition unit from the handwriting input area, for example as shown in FIG. 21F (2226). In some embodiments, the first recognition unit is a spatially first recognition unit in two or more recognition units. In some embodiments, the first recognition unit is a spatially intermediate recognition unit among two or more recognition units, eg, as shown in FIGS. 21E and 21F. In some embodiments, the first recognition unit is the spatially last recognition unit of the two or more recognition units.

In some embodiments, the user device generates a split bundle from a plurality of handwritten strokes (2228), wherein the split bundle includes a set of corresponding recognition units, each identified from the plurality of handwritten strokes. Representing multiple alternative split chains. For example, FIG. 21G shows recognition results 2024 and 2026, where recognition result 2024 is generated from one split chain having two recognition units, and recognition result 2026 is generated from another split chain having three recognition units. Is done. In some embodiments, the user device receives (2230) two or more continuous edit requests from the user. For example, those two or more consecutive edit requests can be several consecutive taps on affordance 2112 in FIG. 21G. In some embodiments, in response to each of the two or more successive edit requests, the user device may recognize corresponding correspondences from different ones of the plurality of alternative split chains within the handwriting input area. A set of units is visually identified (2232). For example, in response to the first tap gesture, within the handwriting input area 804, two (eg, each of the characters “
"as well as"
Recognition units are highlighted and in response to a second tap gesture, three (eg, each of the characters “
, "
",as well as"
The recognition unit is highlighted. In some embodiments, in response to the third tap gesture, visual highlighting is optionally removed from all recognition units, and the handwriting input area returns to normal and further strokes are made. Ready to accept. In some embodiments, the user device provides a means for individually deleting each of the corresponding set of recognition units currently represented in the handwriting input area (2234). In some embodiments, this means is a separate delete button for each highlighted recognition unit. In some embodiments, the means detects a default delete gesture on each highlighted recognition unit and deletes the highlighted recognition unit under the default delete gesture. It is a means to call the function to do.

  As described herein, in some embodiments, the user device provides a continuous input mode within the handwriting input area. Since the area of the handwriting input area is limited on portable user devices, the handwriting input provided by the user is cached, and the user can save screen space without determining the handwriting input provided previously. It may be desirable to provide a method for allowing reuse. In some embodiments, the user device may gradually transition the input area by a certain amount (eg, one recognition unit at a time) when the user is sufficiently close to the end of the handwriting input area. Provide a scrolling handwriting input area. In some embodiments, recognizing recognition units dynamically because migrating existing recognition units within a handwriting input area may interfere with the user's writing process and possibly prevent correct division of recognition units. It may be advantageous to reclaim the previously used area of the input area without shifting. In some embodiments, when a user reuses an area occupied by handwriting input that has not yet been entered in the text input area, the top-level recognition result for the handwriting input area is displayed in the text input area. Automatically entered, the user can continue to provide new handwriting input without explicitly selecting the highest rank recognition result.

  Some conventional systems allow the user to write over existing handwriting input that is still shown in the handwriting input area. In such a system, temporal information is used to determine whether the new stroke is part of the previous recognition unit or part of the new recognition unit. Such a system that relies on temporal information imposes strict requirements on the speed and speed with which a user provides handwriting input, and many users have difficulty meeting that requirement. . Furthermore, the visual rendering of the handwritten input can be cluttered and difficult for the user to decipher. Therefore, the writing process can be frustrating and confusing for the user, resulting in a poor user experience.

  To indicate when a user can reuse an area occupied by a previously written recognition unit and continue writing in that handwritten input area, as described herein The fading process is used. In some embodiments, this fading process gradually reduces the visibility of each recognition unit provided in the handwriting input area over a threshold amount of time and a new stroke is written on top of it. In addition, existing text does not visually conflict with the new stroke. In some embodiments, writing on a faded recognition unit does not require the user to interrupt the write and explicitly provide a selection input for the highest rank recognition result. The recognition result of the highest rank is automatically entered in the text input area. This implicit and automatic confirmation of the top-ranked recognition results improves the input efficiency and speed of the handwriting input interface and imposes a cognitive burden imposed on the user to maintain the current text composition thinking flow Reduce. In some embodiments, writing over a faded recognition unit does not cause automatic selection of the highest rank search results. Instead, the faded recognition units are cached in the handwriting input stack and combined with the new handwriting input as the current handwriting input. The user can visually recognize the recognition result generated based on all of the recognition units accumulated in the handwriting input stack before performing the selection.

  FIGS. 23A-23J show that recognition units provided in different regions of a handwriting input area gradually fade out of their corresponding regions after, for example, a predetermined amount of time, and this fade-out occurs within a particular region. Once generated, the user presents an exemplary user interface and process that can provide new handwritten strokes within the region.

  As shown in FIG. 23A, the user has provided a plurality of handwriting strokes 2302 (eg, three handwriting strokes for capital letter “I”) in handwriting input area 804. The handwritten stroke 2302 is specified as one recognition unit by the user device. In some embodiments, the handwriting input currently shown in handwriting input area 804 is cached in a first layer in the handwriting input stack of the user device. Several recognition results generated based on this identified recognition unit are provided in the candidate display area 804.

  FIG. 23B shows that when the user continues to write one or more strokes 2302 to the right of the stroke 2304, the handwriting stroke 2302 in the first recognition unit gradually begins to fade out in the handwriting input area 804. Indicates to do. In some embodiments, an animation is displayed that mimics the gradual fading or disappearance of the visual rendering of this first recognition unit. For example, this animation can create a visual effect where ink evaporates from the whiteboard. In some embodiments, the fading of recognition units is not uniform across the recognition units. In some embodiments, the fading of the recognition unit increases over time, and eventually the recognition unit becomes completely invisible within the handwritten area. However, even if the recognition unit is no longer visible in the handwriting input area 804, in some embodiments, the invisible recognition unit is maintained at the top of the handwriting input stack, and from the recognition unit. The generated recognition result is continuously displayed in the candidate display area. In some embodiments, the faded recognition unit is not completely removed from view until a new handwriting input is written thereon.

  In some embodiments, the user device allows new handwriting input to be provided over the area occupied by the faded recognition unit as soon as the fading animation begins. In some embodiments, the user device may only initiate a new handwriting after fading has progressed to a particular stage (eg, to the weakest level or until recognition is completely invisible in that area). Allows input to be provided over the area occupied by the faded recognition unit.

  FIG. 23C shows that the first recognition unit (ie, stroke 2302) has completed its fading process (eg, ink color has stabilized at a very weak level or has become invisible. ) The user device identifies a further recognition unit (for example, a recognition unit for handwritten characters “a” and “m”) from the further handwriting stroke provided by the user, and the recognition result presented in the candidate display area 804. Has been updated.

  22D-22F show that over time, the user is providing a plurality of additional handwriting strokes (eg, 2304 and 2306) in the handwriting input area 804. FIG. At the same time, the previously identified recognition unit fades away from the handwriting input area 804 gradually. In some embodiments, each recognition unit takes a predetermined amount of time to begin its own fading process after that recognition unit is identified. In some embodiments, the fading process for each recognition unit does not begin until the user begins to enter a second recognition unit downstream of that recognition unit. As shown in FIGS. 23B-23F, when handwriting input is provided in cursive, a single stroke (e.g., stroke 2304 or stroke 2306) is within a handwriting input area, and multiple recognition units (e.g., In some cases, the word “am” or “back” may be followed via a recognition unit for each handwritten character).

  FIG. 22G shows that even after the recognition unit has started its fading process, the user is shown by a default lift-off, eg, an immediate lift-off following the delete button 2310 (eg, following contact 2308). This indicates that the recognition unit can be restored to a non-fade state by a tap gesture. When the recognition unit is played back, its appearance returns to the normal visibility level. In some embodiments, playback of faded recognition units is performed on a character basis, in the reverse direction of writing in the handwriting input area 804. In some embodiments, playback of faded recognition units is performed on a word-by-word basis within the handwriting input area 804. As shown in FIG. 23G, the recognition unit corresponding to the word “back” is reproduced from the fade completion state to the complete non-fade state. In some embodiments, the clock for initiating the fading process is reset for each recognition unit when the recognition unit is played back in a non-fading state.

  FIG. 22H shows that the last recognition unit in the default writing direction (eg, the recognition unit for the letter “k” in the word “back”) has been deleted from the handwriting input area 804 due to persistent contact on the delete button. Indicates that If this deletion input is continuously maintained, more recognition units (eg, recognition units for the characters “c”, “a”, “b” in the word “back”) will be in the reverse direction of writing, It is deleted one by one. In some embodiments, the recognition unit deletion is word unit, and all characters of the handwritten word “back” deleted from the handwriting input area 804 are removed simultaneously. FIG. 22H also shows that if the contact 2308 is maintained on the delete button 2310 after deletion of the recognition unit for the character “b” in the handwritten word “back”, the previously faded recognition unit “m” It also shows that it is played back in the same way.

  FIG. 23I shows that when the deletion input is stopped before deletion of the reproduced recognition unit “m” in the handwritten word “am” is performed, the reproduced recognition unit is gradually re-entered. Indicates fading. In some embodiments, the state of each recognition unit (eg, a state selected from a set of one or more fade states and non-fade states) is maintained and updated in the handwriting input stack.

  FIG. 23J illustrates the case where the user is providing one or more strokes 2312 over the area occupied by the faded recognition unit (eg, the recognition unit for the letter “I”) in the handwriting input area. In some embodiments, as shown in FIGS. 23I and 23J, the text of the highest-ranked recognition result (eg, result 2314) for the handwriting input performed before stroke 2312 is stored in the text input area 808. Indicates that it is automatically entered. As shown in FIG. 23J, the text “I am” is no longer shown as provisional, but instead is established in the text entry area 808. In some embodiments, when text input is performed on a fully faded or partially faded handwriting input, the handwriting input is removed from the handwriting input stack. The newly input stroke (eg, stroke 2312) becomes the current input in the handwriting input stack.

  As shown in FIG. 23J, the text “I am” is no longer shown as provisional, but instead is established in the text entry area 808. In some embodiments, when text input is performed on a fully faded or partially faded handwriting input, the handwriting input is removed from the handwriting input stack. The newly input stroke (eg, stroke 2312) becomes the current input in the handwriting input stack.

  In some embodiments, before stroke 2312, if stroke 2312 is provided over an area in the handwriting area that was occupied by a faded recognition unit (eg, a recognition unit for the letter “I”). The text of the highest-ranked recognition result (eg, result 2314) relating to the handwriting input performed on is not automatically entered into the text input area 808. Instead, the current handwriting input (both faded and non-fade) in the handwriting input area 804 is erased and cached in the handwriting input stack. A new stroke 2312 is added to the cached handwriting input in the handwriting input stack. The user device determines the recognition result based on the entire handwriting input currently stored in the handwriting input stack. Those recognition results are displayed in the candidate display area. In other words, the recognition results are cached in the handwriting input stack (the visible part and no longer, even if only a portion of the handwriting input currently accumulated is shown in the handwriting input area 804. It is generated based on the entire handwritten input (both invisible parts).

  FIG. 23K shows that the user has entered an additional stroke 2316 in the handwriting input area 804 that fades over time. FIG. 23L shows that the top recognition result 2320 text for the faded strokes 2312 and 2316 has been entered in the text input area 808 with the new stroke 2318 written over the faded strokes 2312 and 2316. It shows that.

  In some embodiments, the user optionally provides handwriting input in multiple lines. In some embodiments, when multiple lines of input are valid, the same fading process can be used to clear the handwriting input area for a new handwriting input.

  24A and 24B are a flow diagram of an example process 2400 for providing a fading process within a handwriting input area of a handwriting input interface. 23A-23K illustrate a process 2400, according to some embodiments.

  In some embodiments, the device receives a first handwriting input from a user (2402). The first handwriting input includes a plurality of handwriting strokes, and the plurality of handwriting strokes are associated with a handwriting input area of the handwriting input interface, and are distributed along respective writing directions. Form. In some embodiments, the user device renders each of the plurality of handwritten strokes in the handwriting input area as the handwritten strokes are provided by the user (2404).

  In some embodiments, the user device initiates a respective fading process for each of the plurality of recognition units after the recognition units are fully rendered (2406). In some embodiments, during each fading process, the rendering of the recognition units in the first handwriting input fades away. This is illustrated in FIGS. 23A-23F in some embodiments.

  In some embodiments, the user device is handwritten input occupied by a faded recognition unit of the plurality of recognition units, eg, as shown in FIGS. 23I, 23J, and 23K, 23L. A second handwriting input is received from the user over the area of the area (2408). In some embodiments, in response to receiving this second handwriting input (2410), the user device renders the second handwriting input within the handwriting input area (2412) and has all faded. The recognition unit is deleted from the handwriting input area (2414). In some embodiments, all recognition units that were entered in the handwriting input area prior to the second handwriting input, regardless of whether the recognition unit has started its fading process , Erased from the handwriting input area. This is illustrated, for example, in FIGS. 23I, 23J, 23K, and 23L.

  In some embodiments, the user device generates one or more recognition results for the first handwriting input (2416). In some embodiments, the user device displays (2418) those one or more recognition results in the candidate display area of the handwriting input interface. In some embodiments, in response to receiving the second handwriting input, the user device may display the highest-ranked recognition result displayed in the candidate display area within the text input area of the handwriting input interface without user selection. Are automatically input (2420). This is illustrated, for example, in FIGS. 23I, 23J, 23K, and 23L.

  In some embodiments, the user device stores (2422) an input stack that includes a first handwriting input and a second handwriting input. In some embodiments, the user device includes one or more multi-letter characters, each of which includes a corresponding array of spatial characters recognized from concatenation of the first handwriting input and the second handwriting input. A recognition result is generated (2424). In some embodiments, the user device may be in a candidate display area of the handwriting input interface while the rendering of the second handwriting input replaces the rendering of the first handwriting input in the handwriting input area. The recognition result of the one or more characters is displayed (2426).

  In some embodiments, each fading process for each recognition unit is initiated when a predetermined period of time has elapsed after that recognition unit has been completed by the user.

  In some embodiments, the fading process for each recognition unit is started after the recognition unit if the user has started entering a stroke for the next recognition unit.

  In some embodiments, the end state of each fading process for each recognition unit is a state with a predetermined minimum visibility for that recognition unit.

  In some embodiments, the end state of each fading process for each recognition unit is a state with zero visibility for that recognition unit.

  In some embodiments, after the last recognition unit in the first handwriting input fades, the user device receives a default playback input from the user (2428). In response to receiving this default playback input, the user device returns its last recognition unit from the faded state to the non-fade state (2430). This is illustrated, for example, in FIGS. 23F-23H. In some embodiments, this default playback input is an initial touch detected on a delete button provided within the handwriting input interface. In some embodiments, the persistent contact detected on the delete button deletes the last recognition unit from the handwriting input area and plays the penultimate recognition unit from the faded state to the non-fade state. This is illustrated, for example, in FIGS. 23G and 23H.

  As described herein, the multi-script handwriting recognition model performs handwritten character recognition in a stroke order independent and stroke direction independent manner. In some embodiments, the recognition model is trained only on spatially derived features that are included in a flat image of the written sample that correspond to various characters in the vocabulary of the handwriting recognition model. Since the images of the written samples do not contain any temporal information related to the individual strokes contained in those images, the resulting recognition model is a stroke order independent and stroke direction independent It is.

  As explained above, stroke order and stroke direction independent handwriting recognition is traditionally dependent on information related to the temporal generation of characters (eg, temporal alignment of strokes within those characters). Provides many advantages over the recognition system. However, in real-time handwriting recognition scenarios, temporal information related to individual strokes is available, and it may be beneficial to use this information to improve the recognition accuracy of the handwriting recognition system. In the following, a technique for integrating temporally derived stroke distribution information into the extraction of spatial features of the handwritten recognition model will be described. The use of this temporally derived stroke distribution information is described in the handwriting recognition system. This does not impair the stroke order and / or stroke direction independence. Based on the stroke distribution information associated with the various characters, disambiguation between similar-looking characters created with distinctly different sets of strokes is possible.

In some embodiments, when handwriting input is converted to an input image (eg, input bitmap image) for a handwriting recognition model (eg, CNN), temporal information associated with individual strokes is lost. For example, the Chinese character “
"Can be written using 8 strokes (e.g., labeled # 1- # 8 in FIG. 27). The arrangement and direction of strokes for this character provide some unique features associated with that character. A naive way to capture stroke order and stroke direction information without compromising the stroke order and stroke direction independence of the recognition system is to provide all possible permutations of stroke order and stroke direction in the training sample. , To enumerate explicitly. However, even for moderately complex characters, this permutation can exceed one billion, making it impractical if not impossible to implement in practice. As described herein, a stroke distribution profile is generated for each written sample that abstracts the time series aspect of stroke generation (ie, temporal information). The stroke distribution profiles of those written samples are trained to extract a set of temporally derived features that are then (for example, from the input bitmap image). ) Combined with spatially derived features, improves recognition accuracy without affecting the stroke order and stroke direction independence of the handwriting recognition system.

  As described herein, temporal information associated with a single character is extracted by calculating various pixel distributions that characterize each handwritten stroke. Every handwritten stroke of a character produces a deterministic pattern (or profile) when projected in a given direction. The pattern itself may not be sufficient to clearly recognize the stroke, but when combined with other similar patterns, it is appropriate to capture the specific characteristics unique to this particular stroke. Can be. This type of stroke representation is similarly integrated with spatial feature extraction (eg, feature extraction based on input images in CNN) to create ambiguity between similar-looking characters in the repertoire of handwriting recognition models. Orthogonality information is provided that can be useful to eliminate sex.

  FIGS. 25A and 25B are flowcharts of an example process 2500 for integrating temporally and spatially derived features of a handwritten sample during training of a handwriting recognition model. In this case, the resulting recognition model remains independent of stroke order and stroke direction. In some embodiments, the example process 2500 is performed on a server device that provides a trained recognition model to a user device (eg, portable device 100). In some embodiments, the server device includes one or more processors and memory, which includes instructions that, when executed by the one or more processors, execute the process 2500.

  In the exemplary process 2500, the device trains (2502) the spatially derived feature set and the temporally derived feature set of the handwriting recognition model separately and spatially derived. A set of features is trained on a corpus of training images, each of which is an image of a handwritten sample for each character of the output character set, and a temporally derived set of features is trained on a corpus of stroke distribution profiles. Each stroke distribution profile numerically characterizes the spatial distribution of multiple strokes within a handwritten sample for each character of the output character set.

  In some embodiments, separately training the set of spatially derived features further includes training a convolutional neural network having an input layer, an output layer, and a plurality of convolution layers (2504). ), The plurality of convolutional layers include an initial convolutional layer, a final convolutional layer, zero or more intermediate convolutional layers between the first convolutional layer and the final convolutional layer, and a concealment between the final convolutional layer and the output layer. Including layers. An exemplary convolution network 2602 is shown in FIG. This exemplary convolution network 2602 may be implemented in substantially the same manner as the convolution network 602 shown in FIG. The convolution network 2602 includes an input layer 2606, an output layer 2608, and a plurality of convolution layers, the plurality of layer convolution layers including an initial convolution layer 2610a, zero or more intermediate convolution layers, and a final convolution layer 2610n, and It includes a hidden layer 2614 between the final convolution layer and the output layer 2608. The convolution network 2602 also includes a kernel layer 2616 and a sub-sampling layer 2612 according to the arrangement shown in FIG. This convolutional network training is based on the written sample image 2614 in the training corpus 2604. Spatially derived features are obtained and the respective weights associated with these various features are determined by minimizing recognition errors for training samples in the training corpus. These same features and weights are used for the recognition of new handwritten samples that are not present in the training corpus after being trained.

  In some embodiments, separately training the temporally derived set of features provides multiple stroke distribution profiles to the statistical model to classify each character of the output character set. Further comprising determining (2506) a plurality of temporally derived parameters and respective weights for the plurality of temporally derived parameters. In some embodiments, a stroke distribution profile 2620 is derived from each written sample in the training corpus 2622, as shown in FIG. The training corpus 2622 optionally includes the same written samples as the corpus 2604, but also includes temporal information associated with stroke generation within each written sample. The stroke distribution profile 2622 is provided to a statistical modeling process 2624 during which temporally derived features are extracted and the respective weights for those various features are determined by statistical modeling methods (eg, Based on CNN, K-neighbor method, etc.) to minimize recognition errors or classification errors. As shown in FIG. 26, the temporally derived set of features and their respective weights are converted into a set of feature vectors (eg, feature vector 2626 or feature vector 2628), and the respective ones in convolutional neural network 2602 Introduced into the layer. The resulting network thus includes spatially derived parameters and temporally derived parameters that are orthogonal to each other and contribute together to character recognition. .

  In some embodiments, the device combines (2508) a spatially derived feature set and a temporally derived feature set in a handwriting recognition model. In some embodiments, combining the spatially derived feature set and the temporally derived feature set in the handwriting recognition model is one of the convolutional layers of the convolutional neural network. Or introducing a plurality of spatially derived parameters and a plurality of temporally derived parameters in the hidden layer (2510). In some embodiments, the plurality of temporally derived parameters and their respective weights for the plurality of temporally derived parameters are the final convolutional layer of the convolutional neural network for handwriting recognition (eg, FIG. 26). In the final convolution layer 2610n). In some embodiments, the plurality of temporally derived parameters and their respective weights for the plurality of temporally derived parameters are a convolutional neural network hidden layer for handwriting recognition (eg, in FIG. 26). In the hidden layer 2614).

  In some embodiments, the device uses this handwriting recognition model to provide real-time handwriting recognition for the user's handwriting input (2512).

  In some embodiments, the device generates a corpus of stroke distribution profiles from a plurality of written samples (2514). In some embodiments, each of the plurality of handwritten samples corresponds to a character in the output character set (2516), and for each constituent stroke of the handwritten sample, the respective spatial as the stroke was written. Store the information separately. In some embodiments, to generate a corpus of stroke distribution profiles, the device performs the following steps (2518).

  For each of the plurality of handwritten samples (2520), the device identifies a constituent stroke in the handwritten sample (2522), and for each of the identified strokes of the handwritten sample, the device Occupancy is calculated as a ratio of the projected span in each stroke direction to the maximum projected span of the written sample, and the occupancy is determined for the specified stroke of the handwritten sample. For each, the device also calculates a respective saturation rate for each stroke based on the ratio of the respective number of pixels in each stroke to the total number of pixels in the written sample (2526). The user device then generates (2528) a feature vector for the handwritten sample as a stroke distribution profile for the written sample, and the feature vector includes the occupancy and the respective occupancy rates of at least N strokes in the handwritten sample. Including each saturation rate, N is a predetermined natural number. In some embodiments, N is less than the maximum number of strokes observed in any single written sample in multiple written samples.

  In some embodiments, for each of the plurality of handwritten samples, the device sorts the occupancy of each identified stroke in descending order in each of the predetermined directions, and occupies the N highest ranks And only the writing samples of the saturation rate are included in the feature vector of the writing samples.

  In some embodiments, the plurality of predetermined directions include a horizontal direction, a vertical direction, a positive 45 degree direction, and a negative 45 degree direction of the written sample.

  In some embodiments, to use this handwriting recognition model to provide real-time handwriting recognition regarding the user's handwriting input, the device receives the user's handwriting input and responds to the user's handwriting input reception. Thus, the handwriting recognition output is provided to the user substantially simultaneously with the reception of the handwriting input.

The characters “
For illustrative purposes, exemplary embodiments are described herein. In some embodiments, each input image of handwritten characters is optionally normalized to a square. Measure the span of each individual handwritten stroke (eg, strokes # 1, # 2, ... and # 3) when projected onto this square, horizontally, vertically, diagonally +45 degrees and diagonally -45 degrees Is done. The span of each stroke Si is recorded as x span (i), y span (i), c span (i), and d span (i) for the four projection directions, respectively. Furthermore, the maximum span observed over the entire image is also recorded. The maximum span of this character is recorded as x span, y span, c span, and d span for the four projection directions, respectively. For illustrative purposes, four projection directions are optionally considered herein, but in principle any arbitrary set of projections can be used in various embodiments. Can do. The characters "" in the maximum span (e.g., shown as x span, y span, c span, and d span) and four projection directions
”(For example, stroke # 4) span (eg, shown as x span (4), y span (4), c span (4), and d span (4)). This is shown in FIG.

In some embodiments, when the above span is measured for all strokes 1-5 (5 is the number of individual handwritten strokes associated with this input image), along each projection direction , The respective occupancy is calculated. For example, the corresponding occupation ratio R _x (i) along the x direction with respect to the stroke S _i is calculated as R _x (i) = x span (i) / x span. Similarly, the corresponding occupancy along other projection directions can be calculated: R _y (i) = y span (i) / y span, R _c (i) = c span (i) / c Span, R _d (i) = d span (i) / d span.

  In some embodiments, the occupancy of all strokes in each direction is sorted separately in descending order, so that each ranking of all strokes in the input image is related to each projection direction. Is obtained in terms of the occupancy of those strokes in that direction. The stroke ranking in each projection direction reflects the relative importance of each stroke along the associated projection direction. This relative importance is independent of the order and direction in which the strokes were created within the written sample. Therefore, the ranking based on the occupation ratio is information derived in time of the stroke order and stroke direction independent expression.

  In some embodiments, each stroke is given a relative weight that indicates the importance of the stroke to the entire character. In some embodiments, this weight is measured by the ratio of the number of pixels in each stroke to the total number of pixels in the character. This ratio is referred to as the saturation rate associated with each stroke.

  In some embodiments, a feature vector can be created for each stroke based on the occupancy and saturation rate of each stroke. For each character, a set of feature vectors is created containing 5S features. This set of features is referred to as the stroke distribution profile for that character.

  In some embodiments, only a predetermined number of higher-ranked strokes are used in building a stroke distribution profile for each character. In some embodiments, this predetermined number of strokes is ten. Based on their top 10 strokes, 50 features derived from the strokes are generated for each character. In some embodiments, these features are introduced at the final convolutional layer of the convolutional neural network or at a subsequent hidden layer.

  In some embodiments, during real-time recognition, input images of recognition units are provided in a handwriting recognition mode that is trained with both spatially derived features and temporally derived features. The input image is processed through each layer of the handwriting recognition model shown in FIG. When the processing of this input image reaches a layer where stroke distribution profile input is required (eg, the final convolution layer or hidden layer), the stroke distribution profile of that recognition unit is introduced into that layer. This input image and stroke distribution profile processing continues until an output classification (eg, one or more candidate characters) is provided in the output layer 2608. In some embodiments, the stroke distribution profile of all recognition units is calculated and provided as input to the handwriting recognition model along with the input images of those recognition units. In some embodiments, the input image of the recognition unit is first passed through the handwriting recognition model (without benefiting from temporally trained features). If two or more similar appearance candidate characters are identified with close recognition confidence values, then the stroke distribution profile for that recognition unit is trained with features derived in time (eg, , The final convolution layer, or hidden layer) in the handwriting recognition model. When the input image of the recognition unit and the stroke distribution profile pass through the final layer of the handwriting recognition model, two or more similar appearance candidate characters can be better distinguished by the difference in their stroke distribution profiles. it can. Therefore, using temporally derived information on how recognition units are formed by individual handwritten strokes, recognition without compromising stroke order and stroke direction independence of the handwriting recognition system Accuracy is improved.

  The foregoing description has been described with reference to specific embodiments for purposes of explanation. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. These embodiments are intended to best illustrate the principles of the invention and its practical application, and thereby allow various modifications to be made by others skilled in the art to suit the particular application envisaged. In order to be able to best utilize the invention and various embodiments, it has been chosen and described.

Claims (17)

A device,
One or more processors;
A memory having stored instructions;
And when the instructions are executed by the one or more processors,
Receiving a handwriting input from a user including a plurality of handwriting strokes provided in a touch-sensitive surface coupled to the device;
Rendering the plurality of handwritten strokes in real time within a handwriting input area of a handwriting input interface;
Identifying two adjacent recognition units from the plurality of handwriting strokes;
Receiving one of a pinch gesture input and an enlarged gesture input on the plurality of handwriting strokes;
Generating a first recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as a single recognition unit upon receiving the pinch gesture input;
Generating a second recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as two separate recognition units separated by the enlarged gesture input upon reception of the enlarged gesture input; When,
Re-rendering the plurality of handwriting strokes to reduce the distance between the two adjacent recognition units in the handwriting input area in response to the pinch gesture input;
Displaying each of the first recognition result and the second recognition result in the candidate display area of the handwriting input interface when generating one of the first recognition result and the second recognition result;
A device that performs an operation including:   A device,
  One or more processors;
  A memory having stored instructions;
  And when the instructions are executed by the one or more processors,
  Receiving a handwriting input from a user including a plurality of handwriting strokes provided in a touch-sensitive surface coupled to the device;
  Rendering the plurality of handwritten strokes in real time within a handwriting input area of a handwriting input interface;
  Receiving one of a pinch gesture input and an enlarged gesture input on the plurality of handwriting strokes;
  Generating a first recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as a single recognition unit upon receiving the pinch gesture input;
  Generating a second recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as two separate recognition units separated by the enlarged gesture input upon reception of the enlarged gesture input; When,
  In response to the enlarged gesture input, a first subset of strokes assigned to a first recognition unit and a second subset of handwriting strokes assigned to a second recognition unit in the handwriting input area. Re-rendering the plurality of handwritten strokes to increase the distance;
  Displaying each of the first recognition result and the second recognition result in the candidate display area of the handwriting input interface when generating one of the first recognition result and the second recognition result;
  A device that performs an operation including: The pinch gesture input, converge toward each other within the area occupied by the plurality of handwriting strokes, characterized in that it comprises two contact on the touch-sensitive surface, according to claim 1 or 2 device. The device of claim 1 or 2 , wherein the magnified gesture input includes two contacts on the touch-sensitive surface that are distributed to each other within an area occupied by the plurality of handwriting strokes. When the instruction is executed by the one or more processors, said processor,
And that before Symbol candidate display area, and displays the initial recognition result including each character recognized from the two adjacent recognition units,
Receiving the pinch gesture input while the initial recognition result is displayed in the candidate display area;
The device according to claim 1, further comprising performing an operation including: The device of claim 5 , wherein displaying the first recognition result further comprises replacing the initial recognition result with the first recognition result within the candidate display area. . When the instructions are executed by the one or more processors, the processors
Identifying a single recognition unit from the plurality of handwritten strokes;
Displaying an initial recognition result including characters recognized from the single recognition unit in the candidate display area;
Receiving the enlarged gesture input while the initial recognition result is displayed in the candidate display area;
Characterized in that to further perform operations including A device according to claim 1 or 2.   8. The device of claim 7, wherein displaying the second recognition result further comprises replacing the initial recognition result with the second recognition result within the candidate display area. . A method for providing handwriting recognition,
A device having one or more processors and memory,
Receiving a handwriting input from a user including a plurality of handwriting strokes provided in a touch-sensitive surface coupled to the device;
Rendering the plurality of handwritten strokes in real time within a handwriting input area of a handwriting input interface;
Identifying two adjacent recognition units from the plurality of handwriting strokes;
Receiving one of a pinch gesture input and an enlarged gesture input on the plurality of handwriting strokes;
Generating a first recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as a single recognition unit upon receiving the pinch gesture input;
Generating a second recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as two separate recognition units separated by the enlarged gesture input upon reception of the enlarged gesture input; When,
Re-rendering the plurality of handwriting strokes to reduce the distance between the two adjacent recognition units in the handwriting input area in response to the pinch gesture input;
Displaying each of the first recognition result and the second recognition result in the candidate display area of the handwriting input interface when generating one of the first recognition result and the second recognition result;
A method comprising the steps of:   A method for providing handwriting recognition,
  A device having one or more processors and memory,
  Receiving a handwriting input from a user including a plurality of handwriting strokes provided in a touch-sensitive surface coupled to the device;
  Rendering the plurality of handwritten strokes in real time within a handwriting input area of a handwriting input interface;
  Receiving one of a pinch gesture input and an enlarged gesture input on the plurality of handwriting strokes;
  Generating a first recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as a single recognition unit upon receiving the pinch gesture input;
  Generating a second recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as two separate recognition units separated by the enlarged gesture input upon reception of the enlarged gesture input; When,
  In response to the enlarged gesture input, a first subset of strokes assigned to a first recognition unit and a second subset of handwriting strokes assigned to a second recognition unit in the handwriting input area. Re-rendering the plurality of handwritten strokes to increase the distance;
  Displaying each of the first recognition result and the second recognition result in the candidate display area of the handwriting input interface when generating one of the first recognition result and the second recognition result;
  A method comprising the steps of: The pinch gesture input, converge toward each other within the area occupied by the plurality of handwriting strokes, characterized in that it comprises two contact on the touch-sensitive surface, according to claim 9 or 10 Method. 11. The method of claim 9 or 10, wherein the magnified gesture input includes two contacts on the touch-sensitive surface that are distributed to each other within an area occupied by the plurality of handwriting strokes. And that before Symbol candidate display area, and displays the initial recognition result including each character recognized from the two adjacent recognition units,
Receiving the pinch gesture input while the initial recognition result is displayed in the candidate display area;
10. The method of claim 9 , further comprising:   The method of claim 13, wherein displaying the first recognition result further comprises replacing the initial recognition result with the first recognition result within the candidate display area. . Identifying a single recognition unit from the plurality of handwritten strokes;
Displaying an initial recognition result including characters recognized from the single recognition unit in the candidate display area;
Receiving the enlarged gesture input while the initial recognition result is displayed in the candidate display area;
The method according to claim 9 or 10, further comprising: The method of claim 15 , wherein displaying the second recognition result further comprises replacing the initial recognition result with the second recognition result within the candidate display area. . 17. A program comprising instructions that, when executed by one or more processors, cause the processor to perform the method of any one of claims 9-16 .