KR102221079B1 - Managing real-time handwriting recognition - Google Patents

Abstract

Methods, systems, and computer-readable media related to techniques for providing handwriting input functionality on a user device are provided. The handwriting recognition module can recognize tens of thousands of characters using a single handwriting recognition model and is trained to have a repertoire containing multiple non-overlapping scripts. The handwriting input module provides real-time, stroke order and stroke direction independent handwriting recognition for multi-character handwriting input. In particular, handwriting recognition independent of real-time, stroke order and stroke direction is provided for multi-character or sentence level Chinese handwriting recognition. User interfaces for providing handwriting input functionality are also disclosed.

Description

Real-time handwriting recognition management {MANAGING REAL-TIME HANDWRITING RECOGNITION}

The present invention relates to providing a handwriting input function on a computing device, and more particularly, to providing a real-time, multi-script, stroke order independent handwriting recognition and input function on a computing device.

The handwriting input method is an important alternative input method for computing devices with touch sensitive surfaces (eg, touch sensitive display screens or touch pads). Numerous users, especially in some Asian or Arab countries, are accustomed to writing in cursive, and may feel comfortable writing by hand as opposed to typing on the keyboard.

For certain logographic writing systems such as Hanzi and Kanji (also known as Chinese characters), alternative syllable input methods (e.g., Pinyin ( Although Pinyin) or Kana) are available, such syllable input methods are not appropriate when the user does not know how to spell the catchword letter as a catword and uses the incorrect catword spelling of the catchword letter. Therefore, whether handwriting input can be used on a computing device becomes important for users who cannot pronounce words well enough or cannot pronounce words at all for the relevant slogan writing system.

Although the handwriting input function is gaining some popularity in certain regions of the world, improvements are still needed. In particular, human handwriting is very diverse (eg, in terms of stroke order, size, typeface, etc.), and high-quality handwriting recognition software is complex and requires extensive training. As such, it has been a challenge to provide efficient real-time handwriting recognition and to calculate resources on a mobile device having a limited memory.

In addition, in today's multicultural world, users from different countries speak multiple languages, and it may often be necessary to write in more than one script (eg, write in Chinese a message referring to a movie title in English). However, it is cumbersome and inefficient to manually switch the recognition system to the desired script or language during writing. In addition, since expanding the recognition capability of a device to handle multiple scripts at the same time greatly increases the complexity of the recognition system and the demand for computer resources, the usefulness of conventional multi-script handwriting recognition techniques is severely limited.

Additionally, conventional handwriting techniques rely heavily on language or script-specific specificities to achieve recognition accuracy. Such specialties are not easily compatible with other languages or scripts. Thus, adding handwriting input capabilities for new languages or scripts is a hard work that is not done lightly by suppliers of devices and software. As a result, users of numerous languages have lost an important alternative input method for their electronic devices.

Conventional user interfaces for providing handwriting input include an area for allowing handwriting input from a user and an area for displaying handwriting recognition results. For portable devices with a small form factor, there is still a need for significant improvements in the user interface to improve efficiency, accuracy and user experience in general.

This specification describes a technique for providing multiscript handwriting recognition using a universal recognizer. A universal recognizer is trained using a large multi-script corpus of writing samples for characters in different languages and scripts. Training of universal recognizers is language independent, script independent, stroke order independent, and stroke direction independent. Thus, the same recognizer can recognize handwritten input of mixed languages, mixed scripts without requiring manual switching between input languages during use. Additionally, the universal recognizer is lightweight enough to be deployed as a standalone module on mobile devices to enable handwriting input in different languages and scripts used in different regions of the world.

Also, since a universal recognizer is trained on stroke order independent and spatially-derived features independent of stroke direction, it does not require any temporal or sequence information at the stroke-level. , The universal recognizer provides a number of additional features and advantages over conventional time-based recognition methods (eg, a recognition method based on the Hidden Markov Method (HMM)). For example, a user is allowed to enter one or more characters, phrases, and strokes of sentences in any order to still obtain the same recognition results. Thus, out-of-order multi-character input, and out-of-order modifications (eg, addition or rewriting) of previously entered characters are now possible.

In addition, a universal recognizer is used for real-time handwriting recognition, where time information for each stroke is available and optionally used to clarify or segment the handwriting input before character recognition is performed by the universal recognizer. . The real-time, stroke order independent recognition described herein is different from conventional offline recognition methods (eg, optical character recognition (OCR)) and may provide better performance than conventional offline recognition methods. In addition, the universal recognizer described herein does not explicitly embed distinct features of different variations in the recognition system (e.g., variations in speed, tempo, stroke order, stroke direction, stroke continuity, etc.). By handling the high variability of writing habits (eg, variability in speed, tempo, stroke order, stroke direction, stroke continuity, etc.), the overall complexity of the recognition system can be reduced.

As described herein, in some embodiments, the time-induced stroke distribution information is optionally reintroduced into a universal recognizer to clarify between similar-looking recognition outputs for the same input image and increase recognition accuracy. Since temporally-derived features and space-derived features are acquired through a separate training process and are combined only in the handwriting recognition model after separate training is completed, reintroduction of time-derived stroke distribution information Does not impair the independence of stroke order and stroke direction of universal recognizers. In addition, the time-induced stroke distribution information is carefully designed to capture distinct temporal characteristics of similar-looking characters, without relying on explicit knowledge of the differences in stroke orders of similar-looking characters.

User interfaces for providing handwriting input functionality are also described herein.

In some embodiments, a method of providing multi-script handwriting recognition comprises: training a multi-script handwriting recognition model based on space-derived features of a multi-script training corpus-multi-script training corpus. Contains individual handwriting samples corresponding to characters of at least three non-overlapping scripts; And providing real-time handwriting recognition for the user's handwriting input using the multiscript handwriting recognition model trained on the space-guided features of the multiscript training corpus.

In some embodiments, a method of providing multiscript handwriting recognition comprises receiving a multiscript handwriting recognition model-the multiscript recognition model has been trained on the space-derived features of the multiscript training corpus, and the multiscript training corpus Contains individual handwriting samples corresponding to characters of at least three non-overlapping scripts; Receiving a handwriting input from a user, the handwriting input comprising one or more handwritten strokes provided on a touch-sensitive surface coupled to the user device; And in response to receiving the handwriting input, providing the user with one or more handwriting recognition results in real time based on the multiscript handwriting recognition model trained on the space-derived features of the multiscript training corpus.

In some embodiments, a method of providing real-time handwriting recognition includes receiving a plurality of handwritten strokes from a user, the plurality of handwritten strokes corresponding to the handwritten character; Generating an input image based on the plurality of handwritten strokes; Performing real-time recognition of handwritten characters by providing the input image as a handwriting recognition model-the handwriting recognition model provides handwriting recognition independent of the stroke order -; And in real time upon receiving the plurality of handwritten strokes, displaying the same first output character regardless of the individual order in which the plurality of handwritten strokes were received from the user.

In some embodiments, the method includes receiving a second plurality of handwritten strokes from a user, the second plurality of handwritten strokes corresponding to a second handwritten character; Generating a second input image based on the second plurality of handwritten strokes; Performing real-time recognition of a second handwritten character by providing a second input image as a handwriting recognition model; And displaying a second output character corresponding to the second plurality of handwritten strokes in real time upon receiving the second plurality of handwritten strokes, wherein the first output character and the second output character The handwriting inputs of and the second plurality of handwriting inputs are simultaneously displayed in the spatial sequence regardless of the individual order provided by the user.

In some embodiments, the second plurality of handwritten strokes spatially follow the first plurality of handwritten strokes along a default writing direction of the handwriting input interface of the user device, and the second output character is in a spatial sequence along the default writing direction. Following the first output character, the method comprises receiving a third handwritten stroke from the user to revise the handwritten character, the third handwritten stroke temporally after the first plurality of handwritten strokes and the second plurality of handwritten strokes Received -; In response to receiving the third handwritten stroke, assigning the handwritten stroke to the same recognition unit as the first plurality of handwritten strokes based on the relative proximity of the third handwritten stroke to the first plurality of handwritten strokes; Generating a revised input image based on the first plurality of handwritten strokes and the third handwritten stroke; Performing real-time recognition of the revised handwritten character by providing the revised input image as a handwriting recognition model; And in response to receiving the third handwriting input, displaying a third output character corresponding to the revised input image, the third output character replacing the first output character and a second in a spatial sequence along the default writing direction. Displayed at the same time as the output text-additionally includes.

In some embodiments, the method includes: receiving a deletion input from a user while the third output character and the second output character are simultaneously displayed as a recognition result in a candidate display area of the handwriting input interface; And in response to the deletion input, deleting the second output character from the recognition result while maintaining the third output character in the recognition result.

In some embodiments, rendering the first plurality of handwritten strokes, the second plurality of handwritten strokes, and the third handwritten stroke in real time in a handwriting input area of a handwriting input interface as each of the handwritten strokes is provided by the user. ; And in response to receiving the deletion input, deleting individual renderings of the second plurality of handwritten strokes from the handwriting input area while maintaining separate renderings of the first plurality of handwritten strokes and the third handwritten stroke in the handwriting input area.

In some embodiments, a method of providing real-time handwriting recognition includes receiving a handwriting input from a user, the handwriting input including one or more handwriting strokes provided in a handwriting input area of the handwriting input interface; Identifying a plurality of output characters for the handwriting input based on the handwriting recognition model; Dividing the plurality of output characters into two or more categories based on a predetermined classification criterion; In the initial view of the candidate display area of the handwriting input interface, displaying individual output characters in the first category among two or more categories-The initial view of the candidate display area is an extended view of the candidate display area. Equipped with the affordance (affordance) for calling) at the same time -; Receiving a user input for selecting an affordance for invoking an extended view; And in response to a user input, in the extended view of the candidate display area, individual output characters in the first category of the two or more categories and individual output characters in at least the second category not previously displayed in the initial view of the candidate display area. And displaying them.

In some embodiments, a method of providing real-time handwriting recognition includes receiving a handwriting input from a user, the handwriting input comprising a plurality of handwriting strokes provided in a handwriting input area of the handwriting input interface; Recognizing a plurality of output characters from the handwriting input based on the handwriting recognition model, the output characters comprising at least a first emoji character and at least a first character from a script of natural human language; And displaying a recognition result comprising the first emoji character and the first character from the script of natural human language in the candidate display area of the handwriting input interface.

In some embodiments, a method of providing handwriting recognition includes receiving a handwriting input from a user, the handwriting input comprising a plurality of handwriting strokes provided at a touch-sensitive surface coupled to the device; Rendering a plurality of handwritten strokes in real time in the handwriting input area of the handwriting input interface; Receiving one of a pinch gesture input and an expand gesture input for a plurality of handwritten strokes; Upon receiving the pinch gesture input, generating a first recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as a single recognition unit; Upon receiving the inflation gesture input, generating a second recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as two separate recognition units separated by the inflation gesture input; And when generating individual recognition results of the first recognition result and the second recognition result, displaying the recognition result generated in the candidate display area of the handwriting input interface.

In some embodiments, a method of providing handwriting recognition includes receiving a handwriting input from a user, the handwriting input comprising a plurality of handwriting strokes provided in a handwriting input area of the handwriting input interface; Identifying a plurality of recognition units from the plurality of handwritten strokes, each recognition unit comprising a separate subset of the plurality of handwritten strokes; Generating a multi-character recognition result including individual characters recognized from a plurality of recognition units; Displaying a multi-character recognition result in a candidate display area of the handwriting input interface; Receiving a deletion input from a user while the multi-character recognition result is displayed in the candidate display area; And in response to receiving the deletion input, removing an end character from the multi-character recognition result displayed in the candidate display area.

In some embodiments, a method of providing real-time handwriting recognition includes determining an orientation of a device; Providing a handwriting input interface on the device in a horizontal input mode according to the device in the first orientation-Individual lines of handwriting input input in the horizontal input mode are divided into one or more individual recognition units along the horizontal writing direction- ; And providing a handwriting input interface on the device in a vertical input mode according to the device in the second orientation.- Individual lines of handwriting input input in the vertical input mode are divided into one or more individual recognition units along the vertical writing direction. -Includes.

In some embodiments, a method of providing real-time handwriting recognition includes receiving a handwriting input from a user, the handwriting input comprising a plurality of handwriting strokes provided on a touch-sensitive surface coupled to the device; Rendering a plurality of handwritten strokes in the handwriting input area of the handwriting input interface; Dividing the plurality of handwritten strokes into two or more recognition units, each recognition unit comprising a separate subset of the plurality of handwritten strokes; Receiving an edit request from a user; Visually distinguishing two or more recognition units in the handwriting input area in response to the editing request; And providing means for individually deleting each of the two or more recognition units from the handwriting input area.

In some embodiments, the method of providing real-time handwriting recognition includes receiving a first handwriting input from a user-the first handwriting input includes a plurality of handwriting strokes, and the plurality of handwriting strokes is a handwriting input of a handwriting input interface. Forming a plurality of recognition units distributed along individual writing directions associated with the area; Rendering each of the plurality of handwritten strokes in the handwriting input area as the handwritten strokes are provided by the user; Starting a separate fading process for each of the plurality of recognition units after the recognition unit has been fully rendered-during the individual fading process, the rendering of the recognition unit in the first handwriting input gradually fades; Receiving a second handwriting input from a user over an area occupied by the faded recognition units of the plurality of recognition units among the handwriting input areas; And in response to receiving the second handwriting input: rendering the second handwriting input in the handwriting input area; And clearing all of the fading recognition units from the handwriting input area.

In some embodiments, a method of providing handwriting recognition comprises separately training a set of space-derived features and a set of time-directed features of a handwriting recognition model-the set of space-derived features Trained on the corpus, each of the images is an image of a handwritten sample for an individual character in the output character set, a set of time-directed features is trained on the corpus of stroke-distribution profiles, and each stroke-distribution profile is output. Numerically characterizing the spatial distribution of a plurality of strokes in a handwritten sample for individual characters of a character set; Combining the set of space-derived features and the set of temporal-derived features in the handwriting recognition model; And providing real-time handwriting recognition for the user's handwriting input using the handwriting recognition model.

The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims.

1 is a block diagram illustrating a portable multifunction device with a touch-sensitive display in accordance with some embodiments.
2 illustrates a portable multifunction device with a touch sensitive display in accordance with some embodiments.
3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface in accordance with some embodiments.
4 illustrates an exemplary user interface for a multifunction device having a touch-sensitive surface separate from a display in accordance with some embodiments.
5 is a block diagram of an operating environment of a handwriting input system according to some embodiments.
6 is a block diagram of a multiscript handwriting recognition model according to some embodiments.
7 is a flowchart of an exemplary process for training a multiscript handwriting recognition model in accordance with some embodiments.
8A and 8B illustrate exemplary user interfaces representing real-time multi-script handwriting recognition and input on a portable multifunction device in accordance with some embodiments.
9A and 9B are flowcharts of an exemplary process for providing real-time multi-script handwriting recognition and input on a portable multifunction device.
10A-10C are flowcharts of an exemplary process for providing real-time stroke order independent handwriting recognition and input on a portable multifunction device in accordance with some embodiments.
11A to 11K illustrate selectively displaying recognition results of one category in a normal view of a candidate display area, and selectively selecting recognition results of other categories in an extended view of the candidate display area, according to some embodiments. Shows exemplary user interfaces for display.
12A and 12B are for selectively displaying recognition results of one category in the normal view of the candidate display area and selectively displaying recognition results of other categories in the extended view of the candidate display area, according to some embodiments. Flowcharts of an exemplary process.
13A-13E illustrate exemplary user interfaces for inputting emoji characters through handwriting input according to some embodiments.
14 is a flowchart of an exemplary process for entering emoji characters through handwriting input in accordance with some embodiments.
15A-15K illustrate exemplary user interfaces for notifying a handwriting input module of how to divide a currently accumulated handwriting input into one or more recognition units using a pinch or inflate gesture, in accordance with some embodiments.
16A and 16B are flowcharts of an exemplary process for notifying a handwriting input module of how to divide a currently accumulated handwriting input into one or more recognition units using a pinch or inflate gesture, in accordance with some embodiments.
17A-17H illustrate exemplary user interfaces for providing character-by-character deletion of a user's handwriting input according to some embodiments.
18A and 18B are flowcharts of an exemplary process for providing character-by-character deletion of a user's handwriting input in accordance with some embodiments.
19A-19F illustrate exemplary user interfaces for switching between a vertical write mode and a horizontal write mode in accordance with some embodiments.
20A-20C are flowcharts of an exemplary process for switching between a vertical write mode and a horizontal write mode in accordance with some embodiments.
21A-21H illustrate user interfaces for providing a means to display and selectively delete individual recognition units identified in a user's handwriting input, according to some embodiments.
22A and 22B are flowcharts of an exemplary process for providing a means to display and selectively delete individual recognition units identified in a user's handwriting input, in accordance with some embodiments.
23A to 23L illustrate an implicit confirmation input for inputting a recognition result displayed for the existing handwriting input into a new handwriting input provided over the existing handwriting input in the handwriting input area, according to some embodiments. input).
24A and 24B illustrate using a new handwriting input provided over an existing handwriting input in the handwriting input area as an implicit confirmation input for inputting a recognition result displayed for the existing handwriting input, according to some embodiments. Flowcharts of an exemplary process for.
25A and 25B are examples for integrating time-induced stroke distribution information into a handwriting recognition model based on space-derived features without impairing the stroke order and stroke direction independence of the handwriting recognition model, according to some embodiments. These are flowcharts of a typical process.
26 is a block diagram illustrating separate training and subsequent integration of spatial- and temporal-induced features of an exemplary handwriting recognition system in accordance with some embodiments.
27 is a block diagram illustrating an exemplary method for calculating a stroke distribution profile of a character.
Like reference numerals refer to corresponding parts throughout the drawings.

Many electronic devices have graphical user interfaces with soft keyboards for text input. On some electronic devices, the user may also install or enable a handwriting input interface that allows the user to enter characters via handwriting on a touch-sensitive surface or touch-sensitive display screen coupled to the devices. have. Conventional handwriting recognition input methods and user interfaces have several problems and disadvantages. E.g,

일반적으로, 종래의 필기 입력 기능은 언어 단위로 또는 스크립트 단위로 가능하게 된다. 각각의 추가적인 입력 언어는 별도의 저장 공간 및 메모리를 차지하는 별도의 필기 인식 모델의 설치를 요구한다. 상이한 언어들에 대한 필기 인식 모델들을 조합함으로써의 시너지 효과는 거의 제공되지 않으며, 혼합 언어 또는 혼합 스크립트 필기 인식은 종래에는 복잡한 명확화 프로세스로 인해 매우 긴 시간이 걸렸다.

In general, the conventional handwriting input function is enabled in units of languages or scripts. Each additional input language requires the installation of a separate handwriting recognition model that occupies a separate storage space and memory. Synergy by combining handwriting recognition models for different languages is hardly provided, and mixed language or mixed script handwriting recognition has conventionally taken a very long time due to a complex disambiguation process.

또한, 종래의 필기 인식 시스템들은 문자 인식을 위한 언어-특정 또는 스크립트-특정 특성들에 지나치게 의존한다. 혼합 언어 필기 입력의 인식은 부족한 정확도를 가졌다. 게다가, 인식된 언어들의 이용 가능한 조합들은 매우 제한된다. 대부분의 시스템들은 사용자에게 각각의 디폴트가 아닌(non-default) 언어 또는 스크립트에서 필기 입력을 제공하기 전에 원하는 언어-특정 필기 인식기를 수동으로 특정하도록 요구하였다.

In addition, conventional handwriting recognition systems rely too heavily on language-specific or script-specific properties for character recognition. The recognition of mixed language handwriting input had insufficient accuracy. In addition, the available combinations of recognized languages are very limited. Most systems required the user to manually specify the desired language-specific handwriting recognizer before providing handwriting input in each non-default language or script.

수많은 기존의 실시간 필기 인식 모델들은 획 단위(stroke-by-stroke) 레벨 상에서 시간 또는 시퀀스 정보를 요구하며, 이는 문자를 기입할 수 있는 방법의 높은 가변성(예컨대, 서체들 및 개인의 습관들로 인한 획들의 형상, 길이, 템포, 분할, 순서 및 방향에서의 높은 가변성)을 다룰 때 부정확한 인식 결과들을 생성한다. 일부 시스템들은 또한 사용자들에게 필기 입력을 제공할 때 (예컨대, 각 문자 입력의 크기, 시퀀스 및 시간프레임에 대한 고유한 가정들에 의한) 엄밀한 공간 및 시간 기준들을 준수하도록 요구한다. 이들 기준들로부터 벗어나면 수정하기 어려운 부정확한 인식 결과들이 야기되었다.

Many existing real-time handwriting recognition models require time or sequence information on a stroke-by-stroke level, which is due to the high variability of the way characters can be written (e.g., due to fonts and personal habits). High variability in shape, length, tempo, segmentation, order and direction of strokes) produces inaccurate recognition results. Some systems also require users to adhere to strict spatial and temporal standards (eg, due to unique assumptions about the size, sequence, and timeframe of each character input) when providing handwriting input to users. Deviations from these criteria have resulted in inaccurate recognition results that are difficult to correct.

현재, 대부분의 실시간 필기 입력 인터페이스들은 사용자가 한 번에 수 개의 문자들만을 입력하는 것을 허용한다. 긴 어구들 또는 문장들의 입력은 짧은 세그먼트들로 분해되어 별도로 입력된다. 이러한 격식적인 입력은 구성(composition)의 흐름을 유지하기 위해 사용자에게 인지적 부담을 지울 뿐만 아니라, 사용자가 먼저 입력된 문자 또는 어구를 수정 또는 개정하는 것을 어렵게 하기도 한다.

Currently, most real-time handwriting input interfaces allow a user to enter only several characters at a time. Input of long phrases or sentences is divided into short segments and input separately. Such formal input not only places a cognitive burden on the user in order to maintain the flow of the composition, but also makes it difficult for the user to correct or revise the characters or phrases first entered.

The embodiments described below address these and related problems.

1 to 4 below provide a description of exemplary devices. 5, 6, 26 and 27 illustrate exemplary handwriting recognition and input systems. 8a and 8b, 11a to 11k, 13a to 13e, 15a to 15k, 17a to 17h, 19a to 19f, 21a to 21h, 23a to 12l are handwriting recognition And example user interfaces for input. Figures 7, 9a and 9b, 10a to 10c, 12a and 12b, 14, 16a and 16b, 18a and 18b, 20a to 20c, 22a and 22b, 24a And FIGS. 24B and 25 illustrate training handwriting recognition models, providing real-time handwriting recognition results, providing means for inputting and revising handwriting input, and inputting the recognition results as text input. Flowcharts illustrating methods for enabling handwriting recognition and input on user devices, including providing a means. Users in FIGS. 8A and 8B, 11A to 11K, 13A to 13E, 15A to 15K, 17A to 17H, 19A to 19F, 21A to 21H, and 23A to 12L The interfaces are in Figs. 7, 9A and 9B, 10A to 10C, 12A and 12B, 14, 16A and 16B, 18A and 18B, 20A to 20C, 22A and 22B, 24A and 24B and are used to illustrate the processes in FIG. 25.

Example devices

Reference will now be made in detail to the embodiments, and examples of the embodiments are shown in the accompanying drawings. In the detailed description that follows, many specific details are set forth to provide a thorough understanding of the invention. However, it will be apparent to a person skilled in the art that the present invention may be practiced without these specific details. In other examples, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

While the terms “first”, “second”, and the like may be used to describe various elements herein, it will also be understood that these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact, without departing from the scope of the present invention. Both the first contact and the second contact are contacts, but they are not the same contact.

In the present specification, terms used in the description of the present invention are for the purpose of describing specific embodiments only, and are not intended to limit the present invention. As used in the description of the invention and in the appended claims, the singular forms ("a", "an" and "the") are intended to include the plural forms as well, unless the context clearly indicates otherwise. . In addition, it will be understood that the term “and/or” as used herein represents and encompasses any and all possible combinations of one or more of the listed related items. The terms “include”, “including”, “comprise”, and/or “comprising”, as used herein, refer to the stated features, integers Specifies the presence of elements, steps, actions, elements, and/or components, but the presence of one or more other features, integers, steps, actions, elements, components, and/or groups thereof It will be further understood that it does not exclude additions or additions.

As used herein, the term “if” means “when” or “upon” or “in response to determining” depending on the context. Or it can be interpreted to mean "in response to detecting". Similarly, the phrase "when determined" or "when [statement or event] is detected" means "on decision" or "in response to decision" or "on detection of [statement or event]" or May be interpreted to mean "in response to detecting [the stated condition or event]."

Embodiments of electronic devices, user interfaces for such devices, and related processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions such as PDA and/or music player functions. Exemplary embodiments of portable multifunction devices include iPhone®, iPod Touch®, and iPad® from Apple Inc. of Cupertino, CA, USA. iPad) (registered trademark) devices are included without limitation. Other portable electronic devices such as laptops or tablet computers with touch sensitive surfaces (eg, touch screen displays and/or touch pads) may also be used. It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer having a touch-sensitive surface (eg, a touch screen display and/or a touch pad).

In the discussion that follows, an electronic device is described that includes a display and a touch-sensitive surface. However, it should be understood that the electronic device may include one or more other physical user-interface devices, such as a physical keyboard, mouse and/or joystick.

Devices typically include drawing applications, presentation applications, word processing applications, website creation applications, disk authoring applications, spreadsheet applications, gaming applications, phone applications, video conferencing applications, email applications, instant messaging applications, exercise support applications, and photos. It supports a variety of applications such as one or more of a management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application.

Various applications that can run on the device can use at least one common physical user-interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface as well as the corresponding information displayed on the device may be adjusted and/or changed from one application to the next and/or within a separate application. In this way, the general physical architecture of the device (such as a touch-sensitive surface) can support a variety of applications with user interfaces that are intuitive and clear to the user.

Attention is now directed to embodiments of portable devices with touch sensitive displays. 1 is a block diagram illustrating a portable multifunction device 100 having touch sensitive displays 112 in accordance with some embodiments. The touch sensitive display 112 is sometimes referred to as a “touch screen” for convenience, and may also be known or referred to as a touch sensitive display system. Device 100 includes memory 102 (which may include one or more computer-readable storage media), memory controller 122, one or more processing units (CPUs) 120, peripherals interface 118, RF circuitry. 108, audio circuit 110, speaker 111, microphone 113, input/output (I/O) subsystem 106, other input or control devices 116 and external port 124. Can include. Device 100 may include one or more optical sensors 164. These components may communicate via one or more communication buses or signal lines 103.

The device 100 is only an example of a portable multifunction device, and the device 100 may have more or fewer components than shown, may combine two or more components, or have a different configuration or arrangement of components. It should be understood that it can be. The various components shown in FIG. 1 may be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and/or application specific integrated circuits.

Memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by other components of device 100 such as CPU 120 and peripherals interface 118 may be controlled by memory controller 122.

The peripherals interface 118 may be used to couple the input and output peripherals of the device to the CPU 120 and memory 102. One or more processors 120 drive or execute various software programs and/or sets of instructions stored in memory 102 to perform various functions for device 100 and process data.

In some embodiments, peripheral interface 118, CPU 120, and memory controller 122 may be implemented on a single chip, such as chip 104. In some other embodiments, they can be implemented on separate chips.

A radio frequency (RF) circuit 108 receives and transmits RF signals, also called electromagnetic signals. The RF circuit 108 converts electrical signals to/from electromagnetic signals, and communicates with communication networks and other communication devices via the electromagnetic signals.

The audio circuit 110, speaker 111 and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111. The speaker 111 converts the electrical signal into sound waves that can be heard by humans. The audio circuit 110 also receives an electrical signal converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signal into audio data and transmits the audio data to the peripherals interface 118 for processing. Audio data may be retrieved from and/or transmitted to memory 102 and/or RF circuit 108 by peripherals interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (eg, 212 in FIG. 2 ).

I/O subsystem 106 couples input/output peripherals on device 100 to peripherals interface 118, such as touch screen 112 and other input control devices 116. The I/O subsystem 106 may include a display controller 156 and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive/transmit electrical signals to/from other input or control devices 116. Other input control devices 116 may include physical buttons (eg, push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and the like. In some alternative embodiments, the input controller(s) 160 may be coupled (or not coupled to any) to any of a keyboard, infrared port, USB port, and pointer device such as a mouse. . One or more buttons (eg, 208 in FIG. 2) may include up/down buttons for controlling the volume of the speaker 111 and/or the microphone 113. One or more buttons may include a push button (eg, 206 in FIG. 2 ).

The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. Display controller 156 receives and/or transmits electrical signals from/to touch screen 112. The touch screen 112 displays a visual output to the user. Visual output may include graphics, text, icons, video, and any combination thereof (collectively referred to as “graphic”). In some embodiments, some or all of the visual output may correspond to user-interface objects.

The touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic and/or tactile contact. The touch screen 112 and the display controller 156 (along with any associated modules and/or sets of instructions in the memory 102) contact on the touch screen 112 (and any movement or interruption of the contact). And converts the detected contact into interactions with user-interface objects (eg, one or more soft keys, icons, web pages or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

The touch screen 112 may use LCD (liquid crystal display) technology, LPD (light-emitting polymer display) technology, or LED (light-emitting diode) technology, although other display technologies may be used in other embodiments. The touch screen 112 and display controller 156 utilize capacitive, resistive, infrared and surface acoustic wave technologies as well as other proximity sensor arrays or other elements to determine one or more points of contact with the touch screen 112. Any of a plurality of touch sensing technologies, including but not limited to, currently known or to be developed in the future, can be used to detect contact and any movement or interruption thereof. In an exemplary embodiment, projection-type mutual capacitive sensing, such as found on the iPhone (registered trademark), iPod Touch (registered trademark), and the iPad (registered trademark) from Apple Inc. of Cupertino, CA, USA Projected mutual capacitance sensing technology is used.

The touch screen 112 may have a video resolution exceeding 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi. A user may contact the touch screen 112 using any suitable object or attachment, such as a stylus, finger, or the like. In some embodiments, the user interface is designed to work primarily with finger-based contacts and gestures, which may be less precise than stylus-based input due to the larger contact area of the finger on the touch screen. In some embodiments, the device converts the coarse finger-based input into a precise pointer/cursor position or command to perform the user's desired actions. Handwriting input may be provided on the touch screen 112 through the location and movement of a finger-based contact or a stylus-based contact. In some embodiments, touch screen 112 renders finger-based input or stylus-based input as immediate visual feedback to the current handwriting input, and uses a writing instrument (e.g., pen) to render a writing surface (e.g., a pen). It provides the visual effect of the actual entry on a sheet of paper).

In some embodiments, in addition to the touch screen, the device 100 may include a touch pad (not shown) to activate or deactivate certain functions. In some embodiments, the touchpad is a touch sensitive area of the device that does not display a visual output unlike a touch screen. The touch pad may be a touch-sensitive surface separate from the touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

Device 100 also includes a power system 162 for supplying power to various components. The power system 162 includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode). (LED)), and any other components associated with the generation, management, and distribution of power within portable devices.

Device 100 may also include one or more optical sensors 164. 1 shows an optical sensor coupled to an optical sensor controller 158 in an I/O subsystem 106. The optical sensor 164 may include a charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors. The optical sensor 164 receives light from the surrounding environment projected through one or more lenses, and converts the light into data representing an image. The optical sensor 164, together with the imaging module 143 (also referred to as a camera module), may capture still images or video.

Device 100 may also include one or more proximity sensors 166. 1 shows a proximity sensor 166 coupled to a peripherals interface 118. Alternatively, proximity sensor 166 may be coupled to input controller 160 in I/O subsystem 106. In some embodiments, the proximity sensor turns off and disables the touch screen 112 when the multifunction device is positioned near the user's ear (e.g., when the user is making a phone call). Let it.

Device 100 may also include one or more accelerometers 168. 1 shows an accelerometer 168 coupled to a peripherals interface 118. Alternatively, accelerometer 168 can be coupled to input controller 160 in I/O subsystem 106. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from one or more accelerometers. The device 100 includes a magnetometer (not shown) in addition to the accelerometer(s) 168, and a GPS (or GLONASS) for obtaining information about the location and orientation (e.g., vertical or horizontal) of the device 100. Or another global navigation system) receiver (not shown).

In some embodiments, software components stored in memory 102 include operating system 126, communication module (or set of instructions) 128, contact/motion module (or set of instructions) 130, graphics module ( Or a set of instructions) 132, a text input module (or set of instructions) 134, a satellite positioning system (GPS) module (or set of instructions) 135, and applications (or a set of instructions) ( 136). Further, in some embodiments, the memory 102 stores a handwriting input module 157 as shown in FIGS. 1 and 3. The handwriting input module 157 includes a handwriting recognition model and provides a handwriting recognition and input function to a user of the device 100 (or device 300 ). Further details of the handwriting input module 157 are provided in connection with FIGS. 5 to 27 and accompanying descriptions thereof.

The operating system 126 (e.g., an embedded operating system such as Darwin, RTXC, Linux, UNIX, OS X, Windows, or VxWorks) includes general system tasks (e.g., memory Management, storage device control, power management, etc.), and facilitates communication between various hardware and software components.

The communication module 128 facilitates communication with other devices via one or more external ports 124, and also for processing data received by the RF circuit 108 and/or external port 124. It contains various software components. External port 124 (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) to couple directly to other devices or indirectly through a network (e.g., Internet, wireless LAN, etc.). It is composed.

The contact/motion module 130 may detect contact with the touch screen 112 (in conjunction with the display controller 156) and other touch sensitive devices (eg, a touchpad or a physical click wheel). The contact/motion module 130 determines whether a contact has occurred (e.g., detecting a finger-down event), determines whether there is a movement of the contact, and moves across the touch-sensitive surface. Tracking (e.g., detecting one or more finger-dragging events), and determining whether the contact has ceased (e.g., a finger-up event or contact interruption To detect a contact), including various software components for performing various operations related to the detection of a contact. The contact/motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point represented by the series of contact data may include determining the speed (magnitude), velocity (magnitude and direction) and/or acceleration (change in magnitude and/or direction) of the contact point. . These actions may be applied to single contacts (eg, one finger contacts) or to multiple simultaneous contacts (eg “multi-touch”/multiple finger contacts). In some embodiments, the contact/motion module 130 and the display controller 156 detect contact on the touchpad.

The contact/motion module 130 may detect a gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns. Thus, a gesture can be detected by detecting a specific contact pattern. For example, detecting a finger tap gesture is to detect a finger-down event and then a finger at the same location (or substantially the same location) as the finger-down event (e.g., the location of the icon). It involves detecting an up (lift) event. As another example, detecting a finger swipe gesture on a touch-sensitive surface detects a finger-down event followed by one or more finger-drag events, followed by a finger-up (lift). It involves detecting the event.

The contact/motion module 130 is configured by the handwriting input module 157 in the handwriting input area of the handwriting input interface displayed on the touch-sensitive display screen 112 (or handwriting displayed on the display 340 of FIG. 3 ). It is selectively utilized to register input of handwritten strokes (within the area of the touch pad 355 corresponding to the input area). In some embodiments, the initial finger-down event, the last finger-up event, the locations associated with the contact during any time between them, the motion path and intensities are recorded as a handwritten stroke. Based on such information, handwritten strokes can be rendered on the display as feedback to user input. Additionally, one or more input images may be generated based on the handwritten strokes registered by the contact/motion module 130.

The graphics module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components for changing the intensity of the displayed graphics. As used herein, the term “graphic” limits text, web pages, icons (eg, user-interface objects including soft keys), digital images, videos, animations, etc. Includes any object that can be displayed to the user, including without.

In some embodiments, graphics module 132 stores data representing graphics to be used. Each graphic can be assigned a corresponding code. If necessary, the graphic module 132 receives one or more codes specifying a graphic to be displayed, along with coordinate data and other graphic characteristic data, from an application, etc., and then generates screen image data and outputs it to the display controller 156. .

The text input module 134, which may be a component of the graphics module 132, includes various applications (e.g., contacts 137, email 140, IM 141), browsers 147, and Soft keyboards for entering text in any other application). In some embodiments, the handwriting input module 157 is selectively invoked through the user interface of the text input module 134, such as through a keyboard selection affordance. In some embodiments, the same or similar keyboard selection affordance is also provided in the handwriting input interface to invoke the text input module 134.

The GPS module 135 determines the location of the device and uses this information to the phone 138 for use in various applications (e.g., for use in location-based dialing, and the camera 143 as photo/video metadata. ), and to applications that provide location-based services such as weather widgets, local yellow page widgets and map/navigation widgets).

Applications 136 may include the following modules (or sets of instructions), or a subset or superset thereof: contacts module 137 (sometimes referred to as an address book or contact list); Telephone module 138; Video conferencing module 139; Email client module 140; Instant messaging (IM) module 141; Exercise support module 142; A camera module 143 for still and/or video images; Image management module 144; Browser module 147; A calendar module 148; Weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user Widget modules 149, which may include one or more of these, and user-generated widgets 149-6; A widget generator module 150 for creating user-generated widgets 149-6; Search module 151; A video and music player module 152 that may be composed of a video player module and a music player module; Memo module 153; Map module 154; And/or online video module 155.

Examples of other applications 136 that may be stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights. Includes management, speech recognition and speech replication.

Along with the touch screen 112, the display controller 156, the contact module 130, the graphic module 132, the handwriting input module 157 and the text input module 134, the contact module 137 is named in the address book. Add(s); Delete name(s) from address book; Associating phone number(s), email address(s), physical address(s), or other information with a name; Associating image with name; Sorting and sorting names; Provide phone numbers or email addresses to initiate and/or facilitate communication by phone 138, video conference 139, email 140, or IM 141; It may be used to manage an address book or contact list (eg, stored in the application internal state 192 of the contacts module 137 in the memory 102 or in the memory 370), including the like.

RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact module 130, graphic module 132, handwriting input module ( 157) and text input module 134, the phone module 138 inputs a sequence of characters corresponding to the phone number, accesses one or more phone numbers in the address book 137, and retrieves the input phone number. It can be used to edit, dial each phone number, have a conversation, and disconnect or hang up when the conversation is complete. As noted above, wireless communication may utilize any of a number of communication standards, protocols and technologies.

RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact module ( 130), a graphic module 132, a handwriting input module 157, a text input module 134, a contact list 137, and a phone module 138, together with the video conference module 139 And executable instructions for initiating, conducting, and ending a video conference between one or more other participants.

Together with RF circuit 108, touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157 and text input module 134, an email client module ( 140) includes executable instructions for generating, sending, receiving, and managing email in response to user instructions. Together with the image management module 144, the email client module 140 makes it very easy to create and send an email with still or video images taken with the camera module 143.

With RF circuit 108, touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157 and text input module 134, an instant messaging module ( 141) inputs a sequence of characters corresponding to an instant message, corrects previously entered characters (e.g., Short Message Service (SMS) for phone-based instant messages or multimedia Send individual instant messages, receive instant messages, and view received instant messages (using the Multimedia Message Service (MMS) protocol or using XMPP, SIMPLE or IMPS for Internet-based instant messages) Includes possible commands. In some embodiments, sent and/or received instant messages are as supported by graphics, photos, audio files, video files, and/or MMS and/or Enhanced Messaging Service (EMS). Other attachments such as may be included. As used herein, "instant messaging" refers to phone-based messages (eg, messages sent using SMS or MMS) and Internet-based messages (eg, XMPP, SIMPLE or IMPS). Messages transmitted using () refers to both.

RF circuit 108, touch screen 112, display controller 156, contact module 130, graphic module 132, handwriting input module 157, text input module 134, GPS module 135, In conjunction with the guidance module 154 and the music player module 146, the exercise support module 142 creates exercises (eg, with time, distance, and/or calorie consumption goals); Communicates with athletic sensors (sports devices); Receive motion sensor data; Calibrate sensors used to monitor exercise; Select and play music for exercise; It contains executable instructions to display, store and transmit athletic data.

Along with the touch screen 112, the display controller 156, the light sensor(s) 164, the light sensor controller 158, the contact module 130, the graphics module 132 and the image management module 144, the camera The module 143 captures still images or video (including a video stream) and stores them in the memory 102, modifies the properties of the still image or video, or Contains executable instructions that cause to be deleted.

Together with the touch screen 112, the display controller 156, the contact module 130, the graphic module 132, the handwriting input module 157, the text input module 134 and the camera module 143, an image management module ( 144) is executable to arrange still and/or video images, modify (e.g., edit), or otherwise manipulate, label, delete, present (e.g., in a digital slide show or album), and store Contains instructions.

Along with the RF circuit 108, the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132, the handwriting input module 157 and the text input module 134, the browser module ( 147) to browse the Internet according to user instructions-including retrieving, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to the web pages- Contains executable instructions that do.

RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, email client module 140 ) And the browser module 147, the calendar module 148 creates and displays calendars and data associated with the calendars (e.g., calendar entries, to-do lists, etc.) according to user instructions, Contains executable instructions that allow you to modify and save.

RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134 and browser module 147 Along with, the widget modules 149 can be downloaded and used by the user (e.g., weather widget 149-1, stock widget 149-2, calculator widget 149-3), or alarm clock widget ( 149-4) and dictionary widget 149-5), or mini-applications that can be created by the user (eg, user-generated widget 149-6). In some embodiments, the widget includes a Hypertext Markup Language (HTML) file, a Cascading Style Sheets (CSS) file, and a JavaScript file. In some embodiments, the widget includes an Extensible Markup Language (XML) file and a JavaScript file (eg, Yahoo! widgets).

RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134 and browser module 147 Together, the widget generator module 150 may be used to generate widgets by a user (eg, to change a user-specific part of a web page into a widget).

Along with the touch screen 112, the display system controller 156, the contact module 130, the graphic module 132, the handwriting input module 157, and the text input module 134, the search module 151 provides user commands. Contains executable instructions to search for text, music, sound, image, video, and/or other files in memory 102 matching one or more search criteria (e.g., one or more user-specific search terms) according to .

With touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108 and browser module 147, The video and music player module 152 includes executable instructions that allow a user to download and play recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, and videos (e.g. , On the touch screen 112 or on an externally connected display through the external port 124), to display, screen, or otherwise play. In some embodiments, device 100 may include the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

Along with the touch screen 112, the display controller 156, the contact module 130, the graphic module 132, the handwriting input module 157 and the text input module 134, the memo module 153 is It contains executable instructions that allow you to create and manage notes, to-do lists, and so on.

RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, GPS module 135 And the browser module 147, the map module 154, according to user instructions, maps and data associated with the maps (e.g., driving direction; stores and other points of interest at or near a specific location). And other location-based data).

Touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, handwriting input module 157, text Together with the input module 134, the email client module 140 and the browser module 147, the online video module 155 allows a user to access, browse, and access online videos in one or more file formats, such as H.264, and Receive (e.g., by streaming and/or download), play (e.g., on a touch screen or on an externally connected display via external port 124), send an email with a link to a specific online video, and , Contains commands that allow it to be managed otherwise. In some embodiments, an instant messaging module 141 is used rather than an email client module 140 to send a link to a particular online video.

Each of the identified modules and applications is an execution for performing one or more functions described above and methods described herein (e.g., computer-implemented methods and other information processing methods described herein). Corresponds to a set of possible instructions. These modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures or modules, so various subsets of these modules may be combined or otherwise rearranged in various embodiments. . In some embodiments, memory 102 may store a subset of the identified modules and data structures. Further, the memory 102 may store additional modules and data structures not described above.

In some embodiments, device 100 is a device in which operation of a predefined set of functions on the device is performed exclusively through a touch screen and/or a touchpad. By using the touch screen and/or touchpad as the primary input control device for operation of the device 100, the number of physical input control devices (such as push buttons, dials, etc.) on the device 100 can be reduced.

2 shows a portable multifunction device 100 with a touch screen 112 in accordance with some embodiments. The touch screen may display one or more graphics within the user interface (UI) 200. In this embodiment as well as other embodiments described below, the user may, for example, have one or more fingers 202 (not drawn to scale in the drawings) or one or more stylus 203 (scaled in the drawings). Not shown) can be used to select one or more of the graphics by making a gesture on the graphics. In some embodiments, the selection of one or more graphics occurs when the user stops contacting the one or more graphics. In some embodiments, the gesture may include one or more taps of a finger in contact with the device 100, one or more swipes (left to right, right to left, up and/or down), and/or Rolling (right to left, left to right, up and/or down) may be included. In some embodiments, careless contact with the graphic may not select the graphic. For example, the swipe gesture for sweeping over an application icon may not select a corresponding application when the gesture corresponding to the selection is a tap.

Device 100 may also include one or more physical buttons, such as a “home” or menu button 204. As previously described, the menu button 204 can be used to navigate to any application 136 in the set of applications that can run on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 112.

In one embodiment, the device 100 includes a touch screen 112, a menu button 204, a push button 206 to power on/off the device and lock the device, a volume control button(s) 208, It includes a subscriber identity module (SIM) card slot 210, a headset jack 212, and an external docking/charging port 124. Push button 206 changes power on/off to the device by pressing the button and holding the button pressed for a predefined time interval; Locking the device by pressing the button and releasing the button before a predefined time interval elapses; And/or may be used to unlock the device or initiate an unlock process. In an alternative embodiment, device 100 may also allow verbal input for activation or deactivation of some functions via microphone 113.

3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface in accordance with some embodiments. Device 300 need not be portable. In some embodiments, the device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as children's learning toys), a game system, a telephone device, or a control device (e.g. For example, a household or industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses for interconnecting these components. 320). Communication buses 320 may include circuitry (sometimes referred to as a chipset) that interconnects system components and controls communication between them. Device 300 includes an input/output (I/O) interface 330 that includes a display 340, which is typically a touch screen display. The I/O interface 330 may also include a keyboard and/or mouse (or other pointing device) 350 and a touchpad 355. Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; Non-volatile memory, for example one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 370 may optionally include one or more storage devices located remotely from CPU(s) 310. In some embodiments, memory 370 may include programs, modules and data structures similar to programs, modules and data structures stored in memory 102 of portable multifunction device 100 (FIG. 1), or Save a subset of it. In addition, the memory 370 may store additional programs, modules, and data structures not present in the memory 102 of the portable multifunction device 100. For example, the memory 370 of the device 300 includes a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disk authoring module 388, and/or Alternatively, the spreadsheet module 390 may be stored, while the memory 102 of the portable multifunction device 100 (FIG. 1) may not store these modules.

Each of the elements identified above in FIG. 3 may be stored in one or more of the previously mentioned memory devices. Each of the identified modules corresponds to a set of instructions for performing the functions described above. The identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures or modules, so various subsets of these modules may be combined or otherwise different in various embodiments. Can be rearranged. In some embodiments, memory 370 may store a subset of the identified modules and data structures. Also, the memory 370 may store additional modules and data structures not described above.

4 shows a device (e.g., device 300) having a touch-sensitive surface 451 (e.g., a tablet or touchpad 355, FIG. 3) separated from a display 450 (e.g., touch screen display 112). , Figure 3) shows an exemplary user interface. While many examples that follow will be given with reference to inputs on the touch screen display 112 (the touch-sensitive surface and display are combined), in some embodiments, the device has a touch independent of the display as shown in FIG. 4. Detect inputs on the sensitive surface. In some embodiments, the touch-sensitive surface (eg, 451 in FIG. 4) has a major axis (eg, 452 in FIG. 4) that corresponds to a major axis (eg, 453 in FIG. 4) on the display (eg, 450 ). According to these embodiments, the device makes contacts with the touch-sensitive surface 451 at locations corresponding to individual locations on the display (e.g., in FIG. 4, 460 corresponds to 468, and 462 corresponds to 470). (E.g., 460 and 462 in FIG. 4) are detected. In this way, user inputs (e.g., contacts 460, 462 and their movements) detected by the device on the touch-sensitive surface (e.g., 451 in FIG. 4) are Used by the device to manipulate the user interface on the display of the multifunction device (eg, 450 in FIG. 4). It should be understood that similar methods may be used for other user interfaces described herein.

Attention is now directed to embodiments of handwriting input methods and user interfaces (“UI”) that may be implemented on a multifunction device (eg, device 100).

5 shows an I/O interface module 500 (e.g., I/O interface 330 of FIG. 3 or I/O subsystem of FIG. 1) to provide handwriting input capabilities on a device in accordance with some embodiments. 106)). As shown in FIG. 5, the handwriting input module 157 includes an input processing module 502, a handwriting recognition module 504, and a result generation module 506. In some embodiments, the input processing module 502 includes a segmentation module 508 and a normalization module 510. In some embodiments, the result generation module 506 includes a radical clustering module 512 and one or more language models 514.

In some embodiments, the input processing module 502 communicates with the I/O interface module 500 (e.g., the I/O interface 330 of FIG. 3 or the I/O subsystem 106 of FIG. 1) to Receive handwriting inputs from the user. Handwriting is entered through any suitable means, such as the touch sensitive display system 112 of FIG. 1 and/or the touchpad 355 of FIG. 3. The handwriting inputs include data representing each stroke provided by the user within a predetermined handwriting input area within the handwriting input UI. In some embodiments, the data representing each stroke of the handwriting input includes the start and end positions of a sustained contact within the handwriting input area (e.g., contact between the user's finger or stylus and the touch-sensitive surface of the device), It includes data such as intensity profiles, and motion paths. In some embodiments, I/O interface module 500 communicates sequences 516 of handwritten strokes with associated temporal and spatial information to input processing module 502 in real time. At the same time, the I/O interface module also provides real-time rendering 518 of handwritten strokes within the handwriting input area of the handwriting input user interface as visual feedback on the user's input.

In some embodiments, as data representing each handwritten stroke is received by the input processing module 502, temporal and sequence information associated with a number of consecutive strokes is also recorded. For example, the data optionally includes a stack showing the shape and size of the strokes along the writing direction of the entire handwriting input, the spatial saturation of individual strokes with individual stroke sequence numbers, and relative spatial positions, and the like. In some embodiments, the input processing module 502 has instructions to render the received strokes on the device's display 518 (e.g., display 340 of FIG. 3 or touch sensitive display 112 of FIG. 1 ). Back to the I/O interface modules 500. In some embodiments, the rendering of the received strokes is animated to provide a visual effect that mimics the actual progress of writing on a writing surface (eg, a sheet of paper) using a writing instrument (eg, a pen). In some embodiments, the user is optionally allowed to specify a pen-tip style, color, texture, etc. of the rendered strokes.

"는 2개의 부수들, 즉, 왼쪽 부수 "口" 및 오른쪽 부수 "斤"를 사용하여 구성된다.In some embodiments, the input processing module 502 processes the currently accumulated strokes in the handwriting input area and assigns the strokes to one or more recognition units. In some embodiments, each recognition unit corresponds to a character to be recognized by the handwriting recognition model 504. In some embodiments, each recognition unit corresponds to an output character or number of copies to be recognized by the handwriting recognition model 504. A minority is a repetitive component found in many complex slogans. The compound slogan may include two or more copies arranged according to a general layout (eg, a left and right layout, a top and bottom layout, etc.). In one example, a single Chinese character "

"Is constructed using two radicals, namely, the left radical "口" and the right radical "斤".

"에 대한 획들을 분할하는 경우, 분할 모듈(508)은 선택적으로 필기 입력의 좌측에 클러스터링된 획들을 (즉, 왼쪽 부수 "口"에 대한) 하나의 인식 유닛에 배정하고, 필기 입력의 우측에 클러스터링된 획들을 (즉, 오른쪽 부수 "斤"에 대한) 다른 인식 유닛에 배정한다. 대안적으로, 분할 모듈(508)은 또한 모든 획들을 (즉, 문자 "

"에 대한) 단일 인식 유닛에 배정할 수 있다.In some embodiments, the input processing module 502 relies on the segmentation module to allocate or divide the currently accumulated handwritten strokes into one or more recognition units. For example, the handwritten character "

When dividing the strokes for ", the segmentation module 508 selectively assigns the clustered strokes to the left of the handwriting input (ie, for the left side number "口") to one recognition unit, and to the right of the handwriting input. The clustered strokes are assigned to different recognition units (ie, for the right side "斤.") Alternatively, the segmentation module 508 also assigns all of the strokes (ie, the letter "

Can be assigned to a single recognition unit.

In some embodiments, the segmentation module 508 creates a segmentation grid 520 by dividing the currently accumulated handwriting input (eg, one or more handwritten strokes) into groups of recognition units in a number of different ways. For example, it is assumed that a total of 9 strokes have been accumulated so far in the handwriting input area. According to the first segmentation chain of the segmentation grid 520, stroke 1, stroke 2, and stroke 3 are grouped into a first recognition unit 522, and stroke 4, stroke 5, and stroke 6 become a second recognition unit 526. Are grouped. According to the second division chain of the division grid 520, all of the strokes 1 to 9 are grouped into one recognition unit 526.

In some embodiments, each segmentation chain is given a segmentation score to measure the likelihood that a particular segmentation chain is the correct segmentation of the current handwriting input. In some embodiments, factors that are optionally used to calculate the split score of each splitting chain are the absolute and/or relative size of the stroke, the stroke's in various directions (e.g., x, y, z directions). Relative and/or absolute span, fluctuations in the saturation level of the stroke and/or its average, absolute and/or relative distances to adjacent strokes, absolute and/or relative positions of the strokes, the strokes being entered It includes the sequence or sequence, the duration of each stroke, the variations and/or their average in the speed (or tempo) at which each stroke was entered, the intensity profile of each stroke along the length of the stroke, and so on. In some embodiments, one or more functions or transforms are optionally applied to one or more of these factors to generate split scores of different split chains in split grid 520.

In some embodiments, after the segmentation module 508 segments the current handwriting input 516 received from the user, the segmentation module 508 passes the segmentation grid 520 to the normalization module 510. In some embodiments, the normalization module 510 is an input image (e.g., input images 528) for each recognition unit (e.g., recognition units 522, 524, 526) specified in the segmentation grid 520. ). In some embodiments, the normalization module performs the necessary or desired normalization (e.g., stretching, cropping, down-sampling or up-sampling) on the input image so that the input image is provided as input to the handwriting recognition model 504. To be able to be. In some embodiments, each input image 528 includes strokes assigned to one individual recognition unit, and corresponds to one character or number of copies to be recognized by the handwriting recognition module 504.

In some embodiments, the input images generated by the input processing module 502 do not include any temporal information associated with individual strokes, but are represented by spatial information (e.g., density and position of pixels in the input image). Information) is preserved in the input image. A handwriting recognition model trained solely on spatial information of training writing samples is capable of only handwriting recognition based on spatial information. As a result, the handwriting recognition model does not exhaustively enumerate all possible permutations of stroke orders and stroke directions for all characters in its vocabulary (i.e. all output classes) during training. Is independent. Indeed, in some embodiments, the handwriting recognition module 502 does not distinguish between a pixel belonging to one stroke and a pixel belonging to another stroke in the input image.

As will be described in more detail later (e.g., with respect to FIGS. 25A to 27), in some embodiments, in order to improve the recognition accuracy, without compromising the stroke order and stroke direction independence of the recognition model, some time -Induction stroke distribution information is reintroduced into a wholly-spatial handwriting recognition model.

In some embodiments, the input image generated by the input processing module 502 for one recognition unit does not overlap with the input image of any other recognition unit in the same segmentation chain. In some embodiments, input images generated for different recognition units may overlap slightly. In some embodiments, some overlap between the input images is a handwriting input that includes cursive and/or run-on characters (e.g., one stroke joins two adjacent characters). Is allowed to recognize.

In some embodiments, some normalization is performed prior to segmentation. In some embodiments, the functions of splitting module 508 and normalizing module 510 may be performed by the same module or two or more different modules.

In some embodiments, as the input image 528 of each recognition unit is provided as an input to the handwriting recognition model 504, the handwriting recognition model 504 may determine that the recognition unit is a repertoire or vocabulary of the handwriting recognition model 504. (I.e., a list of all characters and copies recognizable by the handwriting recognition module 504) produces an output consisting of different possibilities to be an individual output character. As described in more detail below, the handwriting recognition model 504 has been trained to recognize a large number of characters in multiple scripts (eg, at least three non-overlapping scripts encoded by the Unicode standard).

Examples of non-overlapping scripts include Latin scripts, Chinese characters, Arabic characters, Persian, Cyrillic, and artificial scripts such as emoji characters. In some embodiments, the handwriting recognition model 504 generates one or more output characters for each input image (i.e., for each recognition unit), and for each output character based on the confidence level associated with the character recognition. Assign a recognition score.

In some embodiments, the handwriting recognition model 504 generates a candidate lattice 530 according to the segmentation lattice 520, wherein the segmentation lattice 520 (e.g., individual recognition units 522, 524, 526) Each arc in the corresponding) division chain is divided into one or more candidate arcs (e.g., arcs 532, 534, 536, 538, 540 respectively corresponding to individual output characters) within the candidate grid 530. Expands. Each candidate chain in candidate grid 530 is scored according to an individual split score of the split chain on which the candidate chain is based, and recognition scores associated with the output characters in the character chain.

In some embodiments, after the handwriting recognition model 504 generates output characters from the input images 528 of the recognition units, the candidate grid 530 is passed to the result generation module 506 and the currently accumulated handwriting input. Produces one or more recognition results for 516.

In some embodiments, the result generation module 506 utilizes the minor clustering module 512 to combine one or more radicals in the candidate chain into a compound character. In some embodiments, the result generation module 506 uses one or more language models 514 to determine whether the character chain in the candidate grid 530 is a probable sequence in a particular language represented by the language models. Is determined. In some embodiments, the result generation module 506 generates a revised candidate grating 542 by combining two or more arcs in the candidate grating 530 or removing specific arcs.

In some embodiments, the result generation module 506, as modified (e.g., augmented or reduced) by the collateral clustering module 512 and language models 514, the recognition scores of the output characters in the character sequence. Based on the revised candidate grid 542, a unified recognition score is generated for each character sequence (e.g., character sequences 544, 546) still remaining in the revised candidate grid 542. In some embodiments, the result generation module 506 ranks the different character sequences remaining in the revised candidate grid 542 based on their combined recognition scores.

" 및 "

")을 필기 입력 인터페이스의 후보 디스플레이 영역에서 디스플레이한다. 일부 실시예들에서, I/O 인터페이스 모듈은 사용자를 위해 다수의 인식 결과들 (예컨대, "

" 및 "

")을 디스플레이하고, 사용자가 인식 결과를 선택하여 관련 애플리케이션에 대한 텍스트 입력으로서 입력할 수 있게 한다. 일부 실시예들에서, I/O 인터페이스 모듈은 인식 결과의 사용자 확인의 표시들 또는 기타 입력들에 응답하여 최상위 인식 결과(예컨대, "

")를 자동으로 입력한다. 최상위 결과의 효과적인 자동 입력은 입력 인터페이스의 효율을 개선시킬 수 있고, 보다 양호한 사용자 경험을 제공한다.In some embodiments, the result generation module 506 transmits the top-ranked character sequences as ranked recognition results 548 to the I/O interface module 500 for display to the user. In some embodiments, the I/O interface module 500 includes the received recognition results 548 (e.g., "

"And"

") in the candidate display area of the handwriting input interface. In some embodiments, the I/O interface module provides multiple recognition results for the user (eg, "

"And"

") and allows the user to select the recognition result and enter it as text input to the relevant application. In some embodiments, the I/O interface module includes indications of user confirmation of the recognition result or other inputs. In response to the highest recognition result (e.g., "

Automatically input "). Effective automatic input of the top result can improve the efficiency of the input interface and provide a better user experience.

In some embodiments, the result generation module 506 changes the combined recognition scores of the candidate chains using other factors. For example, in some embodiments, the result generation module 506 optionally maintains a log of the most frequently used characters for a particular user or a number of users. If specific candidate characters or character sequences are found in the list of most frequently used characters or character sequences, the result generation module 506 optionally raises the combined recognition scores of the specific candidate characters or character sequences.

In some embodiments, the handwriting input module 157 provides real-time updates to the recognition results displayed to the user. For example, in some embodiments, for each additional stroke input by the user, the input processing module 502 selectively subdivides the currently accumulated handwriting input, and an input image provided to the handwriting recognition model 504 Revises the field and the division grid. Eventually, the handwriting recognition model 504 optionally revises the candidate grid provided to the result generation module 506. As a result, the result generation module 506 optionally updates recognition results presented to the user. As used herein, real-time handwriting recognition refers to handwriting recognition in which handwriting recognition results are presented to a user immediately or within a short period of time (eg, within tens of milliseconds to several seconds). Real-time handwriting recognition is not done at once after the current user session from a recorded image that is stored for later retrieval, but is initiated immediately and is performed substantially simultaneously with the reception of the handwriting input (e.g., offline optical character It is different from offline recognition (as in OCR applications). Further, since offline character recognition is performed without any temporal information regarding individual strokes and stroke sequences, segmentation is performed without the benefit of such information. Further clarification between candidate characters that appear to be similar is also done without the benefit of this temporal information.

In some embodiments, the handwriting recognition model 504 is implemented as a convolutional neural network (CNN). 6 shows an exemplary convolutional neural network 602 trained on a multiscript training corpus 604 that includes writing samples for characters in multiple non-overlapping scripts.

As shown in FIG. 6, the convolutional neural network 602 includes an input plane 606 and an output plane 608. Between the input plane 606 and the output plane 608 a plurality of convolutional layers 610 (e.g., a first convolutional layer 610a, zero or more intermediate convolutional layers (not shown)), and a last convolutional layer. Layer 610n) is present. Each convolutional layer 610 is followed by a separate sub-sampling layer 612 (e.g., a first sub-sampling layer 612a, zero or more intermediate sub-sampling layers (not shown)) and a last sub-sampling layer ( 612n)). A hidden layer 614 exists after the convolutional and sub-sampling layers and just before the output plane 608. Hidden layer 614 is the last layer before output plane 608. In some embodiments, a kernel layer 616 (e.g., a first kernel layer 616a, zero or more intermediate kernel layers (not shown), and a last kernel) in front of each convolutional layer 610 to improve output efficiency. Layer 612n) is inserted.

6, the input plane 606 receives the input image 614 (e.g., handwritten character or number of copies) of the handwriting recognition unit, and the output plane 608 is the recognition unit is a separate output class (e.g., Outputs a set of probabilities indicating a probability of belonging to a specific character in the output character set configured to be recognized by the neural network. Overall the output classes of the neural network (or the output character set of the neural network) are also referred to as the repertoire or vocabulary of the handwriting recognition model. The convolutional neural network described herein can be trained to have a repertoire of tens of thousands of characters.

When the input image 614 is processed through different layers of the neural network, different spatial features embedded in the input image 614 are extracted by the convolutional layers 610. Each convolutional layer 610 is also referred to as a set of feature maps, and acts as filters to select specific features in the input image 614 to differentiate between images corresponding to different characters. The sub-sampling layers 612 ensure that features are captured from the input image 614 at an increasingly larger scale. In some embodiments, the sub-sampling layers 612 are implemented using a max-pooling technique. The max-pooling layers create position invariance over larger localized regions, down-samples the output image of the preceding convolutional layer by a factor of Kx and Ky along each direction, where Kx and Ky are of the max-pooling rectangle. It's the size. Max-pooling results in faster convergence speed by selecting good invariant features, which improves generalization performances. In some embodiments, sub-sampling is achieved using other methods.

In some embodiments, after the last set of convolutional layer 610n and sub-sampling layer 612n and before output plane 608, there is a fully-connected layer, i.e., hidden layer 614. exist. The fully-connected concealed layer 614 is a multi-layer perceptron that fully connects the nodes in the output plane 608 with the nodes in the last sub-sampling layer 612n. Hidden layer 614 occupies output images received from that layer before and during logistic regression reaching one of the output characters in output layer 608.

During training of the convolutional neural network 602, features in the convolutional layers 610 and individual weights associated with the features are minimized so that discrimination errors are minimized for write samples using known output classes in the training corpus 604. As well as the weights associated with the parameters in the hidden layer 614 are tuned. Once the convolutional neural network 602 has been trained and the optimal set of parameters and associated weights have been established for the different layers in the network, using the convolutional neural network 602, which is not part of the training corpus 604. Recognize new writing samples 618, for example input images generated based on real-time handwriting input received from a user.

As described herein, the convolutional neural network of the handwriting input interface is trained using a multiscript training corpus to enable multiscript or mixed script handwriting recognition. In some embodiments, the convolutional neural network is trained to recognize a large repertoire of 30,000 to more than 60,000 characters (eg, all characters encoded by the Unicode standard). Most state-of-the-art handwriting recognition systems are based on hidden Markov methods (HMMs) that are stroke order dependent. In addition, most existing handwriting recognition models are language-specific, and dozens of characters (e.g., English alphabet, Greek alphabet, characters such as 10 numbers in total) to thousands of characters (e.g., most commercially available) Includes a small repertoire up to a set of Chinese characters). As such, the universal recognizer described herein can handle dozens of times more characters than most existing systems.

Some conventional handwriting systems may include several individually trained handwriting recognition models, each handwriting recognition model customized for a specific language or small set of characters. The writing sample is propagated through different recognition models until a distinction is made. For example, a handwriting sample may be provided as a series of consecutive language-specific or script-specific character recognition models, and if the handwriting sample cannot be clearly distinguished by the first recognition model, it is provided as the next recognition model. However, the next recognition model attempts to distinguish handwriting samples within its own repertoire. This approach to classification is time consuming, and memory requirements increase rapidly with each additional recognition model that needs to be used.

Other modern models require the user to specify a preferred language, and use the selected handwriting recognition model to differentiate the current input. Not only are these implementations cumbersome to use and consume significant memory, they cannot be used to recognize mixed language input. It is impractical to ask the user to switch language preferences while entering mixed language or mixed script input.

The multiscript or universal recognizer described herein addresses at least some of the above problems with conventional recognition systems. 7 shows a handwriting recognition module (e.g., convolutional neural network) using a large multi-script training corpus, so that the handwriting recognition module can subsequently be used to provide real-time multi-language and multi-script handwriting recognition for the user's handwriting input. ) Is a flowchart of an exemplary process 700 for training.

In some embodiments, after training of the handwriting recognition model is performed on the server device, the trained handwriting recognition model is provided to the user device. The handwriting recognition model selectively performs real-time handwriting recognition locally on the user device without requiring additional support from the server. In some embodiments, both training and recognition are provided on the same device. For example, the server device may receive a user's handwriting input from the user device, perform handwriting recognition, and transmit recognition results to the user device in real time.

In exemplary process 700, in a device having one or more processors and memory, the device builds a multiscript handwriting recognition model based on the space-derived features (e.g., stroke order independent features) of the multiscript training corpus. Train (702). In some embodiments, the space-derived features of the multiscript training corpus are stroke order independent and stroke direction independent (704). In some embodiments, training of the multiscript handwriting recognition model is independent of temporal information associated with individual strokes in the handwriting samples (706). Specifically, the images of the handwriting samples are normalized to a predetermined size, and the images do not contain any information about the order in which individual strokes are input to form an image. In addition, the images also contain no information about the direction in which individual strokes are input to form the image. In fact, during training, features are extracted from handwritten images regardless of how the images are temporally formed by individual strokes. Thus, during recognition, time information related to individual strokes is not required. As a result, recognition provides firmly consistent recognition results despite delayed, out-of-order strokes in handwriting input, and arbitrary stroke directions.

In some embodiments, the multiscript training corpus includes handwriting samples corresponding to characters of at least three non-overlapping scripts. As shown in Fig. 6, the multiscript training corpus contains handwriting samples collected from many users. Each handwriting sample corresponds to one character of an individual script displayed in the handwriting recognition model. In order to properly train the handwriting recognition model, the training corpus contains a large number of writing samples for each character of the scripts displayed in the handwriting recognition model.

In some embodiments, the at least three non-overlapping scripts include 708 Chinese characters, emoji characters, and Latin scripts. In some embodiments, the multiscript handwriting recognition model has (710) at least 30,000 output classes representing 30,000 characters spanning at least three non-overlapping scripts.

In some embodiments, the multiscript training corpus is applied to each character of all Chinese characters encoded in the Unicode standard (e.g., a significant portion or all of all CJK (Chinese-Japanese-Korean) unified ideographs). Includes individual write samples for. The Unicode standard defines a total of about 74,000 CJK unified ideographic characters. The CJK unified ideographic block (4E00-9FFF) contains 20,941 basic Chinese characters used not only in Chinese, but also in Japanese, Korean and Vietnamese. In some embodiments, the multiscript training corpus includes writing samples for all characters in the basic block of CJK unified ideographic characters. In some embodiments, the multiscript training corpus further includes writing samples for CJK minors that can be used to structurally construct one or more complex Chinese characters. In some embodiments, the multiscript training corpus further includes writing samples for less frequently used Chinese characters, such as Chinese characters encoded with one or more of the CJK unified ideographic extensions.

In some embodiments, the multiscript training corpus further includes individual writing samples for each character of all characters in the Latin script encoded by the Unicode standard. The letters in the basic Latin script include Latin lowercase and uppercase letters, as well as various basic symbols and numbers commonly used on a standard Latin keyboard. In some embodiments, the multiscript training corpus further includes characters in an extended Latin script (eg, various accent forms of basic Latin characters).

"에 대한 필기 샘플이다. 그 외의 이모지 캐릭터들에는, 상이한 감정들(예컨대, 행복, 슬픔, 분노, 당황, 충격, 웃음, 울기, 좌절 등), 상이한 개체들 및 캐릭터들(예컨대, 고양이, 개, 토끼, 심장, 과일, 눈, 입술, 선물, 꽃, 촛불, 달, 별 등), 그리고 상이한 행동들(예컨대, 악수, 키스, 달리기, 댄스, 점프, 수면, 식사, 만남, 사랑, 좋아함, 투표 등) 등을 나타내는 아이콘들의 카테고리들이 포함된다. 일부 실시예들에서, 이모지 캐릭터에 대응하는 필기 샘플에서의 획들은 대응하는 이모지 캐릭터를 형성하는 실제 라인들의 간결화되고 그리고/또는 스타일화된(stylized) 라인들이다. 일부 실시예들에서, 각 디바이스 또는 애플리케이션은 동일한 이모지 캐릭터에 대한 상이한 설계를 이용할 수 있다. 예를 들어, 2명의 사용자들로부터 수신된 필기 입력들이 실질적으로 동일할지라도, 여성 사용자에게 제시되는 스마일상 이모지 캐릭터는 남성 사용자에게 제시되는 스마일상 이모지 캐릭터와는 다를 수 있다.In some embodiments, the multiscript training corpus includes writing samples corresponding to each character of an artificial script that is not associated with any natural human language. For example, in some embodiments, a set of emoji characters is optionally defined in an emoji script, and writing samples corresponding to each of the emoji characters are included in the multiscript training corpus. For example, a hand-drawn heart-shaped symbol is a handwritten sample of the emoji character "♥" in the training corpus. Similarly, a hand-drawn smiling face (e.g., two dots on an inverted arc) appears as an emoji character in the training corpus.

"It is a handwritten sample of. Other emoji characters include different emotions (eg, happiness, sadness, anger, embarrassment, shock, laughter, crying, frustration, etc.), different entities and characters (eg, cat, Dogs, rabbits, heart, fruit, eyes, lips, gifts, flowers, candles, moon, stars, etc.) and different actions (e.g. shaking hands, kissing, running, dancing, jumping, sleeping, eating, meeting, love, liking) , Voting, etc.), etc. In some embodiments, the strokes in the handwriting sample corresponding to the emoji character are simplified and/or stylized of the actual lines forming the corresponding emoji character. In some embodiments, each device or application may use a different design for the same emoji character, for example, even if the handwriting inputs received from two users are substantially the same. , The smiley emoji character presented to a female user may be different from the smiley emoji character presented to a male user.

In some embodiments, the multiscript training corpus also includes other scripts, e.g., Greek script (e.g., including Greek characters and symbols), Cyrillic script, Hebrew script, and one encoded according to the Unicode standard. Includes writing samples for characters in the above other scripts. In some embodiments, at least three non-overlapping scripts included in the multiscript training corpus include Chinese characters, emoji characters, and characters in a Latin script. Chinese characters, emoji characters, and characters in Latin script are naturally non-overlapping scripts. Many different scripts can overlap each other for at least some characters. For example, some letters (eg, A, Z) in Latin scripts may be found in many other scripts (eg, Greek and Cyrillic). In some embodiments, the multiscript training corpus includes Chinese characters, Arabic script and Latin script. In some embodiments, the multiscript training corpus includes other combinations of overlapping and/or non-overlapping scripts. In some embodiments, the multiscript training corpus includes writing samples for all characters encoded by the Unicode standard.

As shown in Figure 7, in some embodiments, to train a multiscript handwriting recognition model, the device provides handwriting samples of the multiscript training corpus to a single convolutional neural network with a single input plane and a single output plane. Do (712). The device uses a convolutional neural network to separate space-derived features (e.g., stroke-order independent features) and space of the handwriting samples to distinguish characters of at least three non-overlapping scripts displayed in the multiscript training corpus. -Determine individual weights for the derived features (714). The multiscript handwriting recognition model differs from conventional multiscript handwriting recognition models in that a single handwriting recognition model with a single input plane and a single output plane is trained using all samples in the multiscript training corpus. Multiscript training, without relying on individual sub-networks each handling a small subset of the training corpus (e.g., sub-networks each trained to recognize characters in a particular script or used in a particular language) A single convolutional neural network is trained to distinguish all characters displayed in the corpus. In addition, a single convolutional neural network is applied to multiple non-overlapping scripts, rather than characters of several overlapping scripts, such as Latin script and Greek script (e.g., having overlapping characters A, B, E, Z, etc. The spanning is trained to distinguish a large number of characters.

In some embodiments, the device provides 716 real-time handwriting recognition for the user's handwriting input using a multiscript handwriting recognition model trained on the space-guided features of the multiscript training corpus. In some embodiments, providing real-time handwriting recognition for the user's handwriting input includes continuously revising the recognition output for the user's handwriting input as the user continues to provide additions or revisions of the handwriting input. . In some embodiments, providing real-time handwriting recognition for the user's handwriting input further comprises providing 718 a multiscript handwriting recognition model to the user device, wherein the user device receives handwriting input from the user. Receive, and locally perform handwriting recognition on the handwriting input based on the multi-script handwriting recognition model.

In some embodiments, the device provides a multi-script handwriting recognition model to a plurality of devices in which individual input languages of the plurality of devices do not overlap, and the multi-script handwriting recognition model is configured with each user device on each of the plurality of devices. Used for handwriting recognition of different languages involved. For example, if a multiscript handwriting recognition model is trained to recognize characters in many different scripts and languages, the same handwriting recognition model will be used worldwide to provide handwriting input for any of these input languages. Can be used. The first device for a user who wants to input only in English and Hebrew may provide a handwriting input function using the same handwriting recognition model as the second device for other users who want to input only in Chinese and emoji characters. Instead of requiring the user of the first device to separately install an English handwriting input keyboard (e.g., implemented with an English-specific handwriting recognition model) and a separate Hebrew handwriting input keyboard (e.g., implemented with a Hebrew-specific handwriting recognition model) In addition, the same universal multi-script handwriting recognition model can be installed and used once on the first device to provide a handwriting input function for both English and Hebrew as well as mixed input of the two languages. In addition, instead of requiring the second user to install a Chinese handwriting input keyboard (e.g., implemented with a Chinese-specific handwriting recognition model) and a separate Emoji handwriting input keyboard (e.g., implemented with an emoji handwriting recognition model) The same universal multi-script handwriting recognition model can be installed and used once on the second device to provide a handwriting input function for both Chinese and emoji as well as mixed input of the two scripts. Handling a large repertoire across multiple scripts (e.g., a significant portion or all of the characters encoded in nearly 100 different scripts) using the same multi-script handwriting model is a significant burden on users and some of the device providers. Without improving the usefulness of the recognizer.

Multiscript handwriting recognition model training using a large multiscript training corpus is different from conventional HMM-based handwriting recognition systems and does not rely on temporal information associated with individual strokes of characters. Also, the resource and memory requirements for a multiscript recognition system do not increase linearly with increasing symbols and languages covered by the multiscript recognition system. For example, in a conventional handwriting system, increasing the number of languages means adding another independently trained model, and the memory requirements will be at least doubled to accommodate the increasing capabilities of the handwriting recognition system. In contrast, if the multiscript model is trained by the multiscript training corpus, increasing the language coverage requires retraining the handwriting recognition model with additional handwriting samples, and increasing the size of the output plane, but that The increase is very gentle. Assuming that the multiscript training corpus contains handwriting samples corresponding to n different languages and the multiscript handwriting recognition model occupies a memory of size m , if the language coverage is increased to N languages (N>n) , The device retrains the multiscript handwriting recognition model based on the space-derived features of the second multiscript training corpus, the second multiscript training corpus comprising second handwriting samples corresponding to N different languages. . The change in M / m remains substantially constant within the range of 1 to 2, where the change in N / n is 1 to 100. Once the multiscript handwriting recognition model is retrained, the device can use the retrained multiscript handwriting recognition model to provide real-time handwriting recognition for the user's handwriting input.

8A and 8B illustrate exemplary user interfaces for providing real-time multiscript handwriting recognition and input on a portable user device (eg, device 100). 8A and 8B, a handwriting input interface 802 is displayed on a touch-sensitive display screen (eg, touch screen 112) of a user device. The handwriting input interface 802 includes a handwriting input area 804, a candidate display area 806, and a text input area 808. In some embodiments, the handwriting input interface 802 further includes a plurality of control elements, each of which may be invoked to cause the handwriting input interface to perform a predetermined function. As shown in FIG. 8A, a delete button, a space button, a carriage return or enter button, and a keyboard switch button are included in the handwriting input interface. Other control elements are possible and can optionally be provided in the handwriting input interface to suit each different application utilizing the handwriting input interface 802. The layout of the different components of the handwriting input interface 802 is merely exemplary and may differ for different devices and different applications.

"), 일부 필기 영어 문자들(예컨대, "Happy"), 및 손으로 그려진 이모지 캐릭터(예컨대, 스마일상)을 포함하는 다수의 필기 획들을 제공하였다. 필기 문자들은 필기 입력 영역(804)에서 다수의 라인들(예컨대, 2개의 라인들)로 분포된다.In some embodiments, the handwriting input area 804 is a touch sensitive area for receiving a handwriting input from a user. The sustained contact within the handwriting input area 804 on the touch screen and its associated motion path is registered as a handwritten stroke. In some embodiments, the handwritten stroke registered by the device is visually rendered within the handwriting input area 804 at the same locations tracked by the sustained contact. As shown in FIG. 8A, in the handwriting input area 804, the user selects some handwritten Chinese characters (eg, “

"), some handwritten English characters (eg, "Happy"), and a number of handwritten strokes including a hand drawn emoji character (eg, on a smiley). The handwritten characters are provided in the handwriting input area 804. It is distributed over a number of lines (eg, two lines).

In some embodiments, the candidate display area 806 displays one or more recognition results (eg, 810 and 812) for the handwriting input currently accumulated in the handwriting input area 804. In general, the highest recognition result (eg, 810) is displayed at the first position in the candidate display area. As shown in Fig. 8A, since the handwriting recognition model described herein can recognize characters of a plurality of non-overlapping scripts including Chinese characters, Latin scripts and emoji characters, the recognition result provided by the recognition model (E.g., 810) accurately includes Chinese characters, English characters and emoji characters, indicated by handwriting input. The user is not required to stop while writing input to select or switch recognition languages.

.")는 메모 애플리케이션으로 이미 제공된 텍스트 입력이다. 일부 실시예들에서, 커서(813)는 텍스트 입력 영역(808)에서의 현재 텍스트 입력 위치를 지시한다.In some embodiments, text input area 808 is an area that displays text input provided by a separate application employing a handwriting input interface. 8A, the text input area 808 is used by the memo application, and the text currently displayed in the text input area 808 (e.g., "America

.") is the text input already provided with the memo application. In some embodiments, the cursor 813 indicates the current text input position in the text input area 808.

In some embodiments, the user may perform an explicit selection input (eg, a tap gesture on one of the displayed recognition results), or an implicit confirmation input (eg, a double tap gesture or “enter” in the handwriting input area), for example. A tap gesture on a button) may select a specific recognition result displayed in the candidate display area 806. As shown in Fig. 8B, the user has explicitly selected the highest recognition result 810 using a tap gesture (as indicated by the contact 814 above the recognition result 810 in Fig. 8A). In response to the selection input, the text of the recognition result 810 is inserted in the text input area 808 at the insertion point indicated by the cursor 813. As shown in Fig. 8B, once the text of the selected recognition result 810 is input into the text input area 808, both the handwriting input area 804 and the candidate display area 806 are cleared. The handwriting input area 804 is now ready to accept new handwriting input, and the candidate display area 806 can now be used to display recognition results for the new handwriting input. In some embodiments, the implicit confirmation input causes the top recognition result to be entered into the text input area 808 without requiring the user to stop and select the top recognition result. Well-designed implicit confirmation input improves text input speed and reduces the cognitive burden placed on the user during text composition.

In some embodiments (not shown in FIGS. 8A and 8B ), the highest recognition result of the current handwriting input is optionally temporarily displayed in the text input area 808. The tentative text input shown in the text input area 808 is visually distinguished from other text input in the text input area, for example by a tentative input box surrounding the tentative text input. The text shown in the tentative input box has not yet been committed or provided to the associated application (e.g., a memo application), e.g., in response to a user revision of the current handwriting input, the top recognition result is changed by the handwriting input module. When it is done, it is automatically updated.

9A and 9B are flowcharts of an exemplary process 900 for providing multiscript handwriting recognition on a user device. In some embodiments, as shown in Figure 900, the user device receives (902) a multiscript handwriting recognition model, which is the spatial-derived features (e.g., stroke-order) of the multiscript training corpus. And stroke-direction independent features), and the multiscript training corpus includes handwriting samples corresponding to characters of at least three non-overlapping scripts. In some embodiments, the multiscript handwriting recognition model is a single convolutional neural network with a single input plane and a single output plane (906), to distinguish characters of at least three non-overlapping scripts displayed in the multiscript training corpus. It includes the space-derived features and individual weights for the space-derived features. In some embodiments, the multiscript handwriting recognition model is configured to recognize characters 908 based on individual input images of one or more recognition units identified in the handwriting input, and individual space-derived features used for recognition. Is independent of individual stroke order, stroke direction and continuity of strokes in handwriting input.

In some embodiments, the user device receives 908 handwriting input from the user, the handwriting input comprising one or more handwritten strokes provided on a touch-sensitive surface coupled to the user device. For example, the handwriting input includes individual data about the location and movement of the contact between a finger or stylus and a touch-sensitive surface coupled to the user device. In response to receiving the handwriting input, the user device provides the user with one or more handwriting recognition results in real time based on the multiscript handwriting recognition model trained on the space-guided features of the multiscript training corpus 912 (910). ).

In some embodiments, when providing real-time handwriting recognition results to a user, the user device divides the user's handwriting input into one or more recognition units (914), each recognition unit being one or more of the handwriting strokes provided by the user. Includes. In some embodiments, the user device segments the user's handwriting input according to the shape, location, and size of individual strokes made by contact between the user's finger or stylus and the touch-sensitive surface of the user device. In some embodiments, the segmentation of the handwriting input further takes into account the relative order and relative position of individual strokes made by contact between the user's finger or stylus and the touch-sensitive surface of the user device. In some embodiments, the user's handwriting input is in a cursive font, and each successive stroke in the handwriting input may correspond to a number of strokes in a character recognized in the print form. In some embodiments, the user's handwriting input may include a continuous stroke spanning multiple recognized characters in a printed form. In some embodiments, the segmentation of the handwriting input produces one or more input images each corresponding to a respective recognition unit. In some embodiments, some of the input images optionally include some overlapping pixels. In some embodiments, the input images do not include any overlapping pixels. In some embodiments, the user device creates a segmentation grid, and each segmentation chain of the segmentation grid represents a separate way of segmenting the current handwriting input. In some embodiments, each arc in the segmentation chain corresponds to a separate group of strokes in the current handwriting input.

"에 대한 필기 입력은 CJK 부수 "西"에 대한 필기 입력과 유사할 수도 있다. 일부 실시예들에서, 필기 입력의 시각적 외관이 인간 독자가 해독하더라도 어려울 것이기 때문에, 멀티스크립트 필기 인식 모델은 종종 사용자의 필기 입력에 대응할 것 같은 다수의 후보 인식 결과들을 생성한다. 일부 실시예들에서, 제1 스크립트는 CJK 기본 문자 블록이고, 제2 스크립트는 유니코드 표준에 의해 인코딩된 바와 같은 라틴어 스크립트이다. 일부 실시예들에서, 제1 스크립트는 CJK 기본 문자 블록이고, 제2 스크립트는 이모지 캐릭터들의 세트이다. 일부 실시예들에서, 제1 스크립트는 라틴어 스크립트이고, 제2 스크립트는 이모지 캐릭터들이다.As shown in Figure 900, the user device provides a separate image of each of the one or more recognition units as input to the multiscript recognition model (914). For at least one of the one or more recognition units, the user device may be configured to: from the multiscript handwriting recognition model, at least a first output character from the first script, and at least a second from a second script different from the first script. The output is obtained (916). For example, the same input image may cause the multiscript recognition model to output two or more similar-looking output characters from different scripts as recognition results for the same input image. For example, the handwritten inputs for the letter "a" in the Latin script and the letter "α" in the Greek script are often similar. In addition, handwriting inputs for the letter "J" in Latin script and the letter "丁" in Chinese characters are often similar. Similarly, the emoji character "

The handwriting input for “may be similar to the handwriting input for CJK collateral “西”. In some embodiments, since the visual appearance of the handwriting input will be difficult even for a human reader to decode, the multiscript handwriting recognition model is often used by the user. Generates a number of candidate recognition results that are likely to correspond to the handwriting input of In some embodiments, the first script is a CJK basic character block and the second script is a Latin script as encoded by the Unicode standard. In embodiments, the first script is a CJK basic character block and the second script is a set of emoji characters In some embodiments, the first script is a Latin script and the second script is emoji characters.

In some embodiments, the user device displays (918) both the first output character and the second output character in a candidate display area of the handwriting input interface of the user device. In some embodiments, the user device outputs one of the first output character and the second output character based on which of the first script and the second script is an individual script used in the soft keyboard currently installed on the user device. The character is selectively displayed (920). For example, assuming that the handwriting recognition model has identified the Chinese character "入" and the Greek character "λ" as the output characters for the current handwriting input, the user device is It is determined whether a keyboard using the method) is installed or a Greek input keyboard is installed. When the user device determines that only the Chinese soft keyboard is installed, the user device selectively displays only the Chinese character "入" as a recognition result to the user and does not display the Greek character "λ".

In some embodiments, the user device provides real-time handwriting recognition and input. In some embodiments, prior to the user making an explicit or implicit selection of the recognition result displayed to the user, the user device may respond to the continued addition or revision of the handwriting input by the user. The above recognition results are continuously revised (922). In some embodiments, in response to each revision of the one or more recognition results, the user displays 924 individually revised one or more recognition results to the user in a candidate display area of the handwriting input user interface.

In some embodiments, the multiscript handwriting recognition model is trained to recognize all characters of at least three non-overlapping scripts, including Chinese characters, emoji characters, and Latin script encoded according to the Unicode standard (926). ). In some embodiments, at least three non-overlapping scripts include Chinese characters, Arabic script, and Latin script. In some embodiments, the multiscript handwriting recognition model has (928) at least 30,000 output classes representing at least 30 characters spanning at least 3 non-overlapping scripts.

in Chinese."을 기입할 수 있는데, 한자들 "

"을 기입하기 전에 입력 언어를 영어로부터 중국어로 전환해야 할 필요 없이 또는 영어 단어들 "in Chinese."을 기입할 때 입력 언어를 중국어로부터 영어로 다시 전환하지 않고 기입할 수 있다.In some embodiments, the user device allows a user to enter multiscript handwriting input, such as a phrase that includes characters in more than one script. For example, a user may write continuously without stopping during writing to manually switch the recognition language and receive handwriting recognition results that include characters in more than one script. For example, in the handwriting input area of the user's device, the user can read the multi-script sentence "Hello means

in Chinese." can be entered, Chinese characters "

The input language can be entered without having to switch the input language from English to Chinese before entering ", or without switching the input language back from Chinese to English when entering the English words "in Chinese."

As described herein, a multi-script handwriting recognition model is used to provide real-time handwriting recognition for a user's input. In some embodiments, real-time handwriting recognition is used to provide real-time multi-script handwriting input functionality on the user's device. 10A-10C are flowcharts of an exemplary process 1000 for providing real-time handwriting recognition and input on a user device. Specifically, real-time handwriting recognition is independent of stroke order at the character level, the phrase level and the sentence level.

In some embodiments, stroke order independent handwriting recognition at the character level requires that the handwriting recognition model provides the same recognition result for a specific handwritten character, regardless of the sequence in which individual strokes of a specific character are provided by the user. do. For example, individual strokes of a Chinese character are typically written in a specific order. While Chinese-native speakers are often trained in schools to write each character in a specific order, many users later adopted personalized styles and stroke sequences that deviated from the conventional stroke order. In addition, cursive writing styles are highly individualized, and multiple strokes in the printed form of Chinese characters are often twisted and merged into a single stylized stroke, sometimes even linked to the next character. The stroke order independent recognition model is trained based on images of writing samples without temporal information associated with individual strokes. Thus, recognition is independent of stroke order information. For example, in the case of the Chinese character "十", the same recognition result "十" will be given by the handwriting recognition model regardless of whether the user writes a horizontal stroke first or a vertical stroke first.

As shown in Fig. 10A, in process 1000, the user device receives 1002 a plurality of handwritten strokes from the user, the plurality of handwritten strokes corresponding to the handwritten characters. For example, handwriting input for the letter “十” typically includes a substantially horizontal handwritten stroke intersecting with a substantially vertical handwritten stroke.

In some embodiments, the user device generates an input image based on the plurality of handwritten strokes (1004). In some embodiments, the user device performs real-time handwriting recognition of handwritten characters by providing the input image as a handwriting recognition model (1006), where the handwriting recognition model provides handwriting recognition independent of stroke order. Thereafter, the user device is in real time upon receiving the plurality of handwritten strokes, the plurality of handwritten strokes (e.g., horizontal and vertical strokes) are the same first output character (e.g. printed The letter "十") in the form is displayed (1008).

Some conventional handwriting recognition systems allow small stroke order variations in a small number of characters, but are specifically made by including those variations in training of the handwriting recognition system. These conventional handwriting recognition systems are not scalable to accommodate arbitrary stroke order variations in a large number of complex characters such as Chinese characters, since even a moderately complex character will already cause a number of variations in stroke order. Because. Moreover, even by simply including more permutations of allowable stroke orders for specific characters, conventional recognition systems still have multiple strokes combined into a single stroke (e.g., as in extreme cursive writing) or It will be impossible to handle handwriting inputs where one stroke is decomposed into multiple sub-strokes (eg, as in a character captured by a very coarse sampling of the input stroke). Thus, a multiscript handwriting system trained on space-directed features as described herein provides an advantage over conventional recognition systems.

In some embodiments, stroke order independent handwriting recognition is performed regardless of temporal information associated with individual strokes within each handwritten character. In some embodiments, stroke order independent handwriting recognition is performed in conjunction with stroke-distribution information that takes into account the spatial distribution of individual strokes before the individual strokes are merged into a flat input image. Further details of how time-induced stroke-distribution information is used to enhance stroke order independent handwriting recognition described above are provided later herein (e.g., with respect to FIGS. 25A-27). The technique described with respect to FIGS. 25A-27 does not impair the stroke order independence of the handwriting recognition system.

In some embodiments, the handwriting recognition model provides 1010 stroke direction independent handwriting recognition. In some embodiments, stroke direction independent recognition, in response to the user device receiving the plurality of handwriting inputs, each of the plurality of handwriting strokes displays the same first output character regardless of the individual stroke direction provided by the user. Demands to do. For example, if the user writes the Chinese character "十" in the handwriting input area of the user device, the handwriting recognition model will have the same recognition result regardless of whether the user draws a horizontal stroke from left to right or from right to left. Will print. Similarly, the handwriting recognition model will output the same recognition result regardless of whether the user drew a vertical stroke in a downward direction or upward direction. In another example, many Chinese characters are structurally made up of two or more radicals. Some Chinese characters include left and right copies, respectively, and people customarily write the left side first and then the right side. In some embodiments, as long as the resulting handwriting input indicates a left copy to the left of the right copy when the user completes the handwritten character, the handwriting recognition model determines whether the user writes the right copy first or the left copy first. Regardless, it will give the same recognition results. Similarly, some Chinese characters include the upper and lower copies, respectively, and people customarily write the upper one first, then the lower one. In some embodiments, the handwriting recognition model will provide the same recognition result regardless of whether the user writes the top number first or the bottom number first, as long as the resulting handwriting input represents the top number above the bottom number. . In other words, the handwriting recognition model does not rely on the directions in which the user provides individual strokes of the handwritten character to determine the identity of the handwritten character.

In some embodiments, the handwriting recognition model provides handwriting recognition based on the image of the recognition unit, regardless of the number of sub-strokes provided by the user by the recognition unit. In other words, in some embodiments, the handwriting recognition model provides stroke count independent handwriting recognition (1014). In some embodiments, in response to receiving the plurality of handwritten strokes, the user device displays the same first output character regardless of how many handwritten strokes are used to form a continuous stroke in the input image. For example, when the user writes the Chinese character "+" in the handwriting input area, the handwriting recognition model uses four strokes (for example, to construct a cross-shaped character) to form the shape of the 2 short horizontal strokes and 2 short vertical strokes), or 2 strokes (e.g., an L-shaped stroke and a 7-shaped stroke, or a horizontal and vertical stroke), or any other number Regardless of whether or not you provide the strokes (eg, hundreds of extremely short strokes or dots), the same recognition result is output.

In some embodiments, the handwriting recognition model is capable of recognizing the same character regardless of the order, direction and number of strokes in which each single character is written, as well as multiple strokes of multiple characters regardless of the temporal order provided by the user. You can also recognize the characters of.

In some embodiments, the user device not only received the first plurality of handwritten strokes, but also received a second plurality of handwritten strokes from the user (1016), wherein the second plurality of handwritten strokes correspond to the second handwritten character. . In some embodiments, the user device generates (1018) a second input image based on the second plurality of handwritten strokes. In some embodiments, the user device performs real-time recognition of the second handwritten character by providing the second input image as a handwriting recognition model (1020). In some embodiments, the user device displays (1022) a second output character corresponding to the second plurality of handwritten strokes in real time upon receiving the second plurality of handwritten strokes. In some embodiments, the first output character and the second output character are displayed simultaneously in a spatial sequence regardless of the individual order in which the first plurality of handwritten strokes and the second plurality of handwritten strokes are provided by the user. For example, if the user writes two Chinese characters (eg, “十” and “八”) in the handwriting input area of the user device, the currently accumulated handwriting input in the handwriting input area is a stroke for the character “八”. As long as the strokes for the letter "十" are shown to the left of the fields, the user device will recognize the recognition result "十八" regardless of whether the user writes the strokes of the letter "十" first or the strokes of the letter "八" first. Will display ". In fact, even if the user has written some of the strokes for the letter "八" (e.g., a stroke slanted to the left) before some of the strokes for the letter "十" (eg, a vertical stroke), the result of the handwriting input in the handwriting input area As long as the typical image shows all the strokes for the character “十” to the left of all the strokes for the character “八”, the user device will show the recognition result “十八” in the spatial sequence of two handwritten characters.

In other words, as shown in Figure 10B, in some embodiments, the spatial sequence of the first output character and the second output character is along the default writing direction (e.g., from left to right) of the handwriting input interface of the user device. It corresponds to the spatial distribution of the first plurality of handwritten strokes and the second plurality of strokes (1024). In some embodiments, the second plurality of handwritten strokes are received temporally after the first plurality of handwritten strokes, and the second output character is along a default writing direction (eg, from left to right) of the handwriting input interface of the user device Precedes the first output character in the spatial sequence (1026).

In some embodiments, the handwriting recognition model provides stroke order independent recognition at the sentence-to-sentence level. For example, a handwritten letter "十" is in a first handwritten sentence and a handwritten letter "八" is in a second handwritten sentence, and two handwritten characters are added to one or more other handwritten characters and/or words in the handwriting input area. Even if separated by, the handwriting recognition model will still provide a recognition result representing two characters in a spatial sequence "十...八". The recognition result and spatial sequence of the two recognized characters remains the same regardless of the temporal order provided by the user, but the recognition unit for the two characters when the user completes the handwriting input. Are spatially arranged in the sequence "十…八". In some embodiments, the first handwritten character (eg, “十”) is provided by the user as part of the first handwritten sentence (eg “十 is a number.”), and the second handwritten character (eg, “八") is provided by the user as part of a second handwritten sentence (eg, “八 is another number.”), and the first handwritten sentence and the second handwritten sentence are simultaneously displayed in the handwriting input area of the user device. In some embodiments, when the user confirms that the recognition result (eg, “十 is a number. 八. is another number.”) is the correct recognition result, two sentences will be entered into the text input area of the user device, The handwriting input area will be cleared for the user to input another handwriting input.

In some embodiments, since the handwriting recognition model is stroke order independent not only at the character level, but also at the phrase level and sentence level, the user can make modifications to previously unfinished characters after subsequent characters have been written. For example, if the user forgets to write a specific stroke for a character before moving to write one or more subsequent characters in the handwriting input area, the user will miss it in the correct position in the specific character to receive an accurate recognition result. You can still write the strokes later.

In conventional stroke order independent recognition systems (eg, HMM-based recognition system), once a character is written, it is committed, and the user can no longer change it. If the user wants any change, the user has to start over by deleting the character and all subsequent characters. In some conventional recognition systems, the user is required to complete a handwritten character within a short predetermined time window, and any strokes entered outside the predetermined time window are included in the same recognition unit as other strokes provided during the time window. Won't be. Such conventional systems are difficult to use and cause great frustration to the user. Stroke order independent systems do not suffer from such shortcomings, and the user can complete the character in any order and any time frame that the user deems suitable. The user may also subsequently modify the first written character (eg, add one or more strokes) after writing one or more characters in the handwriting input interface. In some embodiments, the user may also individually delete the first written character (e.g., using methods described later with respect to FIGS. 21A-22B) and rewrite it to the same location in the handwriting input interface. .

")을 기입하였다. 제1 복수의 획들은 필기 문자 "八"을 형성한다. 사용자가 사실은 문자 "

"를 기입하려고 하였으나 하나의 획을 누락하였음에 주의한다. 제2 복수의 획들은 필기 문자 "

"를 형성한다. 사용자가 "

" 대신에 "

"를 기입하기를 원함을 나중에 알아차린 경우, 사용자는 간단히 문자 "八"에 대한 획들 아래에 수직 획을 하나 더 쓸 수 있고, 사용자 디바이스는 수직 획을 제1 인식 유닛(예컨대, "八"에 대한 인식 유닛)에 배정할 것이다. 사용자 디바이스는 제1 인식 유닛에 대한 새로운 출력 문자(예컨대, "八")를 출력할 것이고, 여기서 새로운 출력 문자는 인식 결과에서 이전의 출력 문자 (예컨대, "八")를 대신할 것이다. 도 10c에 도시된 바와 같이, 사용자 디바이스는, 제3 필기 획을 수신하는 것에 응답하여, 제1 복수의 필기 획들에 대한 제3 필기 획의 상대적인 근접성에 기초하여 제1 복수의 필기 획들과 동일한 인식 유닛에 제3 필기 획을 배정한다(1032). 일부 실시예들에서, 사용자 디바이스는 제1 복수의 필기 획들 및 제3 필기 획에 기초하여 개정된 입력 이미지를 생성한다(1034). 사용자 디바이스는 개정된 입력 이미지를 필기 인식 모델로 제공하여 개정된 필기 문자의 실시간 인식을 수행한다(1036). 일부 실시예들에서, 사용자 디바이스는 제3 필기 입력을 수신하는 것에 응답하여, 개정된 입력 이미지에 대응하는 제3 출력 문자를 디스플레이하는데(1040), 여기서 제3 출력 문자는 제1 출력 문자를 대신하고 디폴트 기입 방향을 따른 공간 시퀀스에서 제2 출력 문자와 동시에 디스플레이된다.10B and 10C, the second plurality of handwritten strokes spatially follow the first plurality of handwritten strokes along the default writing direction of the handwriting input interface of the user device, and the second output character is the handwriting input interface. The first output character is followed in a spatial sequence along the default writing direction in the candidate display area of (1028). The user device receives (1030) a third handwritten stroke from the user to revise the first handwritten character (i.e., a handwritten character formed by the first plurality of handwritten strokes), the third handwritten stroke being temporally divided into the first plurality of It is received after the handwritten strokes and the second plurality of handwritten strokes. For example, the user may select two characters (e.g., ") in a spatial sequence from left to right in the handwriting input area.

"). The first plurality of strokes form the handwritten character "八". The user actually has the character "

Note that you tried to write "but you missed one stroke. The second plural strokes are handwritten characters"

"Forms a user."

" Instead of "

If later noticed that he wants to write ", the user can simply write one more vertical stroke below the strokes for the letter "八", and the user device writes the vertical stroke to the first recognition unit (e.g., "八"). The user device will output a new output character (e.g., "八") for the first recognition unit, where the new output character is the previous output character (e.g., "八") in the recognition result. 10C, the user device, in response to receiving the third handwritten stroke, is based on the relative proximity of the third handwritten stroke to the first plurality of handwritten strokes. Assign a third handwritten stroke to the same recognition unit as the plurality of handwritten strokes (1032). In some embodiments, the user device generates a revised input image based on the first plurality of handwritten strokes and the third handwritten stroke. (1034) The user device provides the revised input image to the handwriting recognition model to perform real-time recognition of the revised handwritten character (1036) In some embodiments, the user device responds to receiving the third handwriting input. Thus, a third output character corresponding to the revised input image is displayed (1040), where the third output character replaces the first output character and is displayed simultaneously with the second output character in a spatial sequence along the default writing direction.

In some embodiments, the handwriting recognition module recognizes a handwriting input written in a default writing direction from left to right. For example, the user can write characters from left to right and in one or more lines. In response to the handwriting input, the handwriting input module presents recognition results comprising the characters in a left-to-right spatial sequence and in one or more rows as needed. When the user selects a recognition result, the selected recognition result is input into a text input area of the user device. In some embodiments, the default writing direction is from top to bottom. In some embodiments, the default writing direction is from right to left. In some embodiments, the user optionally changes the default writing direction to an alternative writing direction after the recognition result is selected and the handwriting input area is cleared.

In some embodiments, the handwriting input module allows the user to input multi-character handwriting input in the handwriting input area, and the deletion of strokes in one recognition unit at a time rather than all recognition units at once from the handwriting input Allow In some embodiments, the handwriting input module allows deletion of one stroke at a time from handwriting input. In some embodiments, deletion of the recognition unit proceeds one by one in a direction opposite to the default writing direction, regardless of the order in which the recognition units or strokes were entered to generate the current handwriting input. In some embodiments, the deletion of the strokes proceeds one by one in the reverse order that the strokes are input in each recognition unit, and when all of the strokes in one recognition unit are deleted, the deletion of the strokes is in a direction opposite to the default writing direction. Proceed to the next recognition unit.

In some embodiments, the user device receives a deletion input from the user while the third output character and the second output character are simultaneously displayed as a candidate recognition result in the candidate display area of the handwriting input interface. In response to the deletion input, the user device deletes the second output character from the recognition result while maintaining the third output character in the recognition result displayed in the candidate display area.

"에서 누락된 수직 획을 제공한 후에, 사용자가 삭제 입력을 입력하는 경우, 문자 "

"에 대한 인식 유닛에서의 획들은 필기 입력 영역으로부터 제거되고, 문자 "

"는 사용자 디바이스의 후보 디스플레이 영역에서 인식 결과 "

"로부터 제거된다. 삭제 후에, 문자 "

"에 대한 획들은 필기 입력 영역에 남아 있고, 한편 인식 결과는 문자 "

"만을 나타낸다.In some embodiments, as shown in FIG. 10C, the user device may generate a first plurality of handwritten strokes, a second plurality of handwritten strokes, and a third handwritten stroke in real time as each of the handwritten strokes is provided by the user. Render (1042). In some embodiments, the user device, in response to receiving a deletion input from the user, of the first plurality of handwritten strokes and the third handwritten stroke (e.g., corresponding together with the revised first handwritten character) in the handwriting input area. While maintaining the individual renderings, the individual renderings of the second plurality of handwriting inputs (eg, corresponding to the second handwritten character) are deleted from the handwriting input area (1044). For example, the user has a character sequence "

"After providing the missing vertical stroke, if the user enters the delete input, the character"

Strokes in the recognition unit for "are removed from the handwriting input area, and the character"

"The recognition result in the candidate display area of the user device"

"Is removed from. After deletion, the character"

The strokes for "are left in the handwriting input area, while the recognition result is the character"

"Only represents.

In some embodiments, the handwritten character is a multi-stroke Chinese character. In some embodiments, the first plurality of handwriting inputs are provided in a cursive font. In some embodiments, the first plurality of handwriting inputs are provided in a cursive font, and the handwritten characters are multi-stroke Chinese characters. In some embodiments, the handwritten characters are written in Arabic in a cursive font. In some embodiments, the handwritten characters are written in other scripts in a cursive font.

In some embodiments, the user device establishes individual predetermined constraints on the set of allowable dimensions for handwritten character input, and based on the individual predetermined constraints, the plurality of recognition units Divided into, where a separate input image provided from each of the recognition units to the handwriting recognition model is generated and recognized as a corresponding output character.

In some embodiments, the user device receives additional handwritten strokes from the user after dividing the currently accumulated plurality of handwritten strokes. The user device allocates an additional handwritten stroke to an individual recognition unit of the plurality of recognition units based on the spatial position of the additional handwritten stroke for the plurality of recognition units.

Attention is now directed to example user interfaces for providing handwriting recognition and input on a user device. In some embodiments, exemplary user interfaces are provided on a user device based on a multiscript handwriting recognition model that provides real-time, stroke order independent handwriting recognition of the user's handwriting input. In some embodiments, exemplary user interfaces include an exemplary handwriting input interface 802 (e.g., FIGS. 8A and 8B) including a handwriting input area 804, a candidate display area 804, and a text input area 808. 8B). In some embodiments, the exemplary handwriting input interface 802 also includes a plurality of control elements 1102, such as a delete button, a space bar, an enter button, a keyboard switch button, and the like. One or more other areas and/or elements may be provided in the handwriting input interface 802 to enable additional functions described below.

As described herein, a multiscript handwriting recognition model can have a very large repertoire of tens of thousands of characters in many different scripts and languages. As a result, for handwriting input, the recognition model is very likely to identify a large number of output characters, all of which have a fairly good likelihood of being a character intended by the user. On a user device with a limited display area, it is advantageous to initially provide only a subset of recognition results, while keeping other results available upon the user request.

11A-11G illustrate exemplary user interfaces for displaying a subset of recognition results in the normal field of view of the candidate display area, with affordance for invoking the extended field of view of the candidate display area to display the remaining recognition results. do. Further, within the extended field of view of the candidate display area, recognition results are divided into different categories and displayed on different tabbed pages in the extended field of view.

11A shows an exemplary handwriting input interface 802. The handwriting input interface includes a handwriting input area 804, a candidate display area 806, and a text input area 808. One or more control elements 1102 are also included in the handwriting input interface 1002.

11A, candidate display area 806 is an area for optionally displaying one or more recognition results, and an affordance 1104 (e.g., an expanded icon) for invoking an expanded version of the candidate display area 806 ).

11A-11C show that as the user provides one or more handwritten strokes (e.g., strokes 1106, 1108, 1110) in the handwriting input area 804, the user device is It shows identifying and displaying a separate set of recognition results corresponding to the strokes. 11B, after the user inputs the first stroke 1106, the user device causes three recognition results 1112, 1114, 1116 (e.g., characters "/", "1", and ",") is identified and displayed. In some embodiments, a small number of candidate characters are displayed in the candidate display area 806 in order according to the recognition reliability associated with each character.

In some embodiments, the top candidate result (eg, “/”) is tentatively displayed in the text entry area 808, eg, in box 1118. The user can optionally confirm that the top candidate is the intended input by a simple confirmation input (eg, by pressing the “Enter” key, or by providing a double-tap gesture in the handwriting input area).

11C shows that, before the user selects any candidate recognition result, as the user inputs two more strokes 1108, 1100 in the handwriting input area 804, additional strokes are handwritten with the initial stroke 1106. Rendered in the input area 804 and shown as candidate results are updated to reflect changes to the recognition unit(s) identified from the currently accumulated handwriting inputs. As shown in Fig. 11C, based on the three strokes, the user device has identified a single recognition unit. Based on a single recognition unit, the user device has identified and displayed multiple recognition results 1118-1124. In some embodiments, one or more of the recognition results currently displayed in the candidate display area 806 (e.g., 1118 and 1122) are each a representative candidate character selected from among a number of similar-looking candidate characters for the current handwriting input. .

As shown in FIGS. , Shown in FIG. 11C) to an extended field of view (eg, shown in FIG. 11D). In some embodiments, the extended field of view represents all recognition results (eg, candidate characters) identified for the current handwriting input.

In some embodiments, the normal field of view initially displayed in the candidate display area 806 represents only the most common characters used in an individual script or language, while the extended field of view includes characters that are rarely used in a script or language. Thus, all candidate characters are represented. The extended field of view of the candidate display area can be designed in different ways. 11D-11G illustrate exemplary designs of an extended candidate display area, according to some embodiments.

11D, in some embodiments, the extended candidate display area 1128 is one or more tabbed pages (e.g., pages 1130, 1132, 1134) each presenting a separate category of candidate characters. , 1136)). The tabbed design shown in Fig. 11D allows the user to quickly find the desired category of characters, and then find the character he or she is trying to enter in the corresponding tabbed page.

", "β", "巾", 등)을 포함한다.In FIG. 11D, the first tabbed page 1130 displays all candidate characters identified for the currently accumulated handwriting input, including both rare characters as well as common characters. As shown in FIG. 11D, the tabbed page 1130 includes all of the characters shown in the initial candidate display area 806 of FIG. 11C and a number of additional characters not included in the initial candidate display area 806 ( For example, "

", "β", "巾", etc.).

In some embodiments, the characters displayed in the initial candidate display area 806 are from the set of commercial characters associated with the script (e.g., all characters in the basic block of the CJK script, as encoded according to the Unicode standard). Contains only the characters of In some embodiments, the characters displayed in the extended candidate display area 1128 are the set of rare characters associated with the script (e.g., all characters in the extended block of the CJK script, as encoded according to the Unicode standard). It additionally includes. In some embodiments, the extended candidate display area 1128 further includes candidate characters from other scripts not commonly used by the user, such as Greek script, Arabic script, and/or emoji script.

In some embodiments, as shown in Fig. 11D, the extended candidate display area 1128 includes individual tabbed pages 1130, 1132, 1134, 1138, each of which is a separate category of candidate characters. (E.g., all letters, rare letters, letters from Latin script, and letters from emoji script, respectively). 11E to 11G illustrate that a user can select each of different tabbed pages to reveal candidate characters in a corresponding category. 11E shows only rare characters corresponding to the current handwriting input (eg, characters from the expansion block of the CJK script). 11F shows only Latin and Greek characters corresponding to the current handwriting input. 11G shows only emoji characters corresponding to the current handwriting input.

In some embodiments, the extended candidate display area 1128 is an individual tabbed page based on individual criteria (e.g., based on frontal spelling, based on number of strokes, based on number of copies, etc.). It further includes one or more affordances to align the candidate characters in. The ability to sort candidate characters in each category according to criteria other than recognition reliability scores provides the user with the additional ability to quickly find the desired candidate character for text input.

11H-11K show that in some embodiments, candidate characters that look similar may be grouped, and that only representative characters from each group of candidate characters that look similar are presented in the initial candidate display area 806. do. Since the multiscript recognition model described herein can generate a large number of candidate characters that are almost equally good for a given handwriting input, the recognition model is always able to eliminate one candidate at the expense of another similar-looking candidate. no. On a device with a limited display area, suddenly displaying a large number of similar-looking candidates does not help the user to select the correct character, which makes fine distinctions difficult to see and even if the user can see the desired character. , Because it can be difficult to select it from a very dense display using a finger or a stylus.

" 및 "T")는 임의의 그룹에 속하지 않는 후보 문자(예컨대, 후보 문자들(1120, 1124), "乃" 및 "J")와는 상이한 방식으로(예컨대, 볼드로 표시된 박스로) 디스플레이된다. 일부 실시예들에서, 그룹의 대표 문자를 선정하기 위한 기준은 그룹에서의 후보 문자들의 상대적인 사용 빈도에 기초한다. 일부 실시예들에서, 다른 기준들이 이용될 수도 있다.In some embodiments, to address the above issues, the user device identifies candidate characters with great similarity to each other (e.g., according to a dictionary or term index of characters that look similar, or some image-based criteria) and , Group them into individual groups. In some embodiments, one or more groups of similar-looking characters may be identified from a set of candidate characters for a given handwriting input. In some embodiments, the user device has identified a representative candidate character among a plurality of similar-looking candidate characters in the same group, and displays only the representative candidate in the initial candidate display area 806. If the common character does not look sufficiently similar to any other candidate characters, it is displayed as such. In some embodiments, as shown in Fig. 11H, representative candidate characters of each group (e.g., candidate characters 1118, 1122, "

"And "T") are displayed in a different way (e.g., in a box marked in bold) than candidate characters that do not belong to any group (e.g., candidate characters 1120, 1124, "乃" and "J"). In some embodiments, the criterion for selecting the representative character of the group is based on the relative frequency of use of the candidate characters in the group In some embodiments, other criteria may be used.

In some embodiments, once the representative character(s) is displayed to the user, the user may optionally expand the candidate display area 806 to represent candidate characters that appear similar in the expanded view. In some embodiments, selection of a particular representative character may result in an expanded view of only those candidate characters in the same group as the selected representative character.

Various designs are possible to provide an expanded field of view of candidates that look similar. 11H-11K illustrate one embodiment in which the expanded field of view of the representative candidate character is invoked by a predetermined gesture (eg, an expanding gesture) detected over the representative candidate character (eg, representative character 1118). The predetermined gesture for invoking an inflated view (eg, inflated gesture) is different from a predetermined gesture for selecting a representative character for text input (eg, tapped gesture).

", "

" 및 "巾")이 제시된다.11H and 11I, when the user provides an inflate gesture over the first representative character 1118 (e.g., as represented by two contacts 1138, 1140 moving away from each other). , The area displaying the representative character 1118 is expanded, and an enlarged field of view (eg, enlarged boxes 1142, 1144, respectively) compared to other candidate characters (eg, “乃”) not within the same expanded group. , 1146)) three similar-looking candidate characters (e.g., "

", "

"And "巾") are presented.

", "

" 및 "巾")의 미세한 구별들이 사용자에 의해 보다 쉽게 보여질 수 있다. 3개의 후보 문자들 중 하나가 의도된 문자 입력인 경우, 사용자는, 예컨대 그 후보 문자가 디스플레이된 영역을 터치함으로써 후보 문자를 선택할 수 있다. 도 11j 및 도 11k에 도시된 바와 같이, 사용자는 팽창 시야에서 박스(1144)에 나타낸 제2 문자(예컨대, "

")를 (접촉(1148)에 의해) 선택하였다. 응답으로, 선택된 문자(예컨대, "

")는 커서에 의해 지시되는 삽입 위치에서 텍스트 입력 영역(808)으로 입력된다. 도 11k에 도시된 바와 같이, 일단 문자가 선택되면, 필기 입력 영역(804)에서의 필기 입력 및 후보 디스플레이 영역(806)(또는 후보 디스플레이 영역의 확장 시야)에서의 후보 문자들은 후속 필기 입력을 위해 클리어된다.11I, when presented in an enlarged field of view, three similar-looking candidate characters (e.g., "

", "

"And "巾") can be more easily seen by the user. If one of the three candidate characters is the intended character input, the user can, for example, touch the area in which the candidate character is displayed. The character can be selected. As shown in Figs. 11J and 11K, the user can select a second character (e.g., ""

") was selected (by contact 1148). In response, the selected character (e.g., "

") is input into the text input area 808 at the insertion position indicated by the cursor. As shown in Fig. 11K, once a character is selected, handwriting input in the handwriting input area 804 and the candidate display area ( 806) (or the extended field of view of the candidate display area) are cleared for subsequent handwriting input.

In some embodiments, when the user does not see the desired candidate character in the expanded field of view 1142 of the first representative candidate character, the user selectively uses the same gesture to select another representative character displayed in the candidate display area 806. Can inflate them. In some embodiments, inflating another representative character in the candidate display area 806 automatically restores the currently presented inflated field of view to the normal field of view. In some embodiments, the user selectively restores the current expanded field of view to the normal field of view using a zoom out gesture. In some embodiments, the user may scroll (eg, to the left or to the right) the candidate display area 806 to reveal other candidate characters that are not visible in the candidate display area 806.

12A and 12B show a second subset of recognition results in an extended candidate display area that is hidden from view until specifically called by the user, while a first subset of recognition results is presented in the initial candidate display area A subset is the flowcharts of an exemplary process 1200 presented. In exemplary process 1200, the device identifies, from a plurality of handwriting recognition results for the handwriting input, a subset of recognition results having a level of visual similarity that exceeds a predetermined threshold. Thereafter, the user device selects a representative recognition result from the subset of recognition results, and displays the selected representative recognition result in a candidate display area of the display. Process 1200 is illustrated in FIGS. 11A-11K.

As shown in FIG. 12A, in an exemplary process 1200, a user device receives (1202) a handwriting input from a user. The handwriting input includes one or more handwriting strokes (eg, 1106, 1108, 1110 in FIG. 11C) provided in a handwriting input area (eg, 806 in FIG. 11C) of a handwriting input interface (eg, 802 in FIG. 11C). Based on the handwriting recognition model, the user device identifies a plurality of output characters (eg, characters displayed on the tabbed page 1130, FIG. 11C) for the handwriting input (1204). The user device divides the plurality of output characters into two or more categories based on a predetermined classification criterion (1206). In some embodiments, the predetermined classification criterion determines whether an individual character is a common character or a rare character (1208).

In some embodiments, the user device, in the initial view of the candidate display area of the handwriting input interface (e.g., 806 as shown in FIG. The commercial characters) are displayed 1210, and an affordance (e.g., 1104 in FIG. 11C) for calling an extended view of the candidate display area (eg, 1128 in FIG.

In some embodiments, the user device receives (1212) a user input selecting an affordance for invoking an extended field of view, eg, as shown in FIG. 11C. In response to the user input, the user device may advance in the initial view of the candidate display area and individual output characters in the first category of two or more categories, e.g. Individual output characters in at least the second category that are not displayed in are displayed (1214).

In some embodiments, individual characters in the first category are characters found in a dictionary of common characters, and individual characters in the second category are characters found in a dictionary of rare characters. In some embodiments, the dictionary of common characters and the dictionary of rare characters are dynamically adjusted or updated based on the usage history associated with the user device.

In some embodiments, the user device is a group of characters that are visually similar to each other from a plurality of output characters, according to a predetermined similarity criterion (e.g., based on a dictionary of similar characters or based on some space-derived features). Is identified (1216). In some embodiments, the user device selects a representative character based on a predetermined selection criterion (eg, based on historical frequency of use) from a group of visually similar characters. In some embodiments, the predetermined selection criteria are based on the relative frequency of use of the characters in the group. In some embodiments, the predetermined selection criterion is based on a preferred input language associated with the device. In some embodiments, the representative candidate is selected based on other factors indicating the likelihood that each candidate is an intended input by the user. These factors include, for example, whether the candidate character belongs to a script for a soft keyboard currently installed on the user's device, or whether the candidate character is among the set of most common characters in a particular language associated with the user or user device Includes.

", "巾") 대신에 대표 문자 (예컨대, "

")를 디스플레이한다(1220). 일부 실시예들에서, 각 후보 문자가 그룹의 대표 문자인지 또는 어떠한 그룹 내에도 있지 않은 일반 후보 문자인지를 나타내기 위해 후보 디스플레이 영역의 초기 시야에서 시각적 표시들(예컨대, 선택적인 시각적 강조, 특별한 배경)이 제공된다. 일부 실시예들에서, 예컨대, 도 11h에 도시된 바와 같이, 사용자 디바이스는, 사용자로부터, 후보 디스플레이 영역의 초기 시야에서 디스플레이되는 대표 문자로 안내되는 사전결정된 팽창 입력(예컨대, 팽창 제스처)을 수신한다(1222). 일부 실시예들에서, 예컨대, 도 11i에 도시된 바와 같이, 사용자 디바이스는, 사전결정된 팽창 입력의 수신에 응답하여, 시각적으로 유사한 문자들의 그룹에서의 대표 문자의 확대 시야 및 하나 이상의 다른 문자들의 개별 확대 시야를 동시에 디스플레이한다(1224).In some embodiments, the user device, in the initial field of view of the candidate display area (eg, 806 in FIG. 11H), may display other characters in a group of visually similar characters (eg, “

Instead of ", "巾"), a representative character (eg, "

1220. In some embodiments, visual indications in the initial field of view of the candidate display area ( For example, optional visual emphasis, special background) is provided In some embodiments, for example, as shown in Fig. 11H, the user device guides from the user to a representative character displayed in the initial field of view of the candidate display area. 1222. In some embodiments, for example, as shown in Figure 11I, the user device, in response to receiving the predetermined inflation input, is visually The enlarged field of view of the representative character in the group of similar characters and the individual enlarged field of view of one or more other characters are simultaneously displayed (1224).

In some embodiments, the predetermined inflation input is an inflation gesture detected over a representative character displayed in the candidate display area. In some embodiments, the predetermined inflation input is a contact that is detected over a representative character displayed in the candidate display area and lasts longer than a predetermined threshold time. In some embodiments, sustained contact to inflate a group has a longer threshold duration than a tap gesture to select a representative character for text input.

In some embodiments, each representative character is displayed concurrently with a separate affordance (eg, a separate inflate button) to invoke the expanded view of its group of similar-looking candidate characters. In some embodiments, the predetermined inflation input is a selection of individual affordances associated with the representative character.

As described herein, in some embodiments, the repertoire of the multiscript handwriting recognition model includes emoji scripts. The handwriting input recognition module may recognize an emoji character based on a user's handwriting input. In some embodiments, the handwriting recognition module presents both emoji characters identified directly from the handwriting, and a letter or word in natural human language representing the identified emoji character. In some embodiments, the handwriting input module recognizes a character or word in natural human language based on the user's handwriting input, and both the recognized character or word and an emoji character corresponding to the recognized character or word Present. In other words, the handwriting input module provides methods for inputting an emoji character without switching from a handwriting input interface to an emoji keyboard. Additionally, the handwriting input module also provides a way to enter regular natural language characters and words by drawing an emoji character by hand. 13A-13E provide exemplary user interfaces illustrating these different ways of entering emoji characters and regular natural language characters.

")를 수신하였다.13A shows an exemplary handwriting input interface 802 invoked under a chat application. The handwriting input interface 802 includes a handwriting input area 804, a candidate display area 806, and a text input area 808. In some embodiments, once the user is satisfied with the text composition in the text input area 808, the user may elect to transfer the text composition to another participant in the current chat session. The conversation history of the chat session is displayed on the conversation panel 1302. In this example, the user may have a chat message 1304 displayed on the conversation panel 1302 (e.g., "Happy Birthday

") was received.

As shown in FIG. 13B, the user provides a handwriting input 1306 for the English word “Thanks” in the handwriting input area 804. In response to handwriting input 1306, the user device has identified a number of candidate recognition results (eg, recognition results 1308, 1310, 1312). The highest recognition result 1303 has been tentatively entered into the text input area 808 in the box 1314.

13C, after the user enters the handwritten word "Thanks" in the handwriting input area 806, the user then follows a stylized exclamation mark with strokes 1316 in the handwriting input area 806 (e.g., Draw a long circle with a round circle below it The user device recognizes that the additional strokes 1316 form a recognition unit separate from other recognition units previously recognized from the accumulated handwriting strokes 1306 in the handwriting input area 806. Based on the newly entered recognition unit (ie, the recognition unit formed by the strokes 1316), the user device identifies an emoji character (eg, a stylized “!”) using the handwriting recognition model. Based on this recognized emoji character, the user device presents a first recognition result 1318 (eg, “Thanks!” with a stylized “!”) in the candidate display area 806. Additionally, the user device also identifies a number "8" that is visually similar to the newly entered recognition unit. Based on this recognized number, the user device presents a second recognition result 1322 (eg, “Thanks 8”) in the candidate display area 806. Additionally, based on the identified emoji character (eg, stylized “!”), the user device also identifies a regular character (eg, the regular character “!”) corresponding to the emoji character. Based on this indirectly identified regular character, the user device presents a third recognition result 1320 (eg, “Thanks!” with the regular character “!”) in the candidate display area 806. At this point, the user can select any one of the candidate recognition results 1318, 1320, and 1322 and input it into the text input area 808.

13D, the user continues to provide additional handwritten strokes 1324 in the handwriting input area 806. This time, the user drew a heart symbol followed by a stylized exclamation mark. In response to the new handwritten strokes 1324, the user device recognizes that the newly provided handwritten strokes 1324 form another new recognition unit. Based on the new recognition unit, the user device identifies the emoji character “♥”, and alternatively the number “0” as candidate characters for the new recognition unit. Based on these new candidate characters recognized from the new recognition unit, the user device presents two updated candidate recognition results 1326, 1330 (eg, "Thanks !♥" and "Thanks 80"). In some embodiments, the user device further identifies the regular character(s) or word(s) (eg, “Love”) corresponding to the identified emoji character (eg, “♥”). Based on the identified canonical character(s) or word(s) for the recognized emoji character, the user device is configured to replace the recognized emoji character(s) with the corresponding regular character(s) or word(s). The third recognition result (1328) is presented. As shown in Fig. 13D, in the recognition result 1328, the emoji character "!" is replaced with a general exclamation mark "!", and the emoji character "♥" is replaced with regular characters or the word "Love".

13E, the user selects one of the candidate recognition results (eg, a candidate result 1326 representing the mixed script text “Thanks!♥”), and the text of the selected recognition result is a text input area 808 ), and subsequently transmitted to other participants in the chat session. The message bubble 1332 represents the text of a message in the dialog panel 1302.

14 is a flowchart of an exemplary process 1400 in which a user enters an emoji character using handwriting input. 13A-13E illustrate an exemplary process 1400 in accordance with some embodiments.

In process 1400, the user device receives (1402) a handwriting input from a user. The handwriting input includes a plurality of handwriting strokes provided in the handwriting input area of the handwriting input interface. In some embodiments, the user device recognizes (1404) a plurality of output characters from the handwriting input based on the handwriting recognition model. In some embodiments, the output characters are at least a first emoji character (eg, the stylized exclamation mark “!” or emoji character “♥” in Fig. 13D) and at least a first character (eg, from a script in natural human language) and at least a first character (eg , Letters from the word "Thanks" in Fig. 13D). In some embodiments, for example, as shown in FIG. 13D, the user device may display a first emoji character from a script in natural human language (eg, a stylized exclamation mark in FIG. 13D) in a candidate display area of the handwriting input interface. !" or the emoji character "♥") and a recognition result (e.g., result 1326 of FIG. 13D) including a first character (e.g., a character from the word "Thanks" in FIG. 13D) is displayed (1406). .

In some embodiments, based on the handwriting recognition model, the user device optionally recognizes at least a first semantic unit (eg, the word “thanks”) from the handwriting input (1408), wherein the first semantic unit is It contains individual letters, words or phrases capable of conveying individual semantic meanings in individual human language. In some embodiments, the user device identifies 1410 a second emoji character (eg, “handshake” emoji character) associated with a first semantic unit (eg, the word “Thanks”) recognized from the handwriting input. . In some embodiments, the user device includes, in a candidate display area of the handwriting input interface, a second recognition result (eg, a second recognition result comprising at least a second emoji character identified from a first semantic unit (eg, word “Thanks”) , A recognition result representing "!" and "♥" emoji characters following the "handshake" emoji character) is displayed (1412). In some embodiments, displaying the second recognition result represents the second recognition result, a third recognition result (eg, recognition result “Thanks!♥”) comprising at least a first semantic unit (eg, the word “Thanks”). It further includes displaying at the same time as ").

In some embodiments, the user receives a user input for selecting a first recognition result displayed in the candidate display area. In some embodiments, the user device, in response to the user input, inputs the text of the first recognition result selected in the text input area of the handwriting input interface, wherein the text is at least a first emoji from a script in natural human language. It includes a character and a first character. In other words, the user can enter mixed script text input using a single handwriting input (even if it is a handwriting input including multiple strokes) in the handwriting input area, without switching between the natural language keyboard and the emoji character keyboard.

In some embodiments, the handwriting recognition model was trained on a multiscript training corpus containing writing samples corresponding to characters of at least three non-overlapping scripts, and the three non-overlapping scripts were emoji characters, Chinese characters, And a set of Latin scripts.

In some embodiments, the user device identifies a second semantic unit (eg, the word “Love”) corresponding to the first emoji character (eg, “♥” emoji character) recognized directly from the handwriting input ( 1414). In some embodiments, the user device is at least a second semantic unit (eg, the word “Love”) identified from a first emoji character (eg, “♥” emoji character) in the candidate display area of the handwriting input interface. A fourth recognition result (eg, 1328 in FIG. 13D) including a is displayed (1416). In some embodiments, as shown in FIG. 13D, the user device sends a fourth recognition result (eg, result 1328 “Thanks! Love”) in the candidate display area, and a first recognition result (eg, result “Thanks”) in the candidate display area. ♥!") and display at the same time.

In some embodiments, the user device allows the user to enter regular text by drawing an emoji character. For example, if the user does not know how to spell the word "elephant", the user optionally draws a stylized emoji character for "elephant" in the handwriting input area, and the user device maps the handwriting input to "elephant". If it can be accurately recognized as an emoji character for, the user device optionally also presents the word "elephant" in plain text as one of the recognition results displayed in the candidate display area. In another example, the user may draw a stylized cat in the handwriting input area instead of writing the Chinese character “猫”. If the user device identifies the emoji character for “cat” based on the handwriting input provided by the user, the user device optionally also has the emoji character for “cat” in the candidate recognition results, and “cat” in Chinese. We present the Chinese character "猫" which means ". By presenting plain text for a recognized emoji character, the user device provides an alternative way of entering complex characters or words using several stylized strokes typically associated with a well-known emoji character. In some embodiments, the user device may display emoji characters in their corresponding plain text (e.g., letters, words, phrases, symbols) in one or more preferred scripts or languages (e.g., English or Chinese). Etc.)

In some embodiments, the user device recognizes the emoji character based on the visual similarity of the emoji character to the image generated from the handwriting input. In some embodiments, to enable recognition of emoji characters from handwriting input, the handwriting recognition model used on the user device includes handwriting samples corresponding to characters of a script in natural human language and also artificially designed emoji. It is trained using a training corpus that includes both handwriting samples corresponding to the set of characters. In some embodiments, emoji characters related to the same semantic concept may have a different appearance when used in mixed input into text of different natural languages. For example, an emoji character for the semantic concept of "Love" is a "heart" emoji character when presented as plain text in one natural language (eg, Japanese) and another natural language (eg, English or French). It may be a "kiss" emoji character when presented in plain text of

As described herein, when performing recognition of a multi-character handwriting input, the handwriting input module divides the currently accumulated handwriting input in the handwriting input area, and divides the accumulated strokes into one or more recognition units. One of the parameters used to determine how to divide the handwriting input may be the way strokes are clustered in the handwriting input area and the distance between different clusters of strokes. Until now, people have different fonts. Some people tend to write very sparse with large distances between strokes or different parts of the same letter, while others tend to write very tightly with very small distances between strokes or different characters. . Even for the same user, due to incomplete planning, handwritten characters may deviate from their balanced appearance and may be skewed, stretched, or squeezed in different ways. As described herein, since the multi-script handwriting recognition model provides stroke order independent recognition, a user can write characters or portions of characters out of order. As a result, it may be difficult to obtain spatial uniformity and balance in handwriting input between characters.

In some embodiments, the handwriting input model described herein informs the handwriting input module whether the user will merge two adjacent recognition units into a single recognition unit or divide a single recognition unit into two separate recognition units. Provides a way. With the help of the user, the handwriting input module can revise the initial segmentation and generate the result intended by the user.

15A-15J illustrate some example user interfaces and processes for a user to provide predetermined pinch and inflate gestures to modify recognition units identified by the user device.

")을 제시하였다.15A and 15B, the user inputs a plurality of handwritten strokes 1502 (eg, three strokes) in the handwriting input area 806 of the handwriting input interface 802. The user device has identified a single recognition unit based on the currently accumulated handwritten strokes 1502, and in the candidate display area 806, three candidate characters 1504, 1506, 1508 (e.g., "巾", "中", respectively. ", and "

") was presented.

In FIG. 15C, several additional strokes 1510 are additionally inputted to the right of the initial handwritten strokes 1502 in the handwriting input area 606. The user device determines that the strokes 1502 and 1510 should be considered as two separate recognition units (eg, based on the dimensions and spatial distributions of the plurality of strokes 1502, 1510). Based on the division of the recognition units, the user device provides input images of the first and second recognition units as a handwriting recognition model, and obtains two sets of candidate characters. Thereafter, the user device generates a plurality of recognition results (eg, 1512, 1514, 1516, 1518) based on different combinations of the recognized characters. Each recognition result includes a recognized character for the first recognition unit and a recognized character for the second recognition unit. As shown in FIG. 15C, each of the plurality of recognition results 1512, 1514, 1516, and 1518 includes two recognized characters.

In this example, the user intended a handwriting input that would in fact be recognized as a single character, but inadvertently the left part (eg, the left side "巾") and the right part (eg, the right side " It is assumed that there is too much space between "冒"). Looking at the results presented in the candidate display area 806 (eg, 1512, 1514, 1516, 1518), the user will notice that the user device has incorrectly divided the current handwriting input into two recognition units.

The segmentation may be based on an objective standard, but it would be undesirable for the user to delete the current handwriting input and rewrite the entire character again with a smaller distance between the left and right portions.

Instead, as shown in Fig. 15D, the user has two of the handwritten strokes 1502, 1510 to instruct the handwriting input module that the two recognition units identified by the handwriting input module should be merged as a single recognition unit. Use a pinch gesture on the clusters of dogs. The pinch gesture is indicated by two contacts 1520 and 1522 that are moved towards each other on the touch-sensitive surface.

", 및 "搢")을 획득하였다. 일부 실시예들에서, 도 15e에 도시된 바와 같이, 사용자 디바이스는 선택적으로 필기 획들의 왼쪽 클러스터와 오른쪽 클러스터 사이의 거리가 감소되도록 필기 입력 영역(806) 내의 필기 입력의 렌더링을 조정한다. 일부 실시예들에서, 사용자 디바이스는 핀치 제스처에 응답하여 필기 입력 영역(608)에 나타낸 필기 입력의 렌더링을 변경하지 않는다. 일부 실시예들에서, 사용자 디바이스는 필기 입력 영역(806)에서 검출된 (하나의 단일 접촉에 대조적으로) 2개의 동시 접촉들에 기초하여 입력 획으로부터 핀치 제스처를 구별한다.15E shows that in response to the user's pinch gesture, the user device revised the division of the currently accumulated handwriting input (eg, strokes 1502, 1510) and merged the handwritten strokes into a single recognition unit. 15E, the user device provides an input image based on the revised recognition unit as a handwriting recognition model, and three new candidate characters 1524, 1526, 1528 for the revised recognition unit (e.g., "帽", "

“, and “搢.” In some embodiments, as shown in Fig. 15E, the user device selectively includes a handwriting input area 806 such that the distance between the left and right clusters of handwritten strokes is reduced. Adjusts the rendering of handwriting input within In some embodiments, the user device does not change the rendering of the handwriting input shown in the handwriting input area 608 in response to the pinch gesture. In some embodiments, the user device does not change the rendering of the handwriting input shown in the handwriting input area 608. The pinch gesture is distinguished from the input stroke based on the two simultaneous contacts (as opposed to one single contact) detected in the input area 806.

As shown in FIG. 15F, the user inputs two more strokes 1530 to the right of the previously input handwriting input (ie, strokes for the character “帽”). The user device determines that the newly input strokes 1530 are new recognition units, and recognizes a candidate character (eg, “子”) for the newly identified recognition unit. Thereafter, the user device combines the newly identified character (e.g., "子") with candidate characters for the first identified recognition unit, and a number of different recognition results (e.g., results) in the candidate display area 806. (1532, 1534)).

As shown in FIG. 15G, following the handwritten strokes 1530, the user continues to write more strokes 1536 (eg, three more strokes) to the right of the strokes 1530. Because the horizontal distance between the strokes 1530 and the strokes 1536 is very small, the user device determines that the strokes 1530 and the strokes 1536 belong to the same recognition unit, and the input formed by the strokes 1530 and 1536 Provides the image as a handwriting recognition model. The handwriting recognition model identifies three different candidate characters for the revised recognition unit, and generates two revised recognition results 1538, 1540 for the currently accumulated handwriting input.

In this example, assume that the last two sets of strokes 1530, 1536 are in fact intended as two separate characters (eg, "子" and "±"). After the user finds out that the user device incorrectly combines the two sets of strokes 1530 and 1536 into a single recognition unit, the user divides the two sets of strokes 1530 and 1536 into two separate recognition units. Proceed to provide an inflation gesture to notify the user device that it should lose. As shown in Figure 15H, after the user has made two contacts 1542, 1544 around the strokes 1530, 1536, the two contacts are generally horizontally (i.e., along the default writing direction) with each other. Move away from

FIG. 15I shows that in response to the user's inflation gesture, the user device revises the previous segmentation of the currently accumulated handwriting input and assigns strokes 1530 and strokes 1536 to two consecutive recognition units. Based on the input images generated for two separate recognition units, the user device is configured with one or more candidate characters for a first recognition unit based on strokes 1530, and a second recognition unit based on strokes 1536. Identify one or more candidate characters for Thereafter, the user device generates two new recognition results 1546 and 1548 based on different combinations of recognized characters. In some embodiments, the user device optionally modifies the rendering of strokes 1536, 1536 to reflect the previously identified division of the recognition unit.

15J and 15K, the user has selected one of the candidate recognition results displayed in the candidate display area 806 (as indicated by the contact 1550), and the selected recognition result (e.g., Results 1548 were entered in the text input area 808 of the user interface. After the selected recognition result is input into the text input area 808, both the candidate display area 806 and the handwriting input area 804 are cleared and ready to display subsequent user input.

16A and 16B illustrate an exemplary process in which a user notifies the handwriting input module of how to divide or revise an existing segmentation of the current handwriting input using a predetermined gesture (e.g., a pinch gesture and/or an inflation gesture) ( 1600). 15J and 15K provide an illustration of an exemplary process 1600 in accordance with some embodiments.

In some embodiments, the user device receives (1602) a handwriting input from the user. The handwriting input includes a plurality of handwritten strokes provided on a touch-sensitive surface coupled to the device. In some embodiments, the user device renders a plurality of handwritten strokes in real time (1604) in a handwriting input area (eg, handwriting input area 806 in FIGS. For example, as shown in FIGS. 15D and 15H, the user device receives one of a pinch gesture input and an inflate gesture input for a plurality of handwritten strokes.

In some embodiments, for example, as shown in FIGS. 15C-15E, the user device, upon receiving a pinch gesture input, processes the plurality of handwritten strokes as a single recognition unit, thereby processing the plurality of handwritten strokes based on the plurality of handwritten strokes. A first recognition result is generated (1606).

In some embodiments, for example, as shown in FIGS. 15G-15I, the user device, upon receiving the inflation gesture input, separates a plurality of handwritten strokes by the inflation gesture input, by two separate recognition units. Processing as a field generates a second recognition result based on the plurality of handwritten strokes (1608).

In some embodiments, for example, as shown in FIGS. 15E and 15I, the user device, when generating the individual recognition results of the first recognition result and the second recognition result, in the candidate display area of the handwriting input interface. The generated recognition result is displayed.

In some embodiments, the pinch gesture input includes two simultaneous contacts that converge towards each other in an area occupied by a plurality of handwritten strokes on the touch-sensitive surface. In some embodiments, the inflation gesture input includes two simultaneous contacts leaving each other in an area occupied by a plurality of handwritten strokes on the touch-sensitive surface.

In some embodiments, the user device identifies two adjacent recognition units from a plurality of handwritten strokes (eg, 1614). For example, as shown in Fig. 15C, the user device, in the candidate display area, includes initial recognition results (e.g., results 1512, 1514, 1516 of Fig. 15C) including individual characters recognized from two adjacent recognition units. , 1518)) is displayed (1616). In some embodiments, when displaying the first recognition result (e.g., result 1524, 1526, or 1528 in FIG. 15E) in response to the pinch gesture, the user device first recognizes the initial recognition result in the candidate display area. Substitute for the result (1618). In some embodiments, as shown in FIG. 15D, the user device receives (1620) a pinch gesture input while the initial recognition result is displayed in the candidate display area. In some embodiments, for example, as shown in FIG. 15E, the user device reads a plurality of handwritten strokes to reduce the distance between two adjacent recognition units in the handwriting input area in response to a pinch gesture input. Render (1622).

" "祉")를 포함하는 초기 인식 결과(예컨대, 도 15g의 결과(1538 또는 1540))를 디스플레이한다(1626). 일부 실시예들에서, 예컨대, 도 15h 및 도 15i에 도시된 바와 같이, 팽창 제스처에 응답하여 제2 인식 결과(예컨대, 도 15i의 결과(1546 또는 1548))를 디스플레이할 때, 사용자 디바이스는 후보 디스플레이 영역에서 초기 인식 결과(예컨대, 결과(1538 또는 1540))를 제2 인식 결과(예컨대, 결과(1546 또는 1548))로 대신한다(1628). 일부 실시예들에서, 도 15h에 도시된 바와 같이, 사용자 디바이스는 초기 인식 결과가 후보 디스플레이 영역에서 디스플레이되는 동안 팽창 제스처 입력을 수신한다(1630). 일부 실시예들에서, 도 15h 및 도 15i에 도시된 바와 같이, 사용자 디바이스는, 팽창 제스처 입력에 응답하여, 필기 입력 영역에서 제1 인식 유닛에 배정되는 획들의 제1 서브세트와 제2 인식 유닛에 배정되는 필기 획들의 제2 서브세트 사이의 거리를 증가시키도록 복수의 필기 획들을 리렌더링한다(1632).In some embodiments, the user device identifies (1624) a single recognition unit from the plurality of handwritten strokes. The user device may, in the candidate display area, a character recognized from a single recognition unit (e.g., "

An initial recognition result (e.g., result 1538 or 1540 in Fig. 15G) including ““祉”) is displayed 1626. In some embodiments, for example, as shown in Figs. 15H and 15I, When displaying the second recognition result (e.g., result 1546 or 1548 in FIG. 15I) in response to the inflation gesture, the user device secondly displays the initial recognition result (e.g., result 1538 or 1540) in the candidate display area. Replaced with the recognition result (e.g., result 1546 or 1548) (1628). In some embodiments, as shown in Figure 15H, the user device inputs an inflation gesture while the initial recognition result is displayed in the candidate display area. 15H and 15I, in some embodiments, the user device, in response to the inflation gesture input, receives a first of the strokes assigned to the first recognition unit in the handwriting input area. The plurality of handwritten strokes are re-rendered to increase the distance between the subset and the second subset of handwritten strokes assigned to the second recognition unit (1632).

In some embodiments, the user optionally recommends processing the plurality of strokes as a single recognition unit immediately after the user provides the strokes and realizes that the strokes may be too spread out for accurate segmentation based on the standard segmentation process. Provides a pinch gesture to notify the user device. The user device can distinguish the pinch gesture from the regular stroke based on the two simultaneous contacts present in the pinch gesture. Similarly, in some embodiments, the user may optionally add a plurality of strokes to two separate strokes, immediately after the user provides them and realizes that the strokes may be too close together to make an accurate segmentation based on the standard segmentation process. Provides an inflation gesture to notify the user device of processing as the recognition units of. The user device can distinguish the inflation gesture from the regular stroke based on the two simultaneous contacts present in the pinch gesture.

,등)에서 2개의 하위 구성요소들을 병합하거나 또는 단일 구성요소를 나누는 것을 제공할 수 있다. 이것은, 사용자들이 손으로 복잡한 복합 문자를 기입할 때 정확한 비율들 및 균형을 잃는 경향이 있기 때문에, 복잡한 복합 한자들을 인식하는 데 특히 도움이 된다. 필기 입력의 완료 후에, 예컨대, 핀치 및 팽창 제스처들에 의해, 필기 입력의 비율들 및 균형을 조정할 수 있는 것은, 정확한 비율들 및 균형으로 되도록 여러 가지 시도를 하지 않고도 사용자가 정확한 문자를 입력하는 데 특히 도움이 된다.In some embodiments, the direction of movement of the pinch or inflate gesture is optionally used to provide additional guidance on how to divide the strokes under the gesture. For example, when multi-line handwriting input is enabled for the handwriting input area, a pinch gesture with two contacts moving in the vertical direction converts two recognition units identified in two adjacent lines into a single recognition unit ( For example, the handwriting input module may be notified of merging (as the upper and lower copies). Similarly, an inflation gesture with two contacts moving in the vertical direction can inform the handwriting input module of dividing a single recognition unit into two recognition units in two adjacent lines. In some embodiments, the pinch and inflate gestures also guide division in the lower part of the character input, e.g., for different parts of the compound character (e.g., top, bottom, left or right parts), the compound character (

,, etc.), it may be provided to merge two sub-elements or to divide a single element. This is particularly helpful in recognizing complex complex Chinese characters, as users tend to lose the correct proportions and balance when writing complex complex characters by hand. Being able to adjust the proportions and balance of the handwriting input after completion of the handwriting input, e.g., by pinch and inflation gestures, allows the user to input the correct character without making various attempts to get the correct proportions and balance It is especially helpful.

As described herein, the handwriting input module allows a user to input multi-character handwriting input, spanning multiple characters in the handwriting input area, and even multiple phrases, sentences, and/or Allows out of order strokes for multi-character handwriting input within a character, even across lines. In some embodiments, the handwriting input module also provides character-by-character deletion in the handwriting input area, wherein the order of character deletion is independent of when the strokes for each character are provided in the handwriting input area and in reverse writing direction. direction). In some embodiments, the deletion of each recognition unit (e.g., a character or a number of copies) in the handwriting input area is optionally performed on a stroke-by-stroke basis, wherein the strokes are in a reverse temporal order provided within the recognition unit. It is deleted. 17A-17H illustrate exemplary user interfaces for responding to a deletion input from a user and providing character-by-character deletion in multi-character handwriting input.

", "凶")은 전부 사전결정된 임계치 미만의 인식 신뢰도를 갖는다. 일부 실시예들에서, 사용자 디바이스는 후보 디스플레이 영역(806)에서 제1 인식 유닛에 대한 후보 문자(예컨대, "日")만을 포함하고 제2 인식 유닛에 대한 어떠한 후보 문자도 포함하지 않는 부분 인식 결과(예컨대, 결과(1712))를 제시한다. 일부 실시예들에서, 사용자 디바이스는, 인식 신뢰도가 사전결정된 임계치를 통과하였는지의 여부에 상관없이, 인식 결과들 둘 모두에 대한 후보 문자를 포함하는 전체 인식 결과(예컨대, 결과(1714 또는 1716))를 추가로 디스플레이한다. 부분 인식 결과를 제공하는 것은 필기 입력의 어느 부분이 개정될 필요가 있음을 사용자에게 통지한다. 추가적으로, 사용자는 또한 필기 입력의 정확하게 인식된 부분을 먼저 입력한 후에, 정확하게 인식되지 않은 부분을 재기입하도록 선정할 수 있다.17A, a user has provided a plurality of handwritten strokes 1702 in the handwriting input area 804 of the handwriting input user interface 802. Based on the currently accumulated strokes 1702, the user device presents three recognition results (eg, results 1704, 1706, 1708) in the candidate display area 806. As shown in FIG. 17B, the user has provided a plurality of additional strokes 1710 in the handwriting input area 806. The user device recognizes the 3 new output characters and replaces the 3 previous recognition results 1704, 1706, 1708 with the 3 new recognition results 1712, 1714, 1716. In some embodiments, as shown in FIG. 17B, the cluster of strokes 1710, even though the user device has identified two separate recognition units from the current handwriting input (e.g., strokes 1702 and strokes 1710). Does not correspond well to any known characters in the repertoire of the handwriting recognition module. As a result, candidate characters identified for the recognition unit including strokes 1710 (e.g., "

", "凶") all have a recognition reliability less than a predetermined threshold. In some embodiments, the user device only has a candidate character (e.g., "Day") for the first recognition unit in the candidate display area 806. And present a partial recognition result (e.g., result 1712) that contains and does not contain any candidate characters for the second recognition unit. In some embodiments, the user device determines whether the recognition reliability has passed a predetermined threshold. Regardless of whether or not, it additionally displays the entire recognition result (eg, result 1714 or 1716), including candidate characters for both of the recognition results, providing a partial recognition result which part of the handwriting input is revised. In addition, the user may also elect to first enter the correctly recognized portion of the handwriting input and then rewrite the portion that was not correctly recognized.

")를 인식하고, 새로운 인식 결과들(1720, 1722, 1724)의 세트를 생성한다. 또 다시, 획들(1710)에 대해 식별된 후보 문자들 중 어느 것도 사전결정된 신뢰도 임계치를 충족하지 않기 때문에, 제1 인식 결과(1720)는 부분 인식 결과이다.17C shows that the user has continued to provide an additional handwritten stroke 1718 to the left of the strokes 1710. Based on the relative position and distance of the stroke 1718, the user device determines that the newly added stroke belongs to the same recognition unit as the cluster of handwritten strokes 1702. Based on the revised recognition units, a new character for the first recognition unit (e.g., "

") and generates a set of new recognition results 1720, 1722, 1724. Again, since none of the candidate characters identified for strokes 1710 meet a predetermined confidence threshold, The first recognition result 1720 is a partial recognition result.

")에 대한 모든 필기 획들의 입력을 완료하였고, 정확한 인식 결과(1728)가 후보 디스플레이 영역(806)에 나타내진다.17D shows that the user has now entered a plurality of new strokes 1726 between strokes 1702 and 1710. The user device allocates newly inputted strokes 1726 to the same recognition unit as the strokes 1710. Now, the user has two Chinese characters (e.g., "

Input of all handwritten strokes for ") has been completed, and an accurate recognition result 1728 is shown in the candidate display area 806.

27E shows that the user has entered the initial portion of the delete input, for example by making a light contact 1730 on the delete button 1732. When the user maintains contact with the delete button 1732, the user can delete the current handwriting input in character units (or recognition units). Deletion is not performed for all handwriting inputs at the same time.

"에 대한 인식 유닛)은 필기 입력 영역(804) 내에 동시에 디스플레이되는 다른 인식 유닛(들)에 비해 시각적으로 강조된다(예컨대, 경계선(1734) 또는 밝게 된 배경 등으로 강조된다).In some embodiments, as shown in Fig. 17E, when the user's finger first touches the delete button 1732 on the touch-sensitive screen, the last recognition unit (e.g., from left to right) in the default writing direction (e.g. , text "

The recognition unit for ") is visually emphasized compared to other recognition unit(s) displayed simultaneously in the handwriting input area 804 (e.g., with a border 1734 or a brightened background, etc.).

In some embodiments, as shown in FIG. 17F, when the user device detects that the user has maintained the contact 1730 on the delete button 1730 for longer than the threshold duration, the user device Remove the highlighted recognition unit (eg, in box 1734) from 806. Additionally, as shown in Fig. 17F, the user device also revises the recognition results shown in the candidate display area 608 to delete any output characters generated based on the deleted recognition unit.

"에 대한 인식 유닛)이 삭제될 다음 인식 유닛이 됨을 추가로 도시한다. 도 17f에 도시된 바와 같이, 나머지 인식 유닛은 (예컨대, 박스(1736) 내에서) 시각적으로 강조되었고, 삭제될 준비가 된다. 일부 실시예들에서, 인식 유닛의 시각적 강조는 사용자가 계속해서 삭제 버튼과의 접촉을 유지하는 경우 삭제되게 될 인식 유닛의 미리보기를 제공한다. 사용자가 임계 지속기간에 도달하기 전에 삭제 버튼과의 접촉을 중단하는 경우, 시각적 강조가 마지막 인식 유닛으로부터 제거되고, 인식 유닛은 삭제되지 않는다. 당업자가 인식하는 바와 같이, 접촉의 지속기간은 인식 유닛이 삭제될 때마다 재설정된다. 추가적으로, 일부 실시예들에서, 접촉 세기(예컨대, 사용자가 터치 감응 스크린과의 접촉(1730)을 가한 압력)는 선택적으로 현재 강조된 인식 유닛을 삭제하려는 사용자의 의도를 확인하기 위해 임계 지속기간을 조정하도록 사용된다. 도 17f 및 도 17g는, 사용자가 임계 지속기간에 도달하기 전에 삭제 버튼(1732) 상에서의 접촉(1730)을 중단하였으며 문자 "

"에 대한 인식 유닛이 필기 입력 영역(806)에서 보존됨을 도시한다. 도 17g 및 도 17h에 도시된 바와 같이, 사용자가 인식 유닛에 대한 제1 인식 결과(예컨대, 결과(1738))를 (예컨대, 접촉(1740)에 의해 지시되는 바와 같이) 선택했을 때, 제1 인식 결과(1738)의 텍스트가 텍스트 입력 영역(808)으로 입력된다.FIG. 17F shows the case where the user continues to maintain contact 1730 on the delete button 1732 after the last recognition unit in the handwriting input area 806 (eg, the recognition unit for the character “腦”) has been deleted. , For the deleted recognition unit, adjacent recognition units (e.g., characters "

It further shows that the recognition unit for ") will be the next recognition unit to be deleted. As shown in Fig. 17F, the remaining recognition units are visually highlighted (eg, in box 1736) and ready to be deleted. In some embodiments, the visual highlighting of the recognition unit provides a preview of the recognition unit that will be deleted if the user continues to maintain contact with the delete button. When stopping contact with, the visual emphasis is removed from the last recognition unit and the recognition unit is not deleted As one skilled in the art will recognize, the duration of the contact is reset each time the recognition unit is deleted. In embodiments, the contact intensity (e.g., the pressure the user exerted on contact 1730 with the touch-sensitive screen) is used to optionally adjust the threshold duration to confirm the user's intention to delete the currently highlighted recognition unit. 17F and 17G show that the user stopped contacting 1730 on the delete button 1732 before reaching the threshold duration and the text "

Shows that the recognition unit for "is preserved in the handwriting input area 806. As shown in Figs. 17G and 17H, the user writes a first recognition result (e.g., result 1738) for the recognition unit (e.g. , As indicated by the contact 1740), the text of the first recognition result 1738 is entered into the text input area 808.

18A and 18B are flowcharts of an exemplary process 1800 in which a user device provides character-by-character deletion in multi-character handwriting input. In some embodiments, the deletion of the handwriting input is performed before characters recognized from the handwriting input are identified and entered into the text input area of the user interface. In some embodiments, deletion of characters in the handwriting input proceeds according to a reverse spatial order of recognition units identified from the handwriting input, and is independent of the temporal sequence in which the recognition units are formed. 17A-17H illustrate an exemplary process 1800 in accordance with some embodiments.

"의 최후에 나타나는 문자 "

")를 제거한다(1812).18A, in an exemplary process 1800, the user device receives 1802 handwriting input from the user, the handwriting input being in a handwriting input area of the handwriting input interface (e.g., area 804 in Fig. 17D). Includes a plurality of handwritten strokes provided in ). The user device identifies (1804) a plurality of recognition units from the plurality of handwritten strokes, each recognition unit comprising a separate subset of the plurality of handwritten strokes. For example, as shown in FIG. 17D, the first recognition unit includes strokes 1702 and 1718, and the second recognition unit includes strokes 1710 and 1726. The user device generates a multi-character recognition result (eg, result 1728 in Fig. 17D) including individual characters recognized from the plurality of recognition units (1806). In some embodiments, the user device displays a multi-character recognition result (eg, result 1728 in FIG. 17D) in a candidate display area of the handwriting input interface. In some embodiments, for example, as shown in FIG. 17E, while the multi-character recognition result is displayed in the candidate display area, the user device receives a delete input from the user (e.g., contact 1730 on the delete button 1732). ) Is received (1810). In some embodiments, in response to receiving a delete input, e.g., as shown in FIGS. 17E and 17F, the user device recognizes the multi-character displayed in the candidate display area (e.g., candidate display area 806). From the result (e.g., result 1728) the last character (e.g., a spatial sequence"

"Letter appearing at the end of"

") is removed (1812).

")에 대응한다.In some embodiments, for example, as shown in FIGS. 17A-17D, the user device renders the plurality of handwritten strokes in real time in the handwriting input area of the handwriting input interface as the plurality of handwritten strokes are provided by the user. (1814). In some embodiments, in response to receiving the deletion input, the user device, from the handwriting input area (e.g., handwriting input area 804 in Fig. 17E), is formed by a plurality of recognition units in the handwriting input area. A separate subset of the plurality of handwritten strokes corresponding to the last recognition unit (eg, a recognition unit including strokes 1726 and 1710) in the spatial sequence is removed (1816). The last recognition unit is the last character (e.g., the character" in the multi-character recognition result (e.g., result 1728 in Fig.

Corresponds to ").

In some embodiments, the last recognition unit does not include (1818) the last handwritten stroke in time among the plurality of handwritten strokes provided by the user. For example, if the user provided stroke 1718 after he or she provided strokes 1726, 1710, the last recognition unit containing strokes 1726, 1710 will still be deleted first.

In some embodiments, in response to receiving the initial portion of the deletion input, e.g., as shown in Fig. 17E, the user device visually distinguishes the last recognition unit from other recognition units identified in the handwriting input area. (1820). In some embodiments, the initial portion of the deletion input is the initial contact detected on the delete button in the handwriting input interface, and the deletion input is detected when the initial contact lasts longer than a predetermined threshold amount of time (1822).

In some embodiments, the last recognition unit corresponds to a handwritten Chinese character. In some embodiments, the handwriting input is written in cursive. In some embodiments, the handwriting input corresponds to multiple Chinese characters written in a cursive font. In some embodiments, at least one of the handwritten strokes is divided into two adjacent recognition units of the plurality of recognition units. For example, sometimes, a user may use a long stroke that is connected with a number of characters, in which case the division module of the handwriting input module selectively divides the long stroke into several recognition units. When the deletion of the handwriting input is performed on a character basis (or on a recognition unit basis), only segments of long strokes (eg, segments in a corresponding recognition unit) are deleted at a time.

In some embodiments, the delete input is a sustained contact on the delete button provided in the handwriting input interface, and removing an individual subset of a plurality of handwritten strokes means that the subset of handwritten strokes are in reverse temporal order provided by the user On a per stroke basis, it further includes removing the subset of handwritten strokes in the last recognition unit from the handwriting input area (1824).

In some embodiments, for example, as shown in FIGS. 17B and 17C, the user device generates (1826) a partial recognition result comprising a subset of individual characters recognized from the plurality of recognition units, wherein the individual Each of the subset of characters meets a predetermined confidence threshold. In some embodiments, the user device converts the partial recognition result (e.g., the result 1712 of FIG. 17B and the result 1720 of FIG. 17C) into the multi-character recognition result (e.g., results ( 1714, 1722)) and simultaneously display (1828).

In some embodiments, the partial recognition result does not include at least the last character in the multicharacter recognition result. In some embodiments, the partial recognition result does not include at least an initial character in the multicharacter recognition result. In some embodiments, the partial recognition result does not include at least an intermediate character in the multicharacter recognition result.

In some embodiments, the minimum unit of deletion is the number of copies, and the handwriting inputs are deleted one copy at a time whenever the number of copies becomes the last recognition unit in handwriting inputs still remaining in the handwriting input area.

As described herein, in some embodiments, the user device provides both a horizontal write mode and a vertical write mode. In some embodiments, the user device allows a user to enter text in both or one of a left-to-right writing direction and a right-to-left direction in a horizontal writing mode. In some embodiments, the user device allows the user to input text in both or both a top-down writing direction and a bottom-up writing direction in a vertical writing mode. In some embodiments, the user device provides various affordances (eg, write mode or write direction button) on the user interface to invoke a separate writing mode and/or writing direction for current handwriting inputs. In some embodiments, the text input direction in the text input area is basically the same as the handwriting input direction in the handwriting input direction. In some embodiments, the user device allows the user to manually set the input direction in the text input area and the writing direction in the handwriting input area. In some embodiments, the text display direction in the candidate display area is basically the same as the handwriting input direction in the handwriting input area. In some embodiments, the user device allows the user to manually set the text display direction in the text input area irrespective of the handwriting input direction in the handwriting input area. In some embodiments, the user device associates a writing mode and/or writing direction of the handwriting input interface with a corresponding device orientation, and the change in device orientation automatically causes a change in the writing mode and/or writing direction. In some embodiments, the change in the writing direction automatically causes the input of the highest recognition result to be entered into the text input area.

19A-19F illustrate exemplary user interfaces of a user device that provides both a horizontal input mode and a vertical input mode.

19A shows a user device in a horizontal input mode. In some embodiments, as shown in FIG. 19A, a horizontal input mode is provided when the user device is in a landscape orientation. In some embodiments, a horizontal input mode is optionally provided in association with when the device is operated in a vertical orientation. In different applications, the association between device orientation and write mode may be different.

In the horizontal input mode, the user may provide handwritten characters in a horizontal writing direction (eg, having a default writing direction running from left to right or a default writing direction running from right to left). In the horizontal input mode, the user device divides the handwriting input into one or more recognition units along the horizontal writing direction.

In some embodiments, the user device only allows single line input in the handwriting input area. In some embodiments, as shown in Fig. 19A, the user device allows multi-line input (two lines of input) in the handwriting input area. In FIG. 19A, a user has provided a plurality of handwritten strokes in a handwriting input area 806 in several rows. Based on the sequence in which the user provided the plurality of handwritten strokes and the relative positions of the plurality of handwritten strokes and the distances between the plurality of handwritten strokes, the user device determines that the user has input two lines of characters. After dividing the handwriting input into two separate lines, the device determines the recognition unit(s) within each line.

As shown in FIG. 19A, the user device recognized the individual character for each recognition unit currently identified in the handwriting input 1902, and generated a number of recognition results 1904 and 1906. As further shown in FIG. 19A, in some embodiments, if the output character (eg, letter “I”) for a particular set of recognition units (eg, recognition unit formed by the initial stroke) is weak, the user device Optionally generates a partial recognition result (e.g., result 1906) representing only output characters with sufficient recognition reliability. In some embodiments, the user may notice from the partial recognition result 1906 that the first stroke may be revised for the recognition model or may be individually deleted or rewritten to generate an accurate recognition result. In this particular example, editing of the first recognition unit is not necessary because the first recognition unit 1904 represents the desired recognition result for the first recognition unit.

In this example, as shown in Figures 19A and 19B, the user has rotated the device in a portrait orientation (eg, shown in Figure 19B). As shown in Fig. 19B, in response to a change in device orientation, the handwriting input interface is changed from a horizontal input mode to a vertical input mode. In the vertical input mode, the layout of the handwriting input area 804, the candidate display area 806, and the text input area 808 may be different from those shown in the horizontal input mode. The specific layout of horizontal and vertical input modes can be varied to suit different device shapes and application needs. In some embodiments, with the rotation of the device orientation and the change of the input mode, the user device automatically enters the top result (eg, result 1904) into the text input area 808 as text input 1910. The orientation and position of the cursor 1912 also reflect changes in the input mode and writing direction.

In some embodiments, the change in the input mode is selectively triggered by the user touching a particular input mode selection affordance 1908. In some embodiments, the input mode selection affordance is a graphical user interface element that also indicates a current writing mode, a current writing direction, and/or a current paragraph direction. In some embodiments, the input mode selection affordance may cycle through all available input modes and writing directions provided by the handwriting input interface 802. 19A, the affordance 1908 indicates that the current input mode is a horizontal input mode, the writing direction is from left to right, and the paragraph direction is from top to bottom. In Fig. 19B, affordance 1908 indicates that the current input mode is a vertical input mode, the writing direction is from top to bottom, and the paragraph direction is from right to left. Other combinations of a writing direction and a paragraph direction are possible according to various embodiments.

"에 대한 필기 획들)을 입력하였다. 필기 입력은 수직 기입 방향으로 기입된다. 사용자 디바이스는 필기 입력을 수직 방향으로 2개의 인식 유닛들로 분할하고, 각각이 수직 방향으로 레이아웃된 2개의 인식된 문자들을 포함하는 2개의 인식 결과들(1916, 1918)을 디스플레이한다.As shown in Fig. 19C, the user selects a plurality of new strokes 1914 (eg, two Chinese characters) in the handwriting input area 804 in the vertical input mode.

The handwriting input is written in the vertical writing direction. The user device divides the handwriting input into two recognition units in the vertical direction, and two recognized characters each laid out in the vertical direction. It displays two recognition results 1916 and 1918 including.

19C and 19D illustrate that when a user selects a displayed recognition result (eg, result 1916), the selected recognition result is inputted into the text input area 808 in a vertical direction.

19E and 19F illustrate that the user inputs the handwriting input 1920 of additional lines in the vertical writing direction. The lines continue from left to right according to the paragraph direction of traditional Chinese writing. In some embodiments, candidate display area 806 also represents recognition results (eg, results 1922, 1924) in the same writing direction and paragraph direction as for the handwriting input area. In some embodiments, different writing directions and paragraph directions may be provided by default according to the main language associated with the user device or the language (eg, Arabic, Chinese, Japanese, English, etc.) of a soft keyboard installed on the user device .

19E and 19F also show that when the user selects a recognition result (eg, result 1922), the text of the selected recognition result is input into the text input area 808. Accordingly, as shown in Fig. 19F, the current text input in the text input area 808 is both text written in the horizontal mode in the left-to-right writing direction, and the text written in the vertical mode in the top-down writing direction. Includes. Paragraph direction for horizontal text is top-down, while paragraph direction for vertical text is from right to left.

In some embodiments, the user device allows the user to separately establish preferred writing directions and paragraph directions for each of the handwriting input area 804, the candidate display area 806 and the text input area 808. In some embodiments, the user device allows the user to establish a preferred writing direction and paragraph direction for each of the handwriting input area 804, the candidate display area 806 and the text input area 808, which are associated with each device orientation To be able to do it.

20A-20C are flowcharts of an exemplary process 2000 for changing a text input direction and a handwriting input direction of a user interface. 19A-19F illustrate a process 2000 in accordance with some embodiments.

In some embodiments, the user device determines the orientation of the device (2002). The orientation of the device and changes in device orientation can be detected by an accelerometer and/or other orientation sensing element within the user device. In some embodiments, the user device provides a handwriting input interface on the device in a horizontal input mode according to the device in the first orientation (2004). Individual lines of handwriting input input in the horizontal input mode are divided into one or more individual recognition units along the horizontal writing direction. In some embodiments, the device provides a handwriting input interface on the device in a vertical input mode according to the device in the second orientation (2006). Individual lines of handwriting input input in the vertical input mode are divided into one or more individual recognition units along the vertical writing direction.

In some embodiments, while operating in a horizontal input mode (2008): The device detects a change in device orientation from a first orientation to a second orientation (2010). In some embodiments, in response to a change in device orientation, the device switches from a horizontal input mode to a vertical input mode (2012). This is illustrated in Figs. 19A and 19B, for example. In some embodiments, while operating in the vertical input mode (2014): the user device detects a change in device orientation from the second orientation to the first orientation (2016). In some embodiments, in response to the change in device orientation, the user device switches from a vertical input mode to a horizontal input mode (2018). In some embodiments, the association between device orientation and input mode may be the opposite of that described above.

In some embodiments, while operating in the horizontal input mode (2020): The user device receives (2022) a first multi-word handwriting input from the user. In response to the first multi-word handwriting input, the user device presents the first multi-word recognition result in the candidate display area of the handwriting input interface along the horizontal writing direction (2024). This is illustrated in Fig. 19A, for example. In some embodiments, while operating in the vertical input mode (2026): The user device receives (2028) a second multi-word handwriting input from the user. In response to the second multi-word handwriting input, the user device presents the second multi-word recognition result in the candidate display area along the vertical writing direction (2030). This is illustrated, for example, in FIGS. 19C and 19E.

In some embodiments, for example, as shown in FIGS. 19A and 19B, the user device receives (2032) a first user input selecting a first multi-word recognition result, where the selection is to change the input direction. Input (eg, rotation of the device or selection of affordance 1908). For example, as illustrated in FIG. 19C or 19E, the user device receives a second user input for selecting a second multi-word recognition result (2034). The user device simultaneously displays the individual texts of the first multi-word recognition result and the second multi-word recognition result in the text input area of the handwriting input interface (2036), wherein the individual texts of the first multi-word recognition result set the horizontal writing direction. Therefore, it is displayed, and the individual texts of the second multi-word recognition result are displayed along the vertical writing direction. This is illustrated, for example, in the text input area 808 of Fig. 19F.

In some embodiments, the handwriting input area allows multiple lines of handwriting input in a horizontal writing direction and has a default top-down paragraph direction. In some embodiments, the horizontal writing direction is from left to right. In some embodiments, the horizontal writing direction is from right to left. In some embodiments, the handwriting input area allows multiple lines of handwriting input in a vertical writing direction and has a default left-to-right paragraph direction. In some embodiments, the handwriting input area allows multiple lines of handwriting input in a vertical writing direction and has a default right-to-left paragraph direction. In some embodiments, the vertical writing direction is from top to bottom. In some embodiments, the first orientation is a horizontal orientation by default and the second orientation is a vertical orientation by default. In some embodiments, the user device provides a separate affordance in the handwriting input interface to manually switch between a horizontal input mode and a vertical input mode, regardless of the device orientation. In some embodiments, the user device provides a separate affordance in the handwriting input interface to manually switch between two alternative writing directions. In some embodiments, the user device provides a separate affordance in the handwriting input interface to manually switch between two alternative paragraph directions. In some embodiments, the affordance is a toggle button that, when called one or more consecutively, cycles through each available combination of input and paragraph directions.

In some embodiments, the user device receives 2038 handwriting input from the user. The handwriting input includes a plurality of handwriting strokes provided in the handwriting input area of the handwriting input interface. In response to the handwriting input, the user device displays one or more recognition results in a candidate display area of the handwriting input interface (2040). While one or more recognition results are displayed in the candidate display area, the user device detects (2042) a user input for switching from the current handwriting input mode to an alternative handwriting input mode. In response to user input (2044): the user device switches from the current handwriting input mode to an alternative handwriting input mode (2046). In some embodiments, the user device clears the handwriting input from the handwriting input area (2048). In some embodiments, the user device automatically inputs the highest recognition result among one or more recognition results displayed in the candidate display area into the text input area of the handwriting input interface (2050 ). This is illustrated, for example, in Figs. 19A and 19B, where the current handwriting input mode is a horizontal input mode and an alternative handwriting input mode is a vertical input mode. In some embodiments, the current handwriting input mode is a vertical input mode and an alternative handwriting input mode is a horizontal input mode. In some embodiments, the current handwriting input mode and alternative handwriting input mode are modes in which any two different handwriting input directions or paragraph directions are provided. In some embodiments, the user input is a rotation of the device from the current orientation to another orientation (2052). In some embodiments, the user input is a call of affordance to manually switch the current handwriting input mode to an alternative handwriting input mode.

As described herein, the handwriting input module allows a user to input handwritten strokes and/or characters in any chronological order. Therefore, it is advantageous to delete individual handwritten characters in the multi-character handwriting input, and to rewrite the same or different handwritten characters as the deleted characters at the same location, which allows the user to revise the long handwriting input without having to delete the entire handwriting input. Because it will help.

20A-20H illustrate exemplary user interfaces for visually highlighting and/or deleting a recognition unit identified in a plurality of currently accumulated handwriting strokes in a handwriting input area. Allowing a user to individually select, view, and delete any one of a number of recognition units identified in a plurality of inputs, especially when multi-character and even multi-line handwriting input is allowed by the user device. useful. Allowing the user to delete a particular recognition unit at the beginning or middle of handwriting inputs allows the user to correct long inputs without requiring the user to delete all recognition units located after the undesirable recognition unit.

21A to 21C, the user has provided a plurality of handwritten strokes (eg, strokes 2102, 2104, 2106) in the handwriting input area 804 of the handwriting input user interface 802. While the user continues to provide additional strokes to the handwriting input area 804, the user device updates the recognition units identified from the currently accumulated handwriting input in the handwriting input area, and output characters recognized from the updated recognition units. Recognition results are revised according to. As shown in Fig. 20C, the user device has identified two recognition units from the current handwriting input, and presented three recognition results (eg, 2108, 2010, 2112) each containing two Chinese characters.

In this example, after the user writes two handwritten characters, the user notices that the first recognition unit is written incorrectly, and as a result, the user device did not identify and failed to present the desired recognition result in the candidate display area. .

In some embodiments, when the user provides a tap gesture on the touch-sensitive display (e.g., an immediate lift following a contact in the same location), the user device interprets the tap gesture as an input, identifying the current in the handwriting input area. Resulting in a visual emphasis of individual recognition units. In some embodiments, another predetermined gesture (eg, a multi-finger swipe gesture over the handwriting input area) is used to cause the user device to highlight individual recognition units in the handwriting input area 804. A tap gesture is sometimes desirable because it is relatively easy to distinguish from a handwritten stroke, which typically has a movement of the contact within the handwriting input area 804 and entails a longer duration of sustained contact. A multi-touch gesture is sometimes desirable because it is relatively easy to distinguish from a handwritten stroke, which typically involves a single contact within the handwriting input area 804. In some embodiments, the user device provides an affordance 2112 in the user interface that can be invoked by the user (e.g., via contact 2114) so that individual recognition units (e.g., boxes 2108, 2110 ) To be visually emphasized). In some embodiments, affordance is desirable when there is sufficient screen space to accommodate such affordance. In some embodiments, affordance may be invoked multiple consecutively by the user, which allows the user device to visually highlight the recognition unit(s) identified according to different split chains in the split grid and all split chains would have been shown. When the emphasis is turned off.

As shown in Fig. 21D, when the user has provided the necessary gesture to highlight the individual recognition units in the handwriting input area 804, the user device is given an individual deletion affordance (e.g., small deletion) over each highlighted recognition unit. Buttons 2116, 2118) are further displayed. 21E and 21F show when the user touches (e.g., via contact 2120) the deletion affordance of an individual recognition unit (e.g., delete button 2116 for the first recognition unit in box 2118), It shows that individual recognition units (eg, in box 2118) are removed from the handwriting input area 804. In this particular example, the deleted recognition unit is not the last input recognition unit temporally, and it is not the last recognition unit spatially along the writing direction. In other words, the user can delete any recognition unit regardless of when and where the recognition unit was provided in the handwriting input area. 21F shows that in response to deletion of the first recognition unit in the handwriting input area, the user device also updates the recognition results displayed in the candidate display area 806. As shown in Fig. 21F, the user device also deleted the candidate character corresponding to the deleted recognition unit from the recognition results. As a result, a new recognition result 2120 is shown in the candidate display area 806.

21G and 21H, after the first recognition unit is removed from the handwriting input interface 804, the user applies a plurality of new handwritten strokes 2122 to the area previously occupied by the deleted recognition unit. Provided. The user device redistributes the currently accumulated handwriting input in the handwriting input area 804. Based on the recognition units identified from the handwriting input, the user device regenerates recognition results (eg, results 2124, 2126) in the candidate display area 806. 21G and 21H show that the user selects one of the recognition results (e.g., result 2124) (e.g., through contact 2128), and the text of the selected recognition result is input into the text input area 808 Shows that it is.

22A and 22B are flowcharts for an exemplary process 2200 in which individual recognition units identified in the current handwriting input may be visually presented and individually deleted, regardless of the time order in which the recognition units were formed. 21A-21H illustrate a process 2200 in accordance with some embodiments.

In an exemplary process 2200, the user device receives (2202) a handwriting input from a user. The handwriting input includes a plurality of handwritten strokes provided on a touch-sensitive surface coupled to the device. In some embodiments, the user device renders (2204) a plurality of handwritten strokes in a handwriting input area (eg, handwriting input area 804) of the handwriting input interface. In some embodiments, the user device divides a plurality of handwritten strokes into two or more recognition units (2206), each recognition unit comprising a separate subset of the plurality of handwritten strokes.

In some embodiments, the user device receives (2208) an edit request from the user. In some embodiments, the edit request is a contact 2210 that is detected over a predetermined affordance (eg, affordance 2112 in FIG. 21D) provided in the handwriting input interface. In some embodiments, the edit request is a tap gesture detected over a predetermined area in the handwriting input interface (2212). In some embodiments, the predetermined area is within the handwriting input area of the handwriting input interface. In some embodiments, the predetermined area is outside the handwriting input area of the handwriting input interface. In some embodiments, another predetermined gesture outside of the handwriting input area (e.g., cross gesture, horizontal swipe gesture, vertical swipe gesture, oblique swipe gesture) may be used as an edit request. . Gestures outside the handwriting input area can be easily distinguished from the handwriting stroke as it is provided outside the handwriting input area.

In some embodiments, in response to the editing request, the user device visually distinguishes 2214 of the two or more recognition units in the handwriting input area, eg, using boxes 2108 and 2110 in FIG. 21D. In some embodiments, visually distinguishing the two or more recognition units further includes emphasizing 2216 individual boundaries between the two or more recognition units in the handwriting input area. In various embodiments, different ways of visually distinguishing recognition units identified in the current handwriting input may be used.

In some embodiments, the user device provides a means for individually deleting each of the two or more recognition units from the handwriting input area (2218). In some embodiments, the means for individually deleting each of the two or more recognition units, e.g., as shown by delete buttons 2116, 2118 in FIG. It's a delete button. In some embodiments, the means for individually deleting each of the two or more recognition units is means for detecting a predetermined deletion gesture input for each recognition unit. In some embodiments, the user device does not visually display individual deletion affordances for the highlighted recognition units. Instead, in some embodiments, the user is allowed to delete an individual recognition unit under the delete gesture using the delete gesture. In some embodiments, because the user device is displaying the recognition units in a visually emphasized manner, the user device does not allow additional handwriting strokes in the handwriting input area. Instead, a predetermined gesture or any gesture detected on the visually emphasized recognition unit will cause the user device to remove the recognition unit from the handwriting input area and thus revise the recognition results displayed in the candidate display area. In some embodiments, the tap gesture causes the user device to visually highlight individual recognition units identified in the handwriting recognition area, and then the user can delete the individual recognition units one by one in the reverse writing direction using a delete button.

In some embodiments, for example, as shown in Fig. 21E, the user device is provided with a deletion input for individually deleting the first of the two or more recognition units from the handwriting input area, from the user and through the means provided. Receive (2224). For example, as shown in Fig. 21F, in response to the deletion input, the user device removes a separate subset of handwritten strokes in the first recognition unit from the handwriting input area (2226). In some embodiments, the first recognition unit is an initial recognition unit spatially among two or more recognition units. In some embodiments, for example, as shown in FIGS. 21E and 21F, the first recognition unit is a spatially intermediate recognition unit among two or more recognition units. In some embodiments, the first recognition unit is a spatially last recognition unit among two or more recognition units.

In some embodiments, the user device generates (2228) a segmentation grid from a plurality of handwritten strokes, the segmentation grid comprising a plurality of alternative segmentation chains each representing a separate set of recognition units identified from the plurality of handwritten strokes. do. For example, FIG. 21G shows recognition results 2024 and 2026, where the recognition result 2024 is generated from one split chain with two recognition units, and the recognition result 2026 is three recognition units. Are created from different splitting chains with s. In some embodiments, the user device receives (2230) two or more consecutive edit requests from the user. For example, two or more consecutive edit requests may be multiple consecutive taps on affordance 2112 of FIG. 21G. In some embodiments, in response to each of the two or more consecutive editing requests, the user device visually distinguishes 2232 a separate set of recognition units in the handwriting input area from another of the plurality of alternative split chains. . For example, in response to a first tap gesture, two recognition units (e.g., for each of the characters "帽" and "子") are highlighted in the handwriting input area 804, and in response to the second tap gesture , Three recognition units (eg, for the characters “巾”, “冒”, and “子” respectively) are highlighted. In some embodiments, in response to the third tap gesture, visual emphasis is selectively removed from all recognition units, and the handwriting input area returns to its normal state, ready to allow additional strokes. In some embodiments, the user device provides a means for individually deleting each of the respective set of recognition units currently displayed in the handwriting input area (2234). In some embodiments, this means is a separate delete button for each highlighted recognition unit. In some embodiments, this means is a means for detecting a predetermined deletion gesture on each highlighted recognition unit and invoking a function of deleting the highlighted recognition unit under the predetermined deletion gesture.

As described herein, in some embodiments, the user device provides a continuous input mode in the handwriting input area. Since the area of the handwriting input area is limited on the portable user device, it is desirable to provide a way to cache handwriting inputs provided by the user and allow the user to reuse screen space without committing previously provided handwriting inputs. Sometimes it is desirable. In some embodiments, the user device provides a scrolling handwriting input area, wherein the input area gradually shifts by a predetermined amount (e.g., one recognition unit at a time) as the user gets close enough to the end of the handwriting input area. . In some embodiments, since shifting the existing recognition units in the handwriting input area can interfere with the user's writing process and possibly impede the correct division of the recognition units, the input area is not dynamically shifted. It is sometimes advantageous to recycle previously used areas. In some embodiments, when a user reuses an area occupied by a handwriting input that has not yet been entered as a text input area, the highest recognition result for the handwriting input area is automatically input to the text input area, so that the user recognizes the top level. It allows you to continuously provide new handwriting input without explicitly selecting the result.

In some conventional systems, the user is allowed to write over existing handwriting input that still appears in the handwriting input area. In such systems, temporal information is used to determine whether a new stroke is part of an old recognition unit or part of a new recognition unit. Such temporal information dependent systems place stringent requirements on the tempo and speed at which the user provides handwriting input, which is difficult to meet by a large number of users. Additionally, the visual rendering of the handwriting input can become a confusing state that is difficult for the user to decipher. Thus, the writing process can be unpleasant and confusing to the user, resulting in an unpleasant user experience.

As described herein, a fading process is used to indicate when the user can reuse the area occupied by the previously written recognition unit and continue writing in the handwriting input area. In some embodiments, the fading process gradually reduces the visibility of each recognition unit provided in the handwriting input area for a threshold amount of time, so that when new strokes are written on it, the existing text does not visually compete with the new strokes. do. In some embodiments, writing over the fading recognition unit automatically causes the top recognition result for the recognition unit to be text input without requiring the user to stop writing and explicitly provide a selection input for the top recognition result. Make sure it is entered as an area. This implicit and automatic confirmation of the top-level recognition result improves the efficient input and speed of the handwriting input interface, and reduces the cognitive burden placed on the user to maintain the thought flow of the current text composition. In some embodiments, writing over the fading recognition unit does not cause automatic selection of the topmost search result. Instead, the faded recognition units are stored in the handwriting input stack and combined with the new handwriting input as the current handwriting input. The user can view the recognition results generated based on all the recognition units accumulated in the handwriting input stack before making a selection.

23A to 23J show, for example, that the recognition units provided in different areas of the handwriting input area gradually fade out from their individual areas after a predetermined amount of time, and after a fade-out occurs in a specific area, the user can write a new handwriting in that area. Shows example user interfaces and processes that are allowed to provide strokes.

As shown in FIG. 23A, the user has provided a plurality of handwritten strokes 2302 (eg, three handwritten strokes for uppercase “I”) in the handwriting input area 804. The handwritten strokes 2302 are identified by the user device as a recognition unit. In some embodiments, the handwriting input currently displayed in the handwriting input area 804 is stored in a first layer in the handwriting input stack of the user device. A number of recognition results generated based on the identified recognition unit are provided in the candidate display area 804.

23B shows that when the user continues to write one or more strokes 2302 to the right of the strokes 2304, the handwriting strokes 2302 in the first recognition unit gradually fade out in the handwriting input area 804. Shows the beginning. In some embodiments, an animation is displayed that mimics the gradual fading or disappearance of the visual rendering of the first recognition unit. For example, animation can create a visual effect of ink evaporating from a whiteboard. In some embodiments, the fading of the recognition unit is not uniform across the entire recognition unit. In some embodiments, the fading of the recognition unit increases over time, and eventually the recognition unit becomes completely invisible in the handwriting area. However, even if the recognition unit is no longer visible in the handwriting input area 804, in some embodiments, the invisible recognition unit remains at the top of the handwriting input stack, and the recognition results generated from the recognition unit are in the candidate display area. Displayed continuously. In some embodiments, the fading recognition unit is not completely removed from the field of view until a new handwriting input is written over it.

In some embodiments, the user device enables a new handwriting input to be provided over an area occupied by the fading recognition unit immediately at the beginning of the fading animation. In some embodiments, the user device is placed over the area occupied by the fading recognition unit only after the fading has progressed to a certain step (e.g., to a level of no guesswork or until no recognition is visible in that area). Allows new handwriting input to be provided.

23C shows that the first recognition unit (ie, strokes 2302) has completed its fading process (eg, the ink color has stabilized at a very weak level or has become invisible). The user device has identified additional recognition units (e.g., recognition units for handwritten characters “a” and “m”) from the additional handwritten strokes provided by the user, and recognizes the recognition results presented in the candidate display area 806. Updated.

22D-22F show that, over time, a user has provided a plurality of additional handwritten strokes (eg, 2304, 2306) in the handwriting input area 804. At the same time, previously identified recognition units gradually fade away from the handwriting input area 804. In some embodiments, starting its own fading process for each recognition unit after the recognition unit is identified takes a predetermined amount of time. In some embodiments, the fading process for each recognition unit does not start until the user starts inputting a second recognition unit thereafter. 23B-23F, when handwriting input is provided in a cursive font, a single stroke (eg, stroke 2304 or stroke 2306) is applied to a number of recognition units (eg, word “) in the handwriting input area. may be followed by a recognition unit for each handwritten character in "am" or "back").

22G shows that even after the recognition unit starts its fading process, the user has a predetermined recovery input, e.g., on the delete button 2310 (e.g., as indicated by an immediate lift following contact 2308). It shows that the tap gesture can bring it back to its non-fading state. When the recognition units recover, their appearance returns to the normal visibility level. In some embodiments, recovery of the faded recognition units is done character by character in the reverse writing direction in the handwriting input area 804. In some embodiments, recovery of the faded recognition units is done word by word in the handwriting input area 804. As shown in Fig. 23G, the recognition units corresponding to the word "back" have recovered from a fully faded state to an unfaded state. In some embodiments, the clock to start the fading process is reset per recognition unit when the recognition unit recovers to an unfading state.

22H shows that the continued contact on the delete button causes the last recognition unit in the default writing direction (e.g., the recognition unit for the letter “k” in the word “back”) to be deleted from the handwriting input area 804. Shows. As the deletion input continues, more recognition units (eg, recognition units for letters “c”, “a”, “b” in word “back”) are deleted one by one in the reverse writing direction. . In some embodiments, the deletion of the recognition unit is word-by-word, and all characters of the handwritten word "back" deleted from the handwriting input area 804 are simultaneously removed. 22H also shows that after deletion of the recognition unit for the letter “b” in the handwritten word “back”, as the contact 2308 remains on the delete button 2310, the previously faded recognition unit “m” is also Shows recovery.

Fig. 23I shows that if the deletion input is stopped before the deletion of the recovered recognition unit "m" in the handwritten word "am" occurs, the recovered recognition unit gradually fades again. In some embodiments, the state of each recognition unit (eg, a state selected from a set of one or more faded states and an unfaded state) is maintained and updated in the handwriting input stack.

23J shows, in some embodiments, when a user has provided one or more strokes 2312 over an area occupied by a fading recognition unit in the handwriting input area (eg, recognition unit for letter “I”), 23I and 23J, the text of the highest recognition result (e.g., result 2314) for the handwriting input made before the strokes 2312 is automatically entered into the text input area 808. . As shown in Fig. 23J, the text "I am" is no longer shown as tentative and has instead been committed in the text entry area 808. As shown in FIG. In some embodiments, once text input is made to a fully or partially faded handwriting input, the handwriting input is removed from the handwriting input stack. Newly input strokes (eg, strokes 2312) become the current input in the handwriting input stack.

As shown in Fig. 23J, the text "I am" is no longer shown as tentative and has instead been committed in the text entry area 808. As shown in FIG. In some embodiments, once text input is made to a fully or partially faded handwriting input, the handwriting input is removed from the handwriting input stack. Newly input strokes (eg, strokes 2312) become the current input in the handwriting input stack.

In some embodiments, when strokes 2312 are provided over an area occupied by a fading recognition unit (e.g., a recognition unit for letter "I") in the handwriting input area, the handwriting made before the strokes 2312 The text of the highest recognition result for input (eg, result 2314) is not automatically entered into the text input area 808. Instead, the current handwriting input (both faded and non-faded) in the handwriting input area 804 is cleared and stored in the handwriting input stack. New strokes 2312 are added to the stored handwriting input in the handwriting input stack. The user device determines recognition results based on the entire handwriting input currently accumulated in the handwriting input stack. The recognition results are displayed in the candidate display area. In other words, even if only a portion of the currently accumulated handwriting input appears in the handwriting input area 804, the recognition results are generated based on the entire handwriting input (both visible and invisible parts) stored in the handwriting input stack. do.

FIG. 23K shows that the user inputs more strokes 2316 that have faded over time in the handwriting input area 804. 23L shows that a new stroke 2318 written on the faded strokes 2312 and 2316 is input so that the text of the top recognition result 2320 for the faded strokes 2312 and 2316 is input into the text input area 808. Shows one thing.

In some embodiments, the user selectively provides handwriting input in multiple lines. In some embodiments, when multiline input is possible, the same fading process may be used to clear the handwriting input area for new handwriting input.

24A and 24B are flowcharts of an exemplary process 2400 for providing a fading process in a handwriting input area of a handwriting input interface. 23A-23K illustrate a process 2400 in accordance with some embodiments.

In some embodiments, the device receives (2402) a first handwriting input from a user. The first handwriting input includes a plurality of handwritten strokes, and the plurality of handwritten strokes form a plurality of recognition units distributed along individual writing directions associated with the handwriting input area of the handwriting input interface. In some embodiments, the user device renders (2404) each of the plurality of handwritten strokes in the handwriting input area as the handwritten stroke is provided by the user.

In some embodiments, the user device starts a separate fading process for each of the plurality of recognition units after the recognition unit is fully rendered (2406). In some embodiments, during the individual fading process, the rendering of the recognition unit in the first handwriting input fades away. This is illustrated in Figures 23A-23F in accordance with some embodiments.

In some embodiments, for example, as shown in Figs. 23I and 23J and Figs. 23K and 23L, the user device is able to remove from the user over an area occupied by the faded recognition unit of the plurality of recognition units among the handwriting input area. 2 A handwriting input is received (2408). In some embodiments, in response to receiving the second handwriting input (2410): the user device renders (2412) the second handwriting input in the handwriting input area, and clears all faded recognition units from the handwriting input area. (2414). In some embodiments, all recognition units input in the handwriting input area prior to the second handwriting input are cleared from the handwriting input area, regardless of whether the recognition unit has started its fading process. This is illustrated, for example, in FIGS. 23I and 23J and in FIGS. 23K and 23L.

In some embodiments, the user device generates 2416 one or more recognition results for the first handwriting input. In some embodiments, the user device displays 2418 one or more recognition results in a candidate display area of the handwriting input interface. In some embodiments, in response to receiving the second handwriting input, the user device automatically, without user selection, inputs the highest recognition result displayed in the candidate display area into the text input area of the handwriting input interface (2420). . This is illustrated, for example, in FIGS. 23I and 23J and in FIGS. 23K and 23L.

In some embodiments, the user device stores an input stack including the first handwriting input and the second handwriting input (2422). In some embodiments, the user device generates (2424) one or more multi-character recognition results each comprising a separate spatial sequence of characters recognized from a concatenation of a first handwriting input and a second handwriting input. In some embodiments, the user device displays one or more multi-character recognition results in a candidate display area of the handwriting input interface while the rendering of the second handwriting input in the handwriting input area replaces the rendering of the first handwriting input (2426). ).

In some embodiments, a separate fading process for each recognition unit begins when a predetermined period of time has elapsed after the recognition unit has been completed by the user.

In some embodiments, the fading process for each recognition unit begins when the user begins to input strokes for the next recognition unit after the recognition unit.

In some embodiments, the final state of the individual fading process for each recognition unit is a state with a predetermined minimum visibility for the recognition unit.

In some embodiments, the final state of the individual fading process for each recognition unit is a state with zero visibility for the recognition unit.

In some embodiments, after the last recognition unit in the first handwriting input has faded, the user device receives (2428) a predetermined recovery input from the user. In response to receiving the predetermined recovery input, the user device returns (2430) the last recognition unit from a faded state to an unfaded state. This is illustrated in Figs. 23F to 23H, for example. In some embodiments, the predetermined recovery input is an initial contact detected on a delete button provided in the handwriting input interface. In some embodiments, the sustained contact detected on the delete button deletes the last recognition unit from the handwriting input area and restores the last-to-second recognition unit from a faded state to an unfaded state. This is illustrated in Figs. 23G and 23H, for example.

As described herein, the multi-script handwriting recognition model performs stroke order independent and stroke direction independent recognition of handwritten characters. In some embodiments, the recognition model is trained only on space-derived features included in flat images of writing samples corresponding to different characters in the vocabulary of the handwriting recognition model. Since the images of the writing sample do not contain any temporal information related to individual strokes included in the images, the resulting recognition model is stroke order independent and stroke direction independent.

As illustrated above, stroke order and stroke direction independent handwriting recognition provides many advantages over conventional recognition systems that rely on information related to the temporal generation of characters (e.g., temporal sequences of strokes in characters). do. However, in real-time handwriting recognition scenarios, temporal information related to individual strokes is available, and it is sometimes beneficial to use this information to improve the recognition accuracy of the handwriting recognition system. The following describes a technique for integrating time-guided stroke-distribution information into spatial feature extraction of a handwriting recognition model, where the use of time-guided stroke-distribution information is the stroke order and/or stroke direction independence of the handwriting recognition system. Does not damage. Based on the stroke-distribution information related to different characters, clarification between characters that appear to be similar and created with distinctly different sets of strokes becomes possible.

"의 경우, 문자를 기입하기 위해 8개의 획들(예컨대, 도 27의 라벨링된 #1 내지 #8)이 사용될 수 있다. 문자에 대한 획들의 시퀀스 및 방향은 문자와 연관된 몇몇 고유한 특징부들을 제공한다. 인식 시스템의 획 순서 및 획 방향 독립성을 손상시키지 않고서 획 순서 및 획 방향 정보를 캡처하는 순진한 방식은, 훈련 샘플들에서 획 순서 및 획 방향의 모든 가능한 치환들을 명시적으로 열거하는 것이다. 그러나 오직 적당한 복잡성의 문자에 대해서도, 이것은 10억 초과의 가능성에 이를 것이고, 이는 불가능하지는 않지만 실제로 구현하는 것이 어렵다. 본 명세서에 설명되는 바와 같이, 각 기입 샘플에 대해 획-분포 프로파일이 생성되는데, 이는 획 생성의 연대순 양태들(즉, 시간 정보) 외에서 추출한다. 기입 샘플들의 획-분포 프로파일들은, 후속하여 공간-유도 특징부들과 조합되는 시간-유도 특징부들의 세트를 (예컨대, 입력 비트맵 이미지들로부터) 추출하도록 훈련되어, 필기 인식 시스템의 획 순서 및 획 방향 독립성에 영향을 주지 않고 인식 정확도를 개선시킨다.In some embodiments, when the handwriting input is converted to an input image (eg, input bitmap image) for a handwriting recognition model (eg, CNN), temporal information associated with individual strokes is lost. For example, Chinese characters "

In the case of ", eight strokes (eg, labeled #1 through #8 in Fig. 27) can be used to write the character. The sequence and direction of the strokes for the character provides some unique features associated with the character. A naive way of capturing stroke order and stroke direction information without compromising the stroke order and stroke direction independence of the recognition system is to explicitly list all possible permutations of stroke order and stroke direction in the training samples. Even for characters of only moderate complexity this will lead to a possibility of more than 1 billion, which is not impossible but difficult to implement in practice. As described herein, a stroke-distribution profile is created for each writing sample, which The stroke-distribution profiles of the write samples are extracted outside of the chronological aspects of the stroke generation (i.e., temporal information), the set of temporal-derived features that are subsequently combined with the spatial-derived features (e.g., the input bitmap image Are trained to extract from), improving recognition accuracy without affecting the stroke order and stroke direction independence of the handwriting recognition system.

As described herein, temporal information associated with a character is extracted by calculating various pixel distributions to characterize each handwritten stroke. Every handwritten stroke of a character creates a deterministic pattern (or profile) when projected onto a given direction. This pattern itself may be insufficient to clearly recognize the stroke, but when combined with other similar patterns, it may be adequate to capture certain characteristics unique to this particular stroke. Integrating this kind of stroke representation with spatial feature extraction (e.g., feature extraction based on input images in a CNN) is, after all, useful for clarifying between seemingly similar characters in the repertoire of the handwriting recognition model. It provides orthogonal information that can be done.

25A and 25B are flowcharts of an exemplary process 2500 for integrating temporal- and spatial-derived features of handwriting samples during training of the handwriting recognition model, where the resulting recognition model is stroke order and stroke. Keeps direction independent. In some embodiments, the example process 2500 is performed on a server device that provides the trained recognition model to a user device (eg, portable device 100 ). In some embodiments, the server device includes one or more processors and a memory containing instructions for performing process 2500 when executed by the one or more processors.

In an exemplary process 2500, the device separately trains 2502 a set of spatially-derived features and a set of temporal-derived features of the handwriting recognition model, where the set of spatially-derived features Trained on the corpus, each of the images is an image of a handwritten sample for an individual character in the output character set, a set of time-directed features is trained on the corpus of stroke-distribution profiles, and each stroke-distribution profile is output. Numerical characterization of the spatial distribution of a plurality of strokes in a handwritten sample for individual characters in a character set.

In some embodiments, separately training the set of spatially-derived features is an input layer, an output layer, and zeros between the first convolution layer, the last convolution layer, the first and last convolution layers. Further comprising training a convolutional neural network having a plurality of convolutional layers including the above intermediate convolutional layers, and a hidden layer between the last convolutional layer and the output layer (2504). An exemplary convolutional network 2602 is shown in FIG. 26. The exemplary convolutional network 2602 may be implemented in substantially the same manner as the convolutional network 602 shown in FIG. 6. The convolutional network 2602 comprises a plurality of convolutions including an input layer 2606, an output layer 2608, a first convolutional layer 2610a, zero or more intermediate convolutional layers, and a last convolutional layer 2610n. Layers, and a hidden layer 2614 between the last convolutional layer and the output layer 2608. The convolutional network 2602 also includes kernel layers 2616 and sub-sampling layers 2612 according to the arrangement shown in FIG. 6. Training of the convolutional network is based on images 2616 of writing samples in the training corpus 2604. Space-derived features are obtained by minimizing recognition errors for training samples in the training corpus and individual weights associated with different features are determined. The same features and weights once trained are used for recognition of new handwriting samples that do not exist in the training corpus.

In some embodiments, separately training a set of time-directed features provides a plurality of stroke-distribution profiles as a statistical model, such that a plurality of time-induced parameters and a plurality of time-directed parameters are used to distinguish individual characters in the output character set. Further comprising determining the individual weights for the time-derived parameters of (2506). In some embodiments, as shown in FIG. 26, a stroke-distribution profile 2620 is derived from each writing sample in the training corpus 2262. Training corpus 2262 optionally includes the same writing samples as corpus 2604, but also includes temporal information associated with stroke generation in each writing sample. Stroke-distribution profiles 2622 are provided to the statistical modeling process 2624, which is recognized or classified based on a statistical modeling method (e.g., CNN, K-Nearest Neighbor, etc.) during the statistical modeling process. By minimizing the error, time-induced features are extracted and individual weights for different features are determined. 26, a set of temporal derived features and individual weights are transformed into a set of feature vectors (e.g., feature vectors 2626 or feature vectors 2628), and a convolutional neural network 2602 ) Are injected into individual layers. Thus, the resulting network includes spatial- and time-derived parameters that are orthogonal to each other and contribute together to the recognition of characters.

In some embodiments, the device combines (2508) the set of space-derived features and the set of temporal-derived features in the handwriting recognition model. In some embodiments, combining the set of spatially-derived features and the set of temporal-derived features in the handwriting recognition model includes a plurality of spatial-derived parameters and a plurality of temporal-derived parameters in a convolutional neural network. And injecting into one of the convolutional layers of the convolutional layer or the hidden layer (2510). In some embodiments, the individual weights for the plurality of time-derived parameters and the plurality of time-derived parameters are the last convolutional layer of the convolutional neural network (e.g., the last convolutional layer of FIG. 26) for handwriting recognition. 2610n)). In some embodiments, the plurality of time-derived parameters and the individual weights for the plurality of time-derived parameters are added to the hidden layer of the convolutional neural network (e.g., hidden layer 2614 of FIG. 26) for handwriting recognition. Is injected.

In some embodiments, the device provides real-time handwriting recognition for the user's handwriting input using the handwriting recognition model (2512).

In some embodiments, the device generates (2514) a corpus of stroke-distribution profiles from the plurality of write samples. In some embodiments, each of the plurality of handwriting samples corresponds to a character in the output character set, and separately preserves individual spatial information for each constituent stroke of the handwriting sample when written (2516). In some embodiments, to create a corpus of stroke-distribution profiles, the device performs the following steps (2518):

For each of the plurality of handwriting samples (2520): the device identifies (2522) construction strokes in the handwriting sample; For each of the identified strokes of the handwriting sample, the device calculates a separate occupancy along each of the plurality of predetermined directions (2524), the occupancy being the ratio between the projection span of each stroke direction and the maximum projection span of the writing sample; For each of the identified strokes of the handwriting sample, the device also calculates an individual saturation ratio for each stroke based on the ratio between the individual number of pixels in each stroke and the total number of pixels in the writing sample (2526). Thereafter, the user device generates a feature vector for the handwriting sample as a stroke-distribution profile of the writing sample (2528), the feature vector comprising individual occupancy ratios and individual saturation ratios of at least N strokes in the handwriting sample. , N is a predetermined natural number. In some embodiments, N is less than the maximum number of strokes observed in any single write sample within the plurality of write samples.

In some embodiments, for each of the plurality of handwriting samples: the device sorts the individual occupancy of the identified strokes in each of the predetermined directions in descending order; Only the N highest occupancy and saturation ratios of the write sample are included in the feature vector of the write sample.

In some embodiments, the plurality of predetermined directions includes a horizontal direction, a vertical direction, a positive 45 degree direction, and a negative 45 degree direction of the writing sample.

In some embodiments, to provide real-time handwriting recognition for the user's handwriting input using the handwriting recognition model, the device receives the user's handwriting input; In response to receiving the user's handwriting input, it provides a handwriting recognition output to the user substantially simultaneously with receiving the handwriting input.

"를 사용하여, 예시적인 목적을 위해 본 명세서에 예시적인 실시예가 기술된다. 일부 실시예들에서, 필기 문자의 각 입력 이미지가 선택적으로 정사각형 내로 정규화된다. 각 개별 필기 획(예컨대, 획(#1, #2,..., #8))의 스팬이 정사각형의 수평, 수직, +45도 대각선, 및 -45도 대각선 상으로 투영될 때 측정된다. 각 획 Si의 스팬들은 4개의 투영 방향들에 대해, 각각 xspan(i), yspan(i), cspan(i), 및 dspan(i)로서 기록된다. 추가적으로, 전체 이미지에 걸쳐서 관찰되는 최대 스팬들이 또한 기록된다. 문자의 최대 스팬들은 4개의 투영 방향들에 대해, 각각 xspan, yspan, cspan, 및 dspan으로서 기록된다. 예시적인 목적을 위해, 여기서는 4개의 투영 방향들이 선택적으로 고려되지만, 원칙적으로 투영들의 어떤 임의의 세트가 다양한 실시예들에서 사용될 수 있다. 4개의 투명 방향들에서 문자 "

"의 획들 중 하나의 획(예컨대, 획(#4))의 최대 스팬들(예컨대, xspan, yspan, cspan, 및 dspan으로서 나타냄), 및 스팬들(예컨대, xspan(4), yspan(4), cspan(4), 및 dspan(4)로서 나타냄)이 도 27에 도시되어 있다.The letter "shown in Fig. 27

An exemplary embodiment is described herein for illustrative purposes using ". In some embodiments, each input image of a handwritten character is optionally normalized into a square. Each individual handwritten stroke (eg, stroke (#)) 1, # 2, ..., # 8)), horizontal, vertical, diagonal +45 degrees, -45 degrees and is measured when projected onto the diagonal. span of each segment Si are four projection direction of the square of the span For s, they are recorded as xspan ( i ), yspan ( i ), cspan ( i ), and dspan ( i ), respectively In addition, the maximum spans observed over the entire image are also recorded. For the four projection directions, each is recorded as xspan , yspan , cspan , and dspan For illustrative purposes, four projection directions are optionally considered, but in principle any arbitrary set of projections can be used in various embodiments. Can be used in the character "in 4 transparent directions

"Maximum spans (e.g., denoted as xspan, yspan , cspan , and dspan ) of one of the strokes (e.g., stroke #4), and spans (e.g., xspan ( 4 ), yspan ( 4 )) , cspan ( 4 ), and dspan (4 )) are shown in FIG. 27.

In some embodiments, once the spans have been measured for all strokes 1 through 5-where 5 is the number of individual handwritten strokes associated with the input image-the individual occupancy along each projection direction is calculated. For example, the individual occupancy rate R _x ( i ) along the x -direction for the stroke S _i is calculated as R _x ( i ) = xspan ( i ) /xspan. Similarly, individual occupancy along different projection directions can be calculated as follows: R _y ( i ) = yspan ( i )/ yspan , R _c ( i ) = cspan (i)/ cspan , R _d ( i ) = dspan ( i ) /dspan .

In some embodiments, the occupancy of all strokes in each direction are sorted separately in descending order, so that an individual ranking of all strokes in the input image is obtained for each projection direction in terms of their occupancy in that direction. The ranking of the strokes in each projection direction reflects the relative importance of each stroke along the associated projection direction. This relative importance is irrespective of the order and direction in which strokes were created in the write sample. Thus, this ranking based on occupancy rates is time-induced information independent of stroke order and stroke direction.

In some embodiments, each stroke is given a relative weight indicating the importance of the stroke to the entire character. In some embodiments, the weight is measured by the ratio of the number of pixels in each stroke to the total number of pixels in the character. This ratio is referred to as the saturation ratio associated with each stroke.

In some embodiments, based on the saturation ratio and occupancy rates of each stroke, a feature vector may be generated for each stroke. For each character, a set of feature vectors comprising 5S features is created. This set of features is referred to as the stroke-distribution profile of the character.

In some embodiments, only a predetermined number of topmost strokes are used when constructing the stroke-distribution profile of each character. In some embodiments, the predetermined number of strokes is 10. Based on the top 10 strokes, 50 stroke-derived features can be created for each character. In some embodiments, these features are implanted in the last convolutional layer of the convolutional neural network, or in a subsequent hidden layer.

In some embodiments, during real-time recognition, the input image of the recognition unit is provided in a handwriting recognition mode trained with both space- and time-induced features. The input image is processed through each layer of the handwriting recognition model shown in FIG. 26. When the processing of the input image reaches the layer where the stroke-distribution profile input is needed (eg, the last convolutional layer or hidden layer), the stroke-distribution profile of the recognition unit is injected into that layer. Processing of the input image and stroke-distribution profile continues until an output segment (eg, one or more candidate characters) is provided at the output layer 2608. In some embodiments, the stroke-distribution profiles of all recognition units are calculated and provided, along with the input images of the recognition units, as a handwriting recognition model as input. In some embodiments, the input image of the recognition unit initially experiences a handwriting recognition model (without the benefit of time-trained features). When two or more similar-looking candidate characters are identified with close recognition confidence values, the handwriting recognition model in the layer where the stroke-distribution profiles of the recognition unit are trained with time-derived features (e.g., the last convolutional layer or the hidden layer). Is injected into. When the input image and stroke-distribution profile of the recognition unit passes through the last layers of the handwriting recognition model, two or more similar-looking candidate characters can be better distinguished due to differences in their stroke-distribution profiles. . Accordingly, in order to improve the recognition accuracy without compromising the stroke order and stroke direction independence of the handwriting recognition system, time-induced information related to the manner in which the recognition unit is formed by individual handwritten strokes is used.

The foregoing description has been described with reference to specific embodiments for purposes of explanation. However, the exemplary discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in terms of the above teachings. The embodiments have been selected and described to best explain the principles of the present invention and their practical applications, thereby allowing those skilled in the art to best utilize the present invention and various embodiments having various modifications as suitable for the specific application contemplated. I can.

Claims (13)

As a method of providing handwriting recognition,
In an electronic device having one or more processors, a touch sensitive surface, and a display,
Displaying a user input interface including a message area and a stroke input area on the display, the stroke input area being distinct from the message area;
Receiving a first set of strokes on the touch-sensitive surface within the stroke input area;
Determining first text based on the first set of strokes;
Displaying the first text on the display in the message area;
Determining one or more candidates based on the first set of strokes;
Displaying the one or more candidates in a candidate display area;
Receiving a user input for selecting a first candidate from the one or more candidates while displaying the one or more candidates;
In response to the user input,
Replacing the display of the first text with the display of the selected first candidate; And
Clearing the one or more candidates from the candidate display area
Handwriting recognition providing method comprising a. The method of claim 1,
The method of providing handwriting recognition, wherein the first text includes a plurality of characters. The method of claim 1,
The first text is a single character, handwriting recognition providing method. The method of claim 1,
Wherein the one or more candidates comprise an emoji identified based on the first text. The method of claim 1,
After receiving the first set of strokes and displaying the first text, receiving a second set of strokes on the touch-sensitive surface in the stroke input area,
Wherein determining the one or more candidates is further based on the second set of strokes. The method of claim 5,
The method of providing handwriting recognition, wherein the second set of strokes is a single continuous stroke. The method of claim 1,
Wherein the first set of strokes is a single continuous stroke. The method of claim 1,
Rendering the first set of strokes within the stroke input area; And
Initiating a fading process for the first set of strokes,
During the fading process, rendering the first set of strokes within the stroke input area becomes increasingly fading. The method of claim 8,
Wherein the fading process for the first set of strokes begins when a predetermined period of time has elapsed since the first set of strokes has been completed by a user. The method of claim 8,
A method of providing handwriting recognition, wherein the final state of the fading process for the first set of strokes is a state with a predetermined minimum visibility for the first set of strokes. The method of claim 8,
A method of providing handwriting recognition, wherein the final state of the fading process for the first set of strokes is a state with zero visibility for the first set of strokes. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a device having a display, comprising:
The one or more programs include instructions for performing the method of providing handwriting recognition according to any one of claims 1 to 11. As an electronic device,
display;
One or more processors; And
And a memory storing one or more programs configured to be executed by the one or more processors,
The electronic device, wherein the one or more programs include instructions for performing the method of providing handwriting recognition according to any one of claims 1 to 11.

Images