KR101488537B1 - system and method for a user interface for text editing and menu selection - Google Patents

Abstract

A method and system for allowing a user of an input action recognition text input system to modify erroneously recognized text without moving a text insertion position to a text position to be modified. The system also ensures that whitespace between text objects is handled correctly when text objects are replaced by objects whose spaces are automatically generated in different ways. The system can also display the menu items in a graphical way by selecting the desired item quickly and easily by performing a selection gesture with a lower accuracy, instead of directly touching the sub-areas of the menu associated with the desired item.

Text input system, text display system, text object

Description

[0001] The present invention relates to a user interface system for text editing and menu selection,

The present invention relates to graphics and gesture user interfaces for computer systems. More specifically, a user performs an input operation using one or more input devices and, in order to recognize the data generated by the input operation as one or more input operations corresponding to text data input to the computer system, Analyzing involves various types of computer systems. The present invention also allows the user to select two or more menu selections with a simple gesture that can be performed quickly and easily by presenting two or more menu selection items to the user in a graphical manner so that they can be used as part of a text input method, It contains the information to use.

A variety of text input systems have been developed that are designed to allow users to enter text more quickly. Often, these fast text inputs are implemented in a way that allows users to perform less accurate actions compared to a regular system, since it is usually faster to perform certain actions less accurately. Typically, you do this by defining a glossary of text objects that the system can create and mapping your input behavior to one or more text objects in the glossary. The system then analyzes the input action (one or more) that the user performs, and checks the text object that is most closely mapped to the intended object of the user's input action and generates the verified object as text corresponding to the input action . An example is a system disclosed by Robinson et al. In US Pat. No. 6,801,190 with a virtual keyboard that allows the user to enter text with an inaccurate tapping motion. Another example is the "ShapeWriter" system (US Patent No. US 2004-0120583 A1 discloses Zhai) that detects the shape of the trace path on a virtual keyboard and recognizes the shape pattern library associated with the word and recognizes the corresponding word pattern You can. A system disclosed by Kushler et al. In U.S. Patent No. 7,098,896 is another example.

Instead of performing actions in the same form, but with less accuracy (as in the case of a virtual keyboard published by Robinson et al.), Other systems (e.g., speech recognition and handwriting recognition systems) , Speech in the case of a speech recognition system, or handwriting in a handwriting recognition system). In this "alternate form" system, the user's input behavior (either words for speech recognition or handwritten text for handwriting recognition) is mapped ("recognized") to one or more text objects in the system's glossary. Other systems, by default, introduce ambiguity into your text input behavior, so you must map your input behavior to one or more text objects in the glossary. A well known example is the "T9" (US Patent No. 5,818,437, published by Grover et al.) Input system, which is commonly used in cell phone keypads with a limited number of keys and commercially available. In this system, a subset of each key on the standard telephone keypad contains each character set, and when the user types a key, one or more words consisting of the characters corresponding to the keys in the order entered are placed in the user's key- It is mapped.

Although there are differences in the nature of the user's input behavior in these various systems (hereinafter referred to as "input motion recognition system") and the manner in which such input operations are mapped to textual entities in the system glossary, input motion recognition systems generally have many common It has features.

1) The text object recognized by the input motion recognition system is output to the text display system (most of which is implemented as a part of the display device in which the text is displayed) so that the user can further edit the generated text.

2) Sometimes a certain input action (or sequential input action) is found to correspond to more than one possible text object, and a text object that is identified as being most closely mapped to the input action is output to the text display system.

3) The system maintains a record of one or more alternate text objects that have been found to be reasonably responsive enough to the input behavior (at least only some recently created text objects) (Temporarily). Alternatively or in lieu of it, the system may record specific data or information associated with the input operation and associate this data with the associated alternate text interpretation record, or may process this record data associated with the input operation later to verify the alternate text object can.

4) It has been confirmed that the input text matches most closely with the input operation, so that the text object output through the text display system may not be the text object intended by the user. In this case, Must be edited for text

5) Text display system has a text insertion point (or location) to insert the next text object that you create. This is collectively referred to as the "cursor" (sometimes referred to as the "caret") location (hereafter referred to as the "text insertion point").

6) There is a text editing action (or "gesture") that the user uses to change the text insertion point to a new position in the text display system. On most systems, the text display system has a text output portion on the display. For example, for mouse-based systems, this behavior is probably a single mouse click in the text output area, with no exceptions. For stylus-based touch screen systems, this behavior corresponds to a single tapping of the stylus in the text output area, with no exceptions.

7) Characters are generally categorized into two types in the system. One is a character that can be used to construct one or more text objects created by the system (hereafter referred to as "text object characters"), and the other is one or more characters that are treated as delimiters that are not included in the system- It is a character or typeface (hereafter called the "delimiter"). Delimiters generally have a "white space" (space, tab, return, etc.) and other punctuation characters.

A common feature among many users in these various input motion recognition systems is that the user performs the operation more quickly until the "error" begins to occur at a certain frequency (consequently, the accuracy continues to drop). An error here refers to the case where the text object output by the system to the text display system is not the text object intended by the user. Another feature common to almost all text editing systems is that whenever a user performs an edit operation (assuming that all text displayed in the text display system is modified by user action), the text insertion point is redirected to the edited text position It is. While these features were reasonable (and in some ways inevitable) in many of the basic models of using text and mouse for keyboard editing and the input motion recognition systems described above, these features are no longer desirable. In this context, "text editing", users typically want to enter a specific text flow. At some point in the input process, the user recognizes that the generated text differs from the intended text after looking at the text output portion (or after examining the text display system). This is because one or more of the actions the user has entered is "misrecognized" by the system and one or more text objects other than what the user intended are displayed at the beginning of the text output area. In most cases, the user will need to modify the erroneously recognized text (basically "remapping" or "re-editing" the system's interpretation of the original input action performed by the user when the text object is created) You want to continue entering new text at the text insertion point. However, in existing systems, the problem is that you can not edit misrecognized text without redirecting the text insertion point to the edit text position.

Standard desktop computing systems use a full-size desktop keyboard and mouse (or a mouse substitute such as a trackball or graphics tablet digitizer) with almost no exceptions. As a result, most users use keyboards and mice as convenient and relatively effective means of text entry on desktop systems. For handheld portable devices, desktop keyboards and mice are not practical because of their size, as well as the requirement for a flat, relatively stable surface (for standard mice). Therefore, many of the above-mentioned input motion recognition text input systems have been developed for special purposes for use in carrying portable devices. Or when used with such a device.

Portable computing devices continue to improve performance and usability. The touch screen has proven to be a very useful, flexible, and easy-to-use interface for mobile devices. The touchscreen interface is used in a wide variety of devices, including large-scale devices such as tablet PCs, but is particularly useful for small devices such as PDAs and cell phones. There are two conflicting goals in this device development process: one is to make the device smaller, and the other is to make the device easier to use, faster and more convenient.

For example, one user interface that is used extensively in various systems is to present various menu items and allow the user to select the desired menu. This user interface approach is often used in the input motion recognition text input system described above, which presents the user with a list of possible alternative text interpretations for one or more input actions and allows the user to select the correct interpretation Because it can. The most natural way to use onscreen menus on a system based on the touchscreen interface is to select the desired option by simply touching the desired menu item with the stylus or finger. It is desirable to minimize the space required for menu display so that it does not interfere with other elements of the display. On the other hand, when the user selects the desired menu option by touch, the smaller the menu size, the more difficult the menu is to use because the user must increase the accuracy of the selection operation.

Therefore, there is an adverse effect that naturally reduces usability. Another similar consideration is in favor of designs that allow the touchscreen system to be used with fingers without the use of special tools to interact with the stylus or screen. Since the fingers are less accurate than the stylus, designing the menu so that the user can select the menu with the user's finger usually causes a problem that the display menu becomes considerably large.

As mentioned above, the previous method requires you to directly touch the item you want to select from the touch screen menu. In some cases, the first time you touch the screen with the stylus (or finger), the user needs to be proficient with the stylus so that these touches are within the area specified in the selection. In other cases, the user may first touch the screen within a specific active selection area, then turn off the stylus (without lifting the stylus or finger off the screen) until it enters the area specified in the desired selection There is. The first approach requires precise control of the first touch location to select the desired item, and the second approach requires precise control of the final touch location. Since each menu choice is displayed as a 2D menu area, in both cases, the user must control the screen touch in two dimensions in order to select the desired menu. The smaller the area of each 2D menu area, the higher the accuracy of the user required for the touch operation. The higher the accuracy, the longer the time required for menu selection.

The method and system of the present invention solves the problems described above for the input motion recognition text entry system by allowing the user to edit the misrecognized text without redirecting the text insertion position to the edit text position. The feature of the present invention is that the text editing system of the present invention tracks and records the point of the text insertion position within the text output area while the user generates text to be inserted into the text output area, Edit ", you can automatically return the text insertion point to this position.

Another aspect of the invention is that the system defines user actions (or "gestures") for text editing that are distinct from those used when changing the text insertion location to the new location of the text output area. That is, this distinct text editing user action changes the user's previously generated (but misrecognized) text object to one of the alternate text objects associated with the output text (hereinafter referred to as the "re-edit" gesture) Indicates that you want to

 For example, in a stylus-based touchscreen system that taps the stylus once to change the insertion position of the text to a new position in the text output area, the distinctive re-editing gesture is to place a stylus near the previously- It can be defined as a "double-tapping action" (two critical strokes within a specified threshold maximum time between two successive tapping times). If the system detects that the user has performed a reediting gesture on an area associated with a previously printed text object, the text output area may contain one or more inputs that cause the text object to create an initial text object (a base selected object or a user selected object) The object text object associated with the action is replaced by the selection object, and the text insertion point automatically reverts to the previous position (the position before the reediting gesture is detected).

When a user tries to select an alternate text object that replaces the original text object, the most effective user interface is to show the user a menu of alternative text objects and to select the desired text object. The present invention increases the speed and convenience of such selection by allowing the user to select a menu item without having to start and end a screen contact operation within a particular two-dimensional menu area when selecting the desired menu item. That is, the menu item activated by the user's screen touch operation is determined by the segment at the outermost boundary of the menu that is traversed according to the touch operation, and each segment of the menu boundary is associated with one menu item. In this approach, initial touch is possible anywhere in the displayed menu and does not need to start (or end) within the area associated with (or appear to be associated with) the specific text object desired. Whenever you need to make a choice between two or more possible choices, the menu selection system of the present invention can be used to further increase efficiency.

BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the present invention will be described in detail below with reference to the following drawings:

Figure 1 is a structural system block diagram illustrating the general hardware components of a system implementing the text editing and menu selection system of the present invention as shown in Figures 2a, 2b and 2c and Figures 3a, 3b and 3c.

Figure 2a shows an example of a text object being re-edited according to the concepts of the method of the present invention.

Figure 2b shows that one of the two cascading text objects is being re-edited according to the concepts of the present invention.

FIG. 2C shows an example of the result of re-editing one of the two cascading text objects of FIG. 2B according to the concept of the present invention.

Figure 3A is an example of a menu constructed utilizing the concepts of the present invention.

Figure 3b is an example of a menu constructed utilizing the concepts of the present invention. Two of the choices presented here have a priori probability of being less likely to be the user's intended choice.

Figure 3c is an example of a menu configured in one way that exploits the concepts of the present invention. It shows the border segments of the menu associated with each presented choice using the corresponding color, shading, or texture in the background of the proposed selection and its associated border segment.

Figure 4 illustrates an example of a touch operation for a menu configured in one fashion utilizing the concepts of the present invention.

Figure 5 is an example of a menu constructed in one fashion that exploits the concepts of the present invention, which can increase the number of choices presented.

Fig. 6 shows an example of a structured menu in the form shown in Fig. 5, which shows a set of menu items that can be selected with a similar selection gesture using a similar background color.

Through the system to which the present invention is applied, the user can correct the erroneously recognized text by the input motion recognition system without unnecessarily stopping the process of inputting and editing the text. If the intended word or phrase is misrecognized (one or more input actions are "mapped" to a different text than intended by the user in performing the input action), the user must correct the misidentification of the system, Is to "remap" or "re-edit" the system's interpretation of the user's initial input behavior at the time the text object was created rather than intended to change or edit the text. If the system finds that one or more previous input actions are misrecognized, then in the system in which the invention is implemented, the user can correct the erroneously recognized text and then continue the text entry at the original point of the text insertion location with minimal effort There is. This is because the text insertion point automatically returns to its original position.

A feature of the present invention is that when the user creates text to be inserted into the text output area, the system tracks and records the text insertion position within the text output area, so that the text insertion position can be automatically returned to this position It is. In another aspect, the system defines user actions (or "gestures") for text editing that are distinct from those used to change the text insertion location to the new location in the text output area. That is, this distinct text editing user action changes the user's previously generated (but misrecognized) text object to one of the alternate text objects associated with the output text (hereinafter referred to as the "re-edit" gesture) Indicates that you want to. For example, in a stylus-based touchscreen system that taps the stylus once to change the insertion position of the text to a new position in the text output area, the distinctive re-editing gesture is to place a stylus near the previously- It can be defined as a double-tapping action. If the system recognizes that the user has performed a reediting gesture on the area associated with the previously printed text object, the text object in the text output area is replaced with one of the associated alternate text objects (described below) The position before the re-edit gesture is detected).

In the following sections, which illustrate various aspects and implementations of the present invention, certain aspects or implementations are generally illustrated and described with the assumption of a stylus touchscreen system. These various features and implementations, however, are not limited to systems that perform input operations using standard mice and keyboards, touch screen systems that do not require a stylus, alternative type mice (e.g., A "gaze" system that allows the user to perform various mouse clicks by checking the position of the screen being watched, automatically moving the mouse cursor of the system to this position, monitoring the blinking of the eye or other action mechanism) , Or a system that allows a user to perform an input action that generates text and / or performs editing operations on already entered text using other methods.

In another aspect, the system allows the user to highlight a word to be edited (or place the text insertion location near or within the bounds of the word) and then execute the specified "re-edit" edit function key, (In this case, the system will track the previous two locations of the text insertion location before re-assigning the text to re-edit, and then return to that position after re-editing). In another aspect, the system recognizes when a reediting gesture is performed in the output text area so as to modify this recognition error as effectively as possible, and determines whether the reediting target word is the closest to the location where the predetermined stylus action or gesture was performed Check the words in the output text area. In one implementation discussed in the previous example, the predetermined editing action corresponds to the "double tapping" of a word displayed on the touch screen with a stylus. In another implementation (for example, when operating a system using a cursor movement control device such as a mouse), a predetermined editing action may be performed by (for example) placing the mouse over the word to be edited (This is just one example, as these gestures can be defined in various ways).

For another implementation, a re-edit gesture is defined to allow you to select multiple words with a single gesture. For example, in speech recognition systems, recognition errors usually affect one or more words that are generated and output. For example, when a user says "It is difficult to recognize speech", the system can generate output as "difficult to sample food". In this case, the reediting gesture can be defined, for example, by drawing a line through the listed words. That is, in the above case, you draw a line in the phrase "tasting the food." The system will then generate one or more alternative interpretations for the relevant phrases without changing the interpretation of the phrase portion that is "difficult". This approach can also be applied to handwriting recognition systems. For handwriting recognition systems, recognition errors often occur where a user's handwritten segment entered by one word is split into more than one word. Of course, other re-editing gestures may also be defined, such as drawing a circle around a series of words to be re-edited, and these are also considered to be within the scope of the present invention. For example, a reediting "gesture" in a speech recognition system can be defined as a verbal command. In other words, it can be a verbal phrase like "re-editing: tasting food".

In one aspect, the system stores a list of alternate text objects that are identified as being closest to one or more input actions for one or more recent output text objects. If a re-edit gesture is detected, the system displays a list of alternate text objects that the system has checked in the closest matching text object in relation to the original input behavior of the user that caused this text object to be re-edited . In another implementation, the originally printed text object is omitted from the list of displayed text object selections, because the re-editing procedure is (usually) done only to replace it. As a further feature, the text object to be re-edited is automatically replaced with the next closest matching alternate text object in the list of alternate text objects found to be closest to the input action that caused the output of the re-edited text object Without requiring the user to select a text object from the displayed list of selected text objects). This approach can be most effective in systems where in most cases the intended text object is the next closest alternative text object. In another aspect, if the previously re-edited text object is re-edited to match the next closest alternate text object to the alternate text object that automatically replaced the originally output text object, Displays a selection list of text objects for you to select.

Figure 1 shows a simplified block diagram of the hardware components of a generic device 100 implemented with a user interface system and method for text editing and menu selection. The device 100 includes one or more input devices 120 that provide inputs to the CPU (processor) 110 to inform the user of the actions performed by the user. This input is typically controlled by a hardware controller that interprets the raw signal received from the input device and then communicates this information to the CPU 110 using a known data port and a known communication protocol. One example of an input device 120 is a touch screen that provides an input to the CPU 110 that notifies the touch event when the user touches the touch screen. Similarly, the CPU 110 communicates with the hardware controller for the display 130 that appears on the display 130. One example of display 130 is a touch screen display that provides the user with visual feedback in both graphical and textual form. Optionally, the speaker 140 may be connected to the processor to provide an auxiliary audible signal to the user (primarily used for error signals), or to connect the microphone 141 to the processor to receive a verbal input from the user This is mainly used in an input motion recognition system that implements speech recognition by a text input method. The processor 110 may include one or more of temporary and / or permanent storage, read only and writeable memory (random access memory or RAM), read only memory (ROM), writable nonvolatile memory such as FLASH memory, hard drive, floppy disk, You can access memory 150 that can be combined more than. Memory 150 includes program memory 160, which includes all programs and software, such as operating system 161, input motion recognition system software 162, and all other applications 163. The program memory 160 also includes at least one text editing system software 164 for recording and storing text insertion locations in accordance with an implementation of the present invention, This includes the menu selection system software 165 to determine which of these graphical display items the user has selected. The memory 150 also includes a data memory 170, which includes a text object database 171 required by the input motion recognition system software 162, an optional repository for managing records of user options and preferences 172, and other data 173 required for all elements of the device 100.

2A, a reediting function key 208 displayed on the display 130 of the system 100 may be used, or a predetermined reediting gesture may be performed on a previously output text object ("great" 200 in FIG. 2A) Demonstrates how to modify the output text object that does not correspond to an object to perform the re-editing procedure. In another implementation, if the re-edit function key 208 is activated and there is no currently highlighted text object, the system checks the text object that contains or is adjacent to the current text insertion location and automatically selects I will.

FIG. 2A shows the resulting text object selection list 202. In this example, the originally intended text object "heat" is identified as the second closest matching text object (following the word "great" first printed out as the text object corresponding to the first input action) Because the text object is selected in list 202, the first text object appears as 204. When the user selects the text object 204 in the text object selection list 202 using the input device 120, the processor 110 automatically substitutes the text object "great" 200 highlighted in the text output area 206 of the display with the original intended text object & I will.

In another aspect, the re-edit process inserts or deletes blanks according to how the blanks are automatically generated by the input motion recognition system software 162. When an input motion recognition system automatically creates a space between text objects, if spaces are automatically generated between every pair of adjacent text objects, it is often the case that you want to delete unwanted but automatically generated spaces. For example, in English, you often need to add a text object "'s" (apostrophe -s) after a noun to make it a possessive of a noun. In French, if the articles "le" and "la" precede a word beginning with a vowel, then this article is written as "l '" (l-apostrophe) without any spaces between the apostrophe and the first letter of the next word. In other languages, such as German, you often link two or more words together to form long "composite" words. Although other methods of solving this problem are possible and these methods are not considered to be beyond the scope of the present invention, a simple way to solve this problem in an input motion recognition system is to use a general rule The exception to this is to flag certain text objects. For example, in English, the text object "'s" (apostrophe -s) indicates that a space is created between itself and the next text object in the system's text object database, but not between itself and the previous text object Will be processed. By contrast, in the case of French, the text object "l '" (l-apostrophe) indicates that a space is created between itself and the previous text object in the system's text object database, but not between itself and the next text object Will be processed. Another feature is that, in order to handle languages such as German, where many text objects are often concatenated together to form long "compound" text objects that are normally separated by spaces, The input motion recognition system includes features that can later be used to suppress the automatic generation of blanks so that text objects are created consecutively. In other words, this feature suppresses the automatic creation of whitespace that is normally generated until you enable automatic generation of blanks again. In addition, the Input Motion Recognition system includes the ability to remove recently generated automatic blanks. As described below, the re-editing process of the present invention accommodates these various exceptions to the automatic generation of blank, so that when the text object is replaced with a text object to which various rules relating to automatic blank processing are applied during the re- Whitespace in the text is handled appropriately.

As a further feature, re-editing gestures are defined to allow selection of a single gesture as a substring of a sequence of text object strings. In the example shown in FIG. 2B, the word " of "210 consists of the words" of "212 and" In this example, it should be noted that these two text objects were created as a result of two separate input operations performed in succession. In addition, this text was generated by a user input motion recognition system which basically outputs a blank space between consecutively generated text objects, wherein the text object "s" 214 is an exception to this basic operation in the system's text object database Because it is flagged, the "" s "214 output does not generate a space before, and it continues to the end of the previous text object, so the correct ownership is created. However, in the example shown in FIG. 2B, the text object "'s" 214 was not the text object that the user intended when the corresponding input action was performed.

As another feature, a re-edit gesture in an area associated with a text string (of the text column "of's" 210 in the example of FIG. 2B) that consists of multiple text object characters and is delimited by one or more delimiters before and after If detected, the system first determines if a full-text string ("of's" 210 in the example of FIG. 2B) was generated by a single user input operation, and if so, Reacts in the manner described above. In another aspect, if a verified text string ("of's" 210 in this example) is generated by more than one individual user input operation and a configuration substring text object is generated from each of these actions, Identify the configuration substring text object that most closely resembles the gesture, for example, highlight this configuration substring text object, and construct a list of alternate text objects associated with the user input behavior that caused the configuration substring text object to be created. Is displayed as a selectable item. In FIG. 2B, a position 216 associated with the detected re-editing gesture (e.g., double-tapping action at location 216) is identified as a reediting "target " Closest to. In the example of FIG. 2B where the user just entered the complete sentence, the user is ready to enter the next sentence because the text insertion position is located at the end of the sentence just entered at position 218 before detecting the reediting gesture at position 216 .

In this example, the originally intended text object "Oz" is associated with the first input action (following text object "'s" 214 first printed with the text object corresponding to the first input action) , It appears as the first text object 220 in the text object selection list 202. 2c selects the text object "Oz" 220 in the text object selection list 202 to automatically convert the text object "'s" 214 highlighted in the output text area 206 to the original intended text object "Oz & Shows the results of the replacement. Notice also that a space 222 was inserted between the text objects "of" 212 and "Oz" 220 preceding the text editing system software 164. As previously mentioned, the text object "'s" 214 is flagged as an exception to the basic operation that automatically generates a space between consecutively generated text objects in the system's text object database 171, No space was created in the beginning before printing 214. The selected alternate text object "Oz" 220 is not flagged with this exception in the text object database 171 of the system, so when the system replaces the text object "'s" 214 with the text object "Oz" 220, Is created and a space of 222 appears between the words "of" 212 and "Oz" 220. Similarly, if the situation was reversed and the text object "Oz" 220 was first created and then replaced with the text object "'s" 214 in the re-editing procedure, then a blank space 222 preceding the text object & If "Oz" 220 has been replaced with "'s" 214, this leading space will be deleted and "' s" 214 would have been directly connected to the previous text object. Thus, as a further feature, if one text object is replaced by another text object as a result of the re-editing procedure and different automatic blanking operations are applied to the two text objects, then the difference in white- The white space surrounding the alternate text object is adjusted to compensate.

In another implementation, the text editing system software 164 tracks the text insertion location in the text output area 206, so that immediately after the reediting text object is replaced with the replacement text object, the text insertion location is moved to the previous output text position To be automatically returned to. In the example of FIG. 2C, the text object "s" 214 is replaced with the text object "Oz" 220 and the text insertion position is automatically changed from the original position 218 of the completed sentence (position 216 to the position before the reediting gesture is detected) The user can continue entering text without manually moving the text insertion location. Another feature is that if the selected text object selection list is canceled without selecting an alternate text object, the identified text object is still highlighted (selected) and the text insertion position is not returned to the previous position of the output text, You can perform other tasks related to the text object. In another aspect, when performing one or more additional input operations before selecting an alternative text object in the generated text object selection list, the text object selection list is automatically canceled and the text insertion position is returned to the previous position of the output text And the text generated by one or more additional input operations is sent to the text output area and, due to the fact that the previously outputted text object is currently highlighted (selected) text area according to the normal operation of the word processing program, Replaces text objects. In another aspect, control characters generated by input actions corresponding to the generation of control characters are sent to the target text application. For example, if you perform an input action that creates Control-B, the target application receives Control-B and applies boldface formatting to the text object that was previously highlighted and printed. In another aspect, when the system detects that the text area displayed by the user is scrolled so that the text insertion position is no longer displayed on the display screen when performing the reediting gesture, even if the system selects a substitute text object in the automatically generated text object selection list, The insertion position does not return to the previous (no longer visible) position of the output text. In another aspect, the user may vary the system's response to predetermined re-editing gestures and subsequent actions (e.g., returning the text insertion position to the previous position of the output text) by specifying a set of system preferences Select.

This aspect of the system greatly improves the efficiency of text input. The input motion recognition system basically prints a number of erroneously generated text objects, and these erroneously generated text objects are typically randomly distributed throughout the previously generated text. In most cases, the user wants to edit only the misplaced text objects and continue typing the text where he stopped reviewing the previously printed text. Tracking and automatically returning text to the system is a great advantage because it means that the user will not have to move the text insertion point back to its original position. By handling the error recovery process very effectively, the following two characteristics significantly enhance the overall efficiency of the system. First, it's easy to fix accidental errors, and it's easy to see that overall input speed will improve effectively because it takes less time to fix. Second, the error-recovery process is simple and straightforward, so users will not have to worry about the possibility of recognition errors, so they can quickly enter text with less accuracy. In many input motion recognition systems, this aspect increases the overall input speed.

In another aspect, the text editing system software 164 detects when the user redirects the text insertion cursor within the text output area and modifies automated system behavior to suit the new text insertion location background. One feature is that when a system that automatically prints blanks between words that are created normally senses that the text insertion point has moved to the new environment, the system will try to print the blanks automatically before the first word is printed in the new environment Will be released. In another aspect, if the system has moved the text insertion location to a new environment and such an automatic white space operation has been activated, the system checks the character to the left of the new text insertion position and determines that the character to the left of the text insertion position is "white space ) ", Or if the text insertion position is the first character position of the text field and / or if the text field is the password input field, the system automatically releases the automatic blank output before printing the first word in the new environment.

In another aspect, when a user re-edits a text object, one or more alternate text object lists are presented as a selection of text objects intended by the user. These alternate text objects correspond to alternate "mapping" of one or more input actions that the system originally mapped to the re-edit text object in the input motion recognition system. In another aspect, the alternative text object list is presented in a special graphical representation that can be selected by the user, and the graphical representation allows the user to select a desired object You can choose what you want with a simple, intuitive gesture that lets you quickly select and quickly process the reediting process. In another feature, this special graphical menu representation is also used elsewhere in the user interface to quickly process the selection of two or more alternate choices from different locations in the system. The graphical display and gesture selection method of the present invention is particularly effective when one or more of the possible alternative selection items presented is higher than the average priori probability, which is the intended item of the user. This is true in many situations where the user is presented with a choice.

Figure 3A is an example of a menu 300 constructed utilizing the concepts of the present invention. In the example shown in FIG. 3A, there are six sub-selection regions: five sub-selection regions 301 to 305 corresponding to the selections 1 to 5 and a sixth sub-region 301 to 305 of the icon label indicating the operation for canceling the menu 300 There is 306. Each of these six sub-selections 301 - 306 is associated with its corresponding segment 311 - 316 at the perimeter boundary of the menu 300. In the present invention, the screen is touched at any position in the area surrounding the periphery of the periphery of the menu and maintained in contact with the screen until a route is reached beyond the menu area to the menu periphery segment associated with the desired selection, Select the menu by releasing contact with the screen (such as lifting the stylus) at a position outside the perimeter of the 300 and ending the touch operation. Otherwise, the menu selection is made as soon as the touch operation leaves the menu area without having to end the touch operation. If a menu is selected because the contact action must be terminated, the user can re-enter the menu area without releasing the contact and then change the menu selection that has not been completed by exiting through the other segments of the menu perimeter before exiting the contact action. Here, the touch screen is taken as an example, so that it is possible to "activate" and "deactivate" the control point, which is determined by the position of the stylus (or finger or other contact means) in contact with and releasing contact with the screen. It is not considered to be out of range. For example, a control point can be a cursor that controls the movement of the mouse, in which "activation" and "deactivation" are performed for each click and release of the mouse button.

3A, the perimeter border segment of the menu 300 associated with the menu selection subarea is that segment of the perimeter of the menu that pertains to the perimeter boundary of the menu selection subarea itself. The example menu shown in FIG. 3A relates to the menu periphery segment 311 that constitutes the entire upper border of the menu 300, and the selection 1 of the lower selection area 301 touches any screen position in the menu 300 to select it, Of the menu area and then release the contact with the screen. 4 shows an example of a contact action 400 starting at an initial contact location 401 and exiting the menu at an exit position 402 of the peripheral border segment 311 to terminate the contact at the final contact location 403. FIG. As a result of the contact action 400, the initial contact position is within the menu selection sub-area 303 associated with "selection 3", but menu selection 301 ("selection 1") is selected.

One advantage of the present invention is that users can select menus more quickly because they can perform contact actions much less accurately and thus perform much faster. This advantage can be even greater if there is a higher likelihood of expecting to choose a particular menu choice than another. For example, in the case of the menu shown in FIG. 3A, it can be assumed that the menu choices 301 ("select 1") and 305 ("select 5") are selected twice more often than the choices 302, 303, . The menu illustrated in FIG. 3A is such that the entire upper border segment 311 of the menu perimeter boundary is associated with the lower selection region 301 of "Choice 1" and the entire lower border segment 315 of the menu periphery is associated with the lower selection region of "Choice 5" It is designed. In this example menu, segments 311 and 315 are almost three times longer than segments 312, 313, and 314 (respectively associated with sub-select regions 302, 303, and 304, respectively) It is fairly easy to select "Select 1" and "Select 5" because they have a wide margin when moving along. It is a simple matter to design the various menu selection subareas such that the relative length of the associated perimeter boundary segments is close to the relative expected probability of the various menu selections within the appropriate tolerance.

Another great advantage of the present invention is that a touch operation that starts within an area surrounded by the perimeter boundary of the menu but does not exit the menu across the perimeter boundary can still be interpreted as an existing menu selection. In other words, it corresponds to the operation of "tapping" the menu selection sub-area. This allows the user to adopt the improved menu selection method of the present invention in systems that have not obtained any guidance or information regarding this new menu selection method. Users who do not have information about the new selection method can use the existing "knock" menu selection method without any problems until they know how to select a new menu or get information. In addition, a menu choice with a low chance of being prospective can be displayed as a menu item "closed" that does not include the outer perimeter boundary segment of the menu on the menu selection border, thus improving the flexibility of menu design. This allows you to allocate it to other menu choices that have room for margin around the menu and have a higher chance of being prospective. For example, FIG. 3B shows a menu layout that can be implemented in the same set of choices as FIG. 3A. However, in FIG. 3B, the placement of the menu selection items at a single stage is much higher than that of FIG. 3A in the case of "selection 2" 322, and "selection 3" and "selection 4" You can see that the probability is much lower. In the example of FIG. 3B, the entire right side 332 of the menu is associated with "select 2" and "select 3" and "select 4" You should choose not to get out of touch (ie use the existing "knock" menu selection).

By combining the two submenus, you can effectively take advantage of this same design principle for a menu of many choices. Figure 5 shows menu 500 with eight selection sub-areas 501-508. The selection sub-areas 501 to 504 constitute the first sub-menu area 521, and the sub-areas 505 to 508 constitute the second sub-menu area 522. Selection sub-regions 501-504 are associated with submenu 521 peripheral boundary segments 511-514 and selection sub-regions 505-508 are associated with submenu 522 peripheral boundary segments 515-518. Submenu 521 Perimeter border segment 514 and submenu 522 Perimeter border segment 515 are not actually along the outer perimeter boundaries of the menu 500, but are the same segment (514/515) belonging to the interior of the menu 500. However, the selection operation for the menu selection items 504 and 505 is not a problem since the direction in which the line segments 515/515 cross is different from the submenu area where the initial touch is made. The menu selection 504 is clearly selected with the touching operation of performing initial contact in the submenu 521 area and segmenting the line segment 514/515 in the downward direction, performing an initial contact in the area of the submenu 522, (514/515), the menu selection 505 is clearly selected. In yet another implementation, when an initial touch is detected within a submenu area, the submenu containing the initial touch position is visually highlighted to move along a path exiting the submenu, Indicates that a set of items (contained within this submenu) can be selected.

The simplest way to associate a menu's perimeter border segment with a menu choice is to associate a perimeter border segment of the menu that pertains to the perimeter boundary of the menu choice itself (as described above). Other methods are also possible and are not considered to depart from the scope of the invention. For example, the area associated with each menu selection is represented by a background (or font) color that is clearly distinct from each other, and the relevant segments of the menu perimeter border are represented by corresponding (or the same) colors as their corresponding segments. For example, in FIG. 3C, the six menu choices 341 through 346 are shown as background colors of orange 341, yellow 342, green 343, blue 344, purple 345, and red 346, respectively. And the corresponding perimeter border segments are shown as orange (351), yellow (352), green (353), blue (354), purple (355), and red (356) segments. Alternatively, the six menu choices 341 to 346 display the background color of light orange, light yellow, light green, light blue, light purple, and light red, respectively (to make each menu item easier to distinguish by light background coloring) The corresponding perimeter border segments can be represented by sharp orange (351), yellow (352), green (353), blue (354), purple (355), and red (356) segments, respectively. By combining the two approaches, you can clearly distinguish the relevant menu selection sub-areas with the colors of the surrounding border segments using less color. In this way, each menu selection subregion can be represented by a conventional rectangular shape, as in the example shown in FIG. 3C, rather than a somewhat more complex polygon as in FIG. 3A. As an additional feature, you can use a pattern or shading instead of a color to separate the various menu sub-areas and associate them with their perimeter border segments.

In addition, menu structure 500, as shown in FIG. 6, uses a consistent color combination of four colors to simplify and intuitively perform the menu selection operations required for each case. For example, menu selection sub-areas 501 and 505 may be represented by a blue background, sub-areas 502 and 506 may be represented by a red background, sub-areas 503 and 507 may be represented by a yellow background, and sub- areas 504 and 508 may be represented by a green background. The user then only has to remember the following: blue tapping and bending down the background, tapping near the red background and dragging to the right, tapping near the yellow background and dragging to the left, tapping down on the green background and down. It's easy to remember this particular way, reminiscent of blue "towards the sky", green "towards the ground", "red on the right" and "yellow on the left" This modular approach allows us to construct a menu with any number of choices using the principles of the present invention.

Although the embodiments of the present invention have been shown and described with reference to the drawings, various implementations are possible without departing from the spirit and scope of the invention. Therefore, the scope of the methodology should be determined not by the specific aspects described above, but by the claims and equivalent legal interpretations contained in the appendix.

Claims (36)

1. A user interface system comprising at least one input system for a user to input and edit text and at least one text display system for presenting the text to the user, A method for inputting and editing text on an electronic device through a user interface includes: The input system detecting one or more input actions performed by the user; Processing the one or more input actions by the electronic device to determine one or more textual objects corresponding to the one or more input actions; Identifying one of the determined text entities by the electronic device; Outputting the identified text object to the text insertion system at a determined text insertion position in the text display system; The electronic device recording a point of the text insertion position in the text display system in which a text object to be next confirmed is to be output; Wherein the input system detects a user action performed to select one or more of the textual objects previously output to the text display system, the user action being an action used to change a point of the textual insertion position Detecting a user action, not a user action; The electronic device identifying one or more alternative text objects determined to correspond to one or more sensed input actions from the one or more selected text objects being selected; Replacing the selected one or more text objects in the text display system with one or more identified alternative text objects; And And returning the text insertion position in the text display system to a point recorded before detection of the user action. How to enter and edit text on electronic devices through the user interface. The method according to claim 1, Further comprising the step of presenting the identified one or more of the alternative text entities to the user who selects the text entities and replacing the selected one or more of the selected text entities How to enter and edit text on electronic devices through the user interface. The method according to claim 1, After the electronic device determines a point in the text display system associated with performing the user action after sensing a user action performed to select one or more of the previously output textual objects, Further comprising the step of the electronic device verifying the textual object previously output to the text display system, Characterized in that the identified text object is closest to the determined point How to enter and edit text on electronic devices through the user interface. The method according to claim 1, Further comprising the step of the electronic device automatically generating one or more spaces between the one or more output text entities How to enter and edit text on electronic devices through the user interface. 5. The method of claim 4, Determining that two or more text entities each belong to one or more of the two or more text entity classifications, wherein each of the two or more text entity classifications is associated with at least one of the text entities Different from said other classifications in a manner in which one or more spaces are automatically generated; Further comprising the step of automatically inserting or deleting one or more blanks before or after the text object replacing the text object belonging to the classification different from the classification of the text object How to enter and edit text on electronic devices through the user interface. 3. The method of claim 2, Wherein the user performs one or more other input operations before selecting one of the alternate text entities presented to replace the selected text entity, and wherein the return of the text insertion position is not performed by the electronic device, Wherein the selected text entity is selected as the text insertion position by the text display system such that the result of the other input operation modifies or replaces the selected text entity How to enter and edit text on electronic devices through the user interface. The method according to claim 1, Further comprising the step of the electronic device selecting the alternate text entity replacing the selected text entity, without further input operations being performed by the user How to enter and edit text on electronic devices through the user interface. One or more input devices for sensing one or more input actions performed by a user; An output device for displaying the generated text to a user; And A processor coupled to the input device and the output device, The processor comprising: A first component for sensing one or more input actions performed by a user; A second component for processing the one or more input operations to determine one or more text entities corresponding to the one or more input operations; A third component for identifying one of the determined text objects; A fourth component for outputting the identified text entity at a determined text insertion position of the output device; A fifth component for recording a point of the text insertion position at which the next identified text object is output; A sixth component for detecting a user action performed to select one or more of the textual objects previously output to the output device, wherein the user action is not an action used to change a point of the textual insertion location A sixth component for detecting a user action; A seventh component for identifying one or more alternative text entities determined to correspond to the one or more sensed input operations that determine the one or more selected text entities; An eighth component for replacing said selected one or more said textual objects of said output device with said identified one or more alternative textual objects; And And a ninth component for returning the text insertion position of the output device to a point recorded before the user operation is sensed Text input mechanism. The method according to claim 1, Further comprising graphically presenting at least two selection items for selection within one or more defined bounded regions of the text display system, wherein the user places, moves and activates or deactivates the control points of the text display system Can, Wherein the step of graphically presenting the two or more selection items comprises: The electronic device displaying a graphical presentation of two or more selections within the one or more boundary regions; The electronic device defining one or more distinct segments of one or more boundaries of the one or more boundary regions; Associating each of the one or more segments defined by the electronic device solely with a delimited item of the selection graphically presented in a boundary area including the segment associated as part of the boundary of the boundary area; Wherein the electronic device senses activation of the control point within one of the boundary regions, wherein the activated control point then moves and the control point leaves the boundary region of the activation step Activation detection step; Identifying, by the electronic device, one of the distinct boundary segments that goes when the activated control point leaves the boundary area; Characterized in that the electronic device determines the selection by the user of the graphically presented selection item associated solely with the identified border segment How to enter and edit text on electronic devices through the user interface. 10. The method of claim 9, Each of said selection items is presented in a separately bounded sub-area within said boundary area, said defined boundary segment defined as a boundary segment coinciding with one of said sub-areas, and said sub- Characterized in that the choice presented in the region is associated exclusively with the matching border segment How to enter and edit text on electronic devices through the user interface. 10. The method of claim 9, Each said selection item being presented in such a way that said selection item is associated with a color, each defined defined boundary segment being associated with a color, and a selection associated with a given color being associated with a border segment Characterized by being associated with How to enter and edit text on electronic devices through the user interface. 10. The method of claim 9, Characterized in that the electronic device senses the activation of an external control point of any of the boundary regions so that the activated control point then moves into the one of the border regions in the activated state, Sensing activation; The electronic device identifying one of the delimited boundary segments as the activated control point enters the boundary region; And Further comprising the step of the electronic device determining the selection by the user of the selection of the graphically presented selection associated solely with the identified boundary segment How to enter and edit text on electronic devices through the user interface. 9. The method of claim 8, Two or more selection items for selection are graphically presented to the output device, The input device senses one or more input actions performed by a user to place, move and activate or deactivate a control point in the output device, The processor comprising: A tenth component for displaying a graphical presentation of the two or more choices within one or more defined boundary regions of the output device; An eleventh component for defining one or more distinct segments of one or more boundaries of one or more of the boundary regions; A twelfth component for independently associating each of the one or more segments defined with a delimited item of the selection graphically presented within a boundary region including a segment associated with the boundary region as part of the boundary region; A thirteenth component for detecting activation of the control point within one of the border regions, wherein the subsequently activated control point moves and the control point leaves the bordered area of the activated state; A fourth component for identifying one of said delimited boundary segments that passes when said activated control point leaves said boundary area; And And a fifteenth component for determining a selection by the user of the selection item graphically presented solely associated with the identified border segment Text input mechanism. delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete

Images