CN114047872A - Text input method and system - Google Patents
Text input method and system Download PDFInfo
- Publication number
- CN114047872A CN114047872A CN202111183508.6A CN202111183508A CN114047872A CN 114047872 A CN114047872 A CN 114047872A CN 202111183508 A CN202111183508 A CN 202111183508A CN 114047872 A CN114047872 A CN 114047872A
- Authority
- CN
- China
- Prior art keywords
- input
- user
- gesture
- signal
- text input
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 24
- 238000001514 detection method Methods 0.000 claims abstract description 17
- 230000015654 memory Effects 0.000 claims abstract description 16
- 230000004886 head movement Effects 0.000 claims description 10
- 230000004424 eye movement Effects 0.000 claims description 8
- 238000004891 communication Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 6
- 230000006399 behavior Effects 0.000 claims description 4
- 230000007613 environmental effect Effects 0.000 claims description 3
- 230000003183 myoelectrical effect Effects 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 claims description 2
- 210000003811 finger Anatomy 0.000 description 14
- 238000010586 diagram Methods 0.000 description 8
- 230000009471 action Effects 0.000 description 5
- 230000003190 augmentative effect Effects 0.000 description 4
- 210000004247 hand Anatomy 0.000 description 4
- 210000003813 thumb Anatomy 0.000 description 4
- 230000006870 function Effects 0.000 description 2
- 230000033001 locomotion Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 210000001145 finger joint Anatomy 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012905 input function Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 201000003152 motion sickness Diseases 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/04812—Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
Abstract
The invention discloses a text input method and a text input system, which can combine gestures and an annular character input interface, reduce the input memory burden of a user, improve the use acceptance of the user in public places, and have high efficiency and accuracy in input. The specific scheme comprises the following steps: the start signal and end signal detection device detects the receiving trigger signal in real time and sends the receiving trigger signal to the computing device. The user gesture detection device detects and receives input gestures of a recognized user and sends the input gestures to the computing device. The display equipment is used for displaying the CGR environment and comprises a virtual character interface, a text input interface and a virtual environment; the computing equipment is provided with the following modules: the user gesture recognizer is used for recognizing input gestures; the starting signal recognizer and the ending signal recognizer are used for recognizing the trigger signal; and the text input controller predicts the alternative characters, the input characters and the candidate word list input by the user through a prediction algorithm according to the recognition result of the input gesture, and controls a text input interface and a virtual character interface to be displayed in the display.
Description
Technical Field
The invention relates to the technical field of Computer Generated Reality (CGR), in particular to a text input method and a text input system.
Background
At present, in a computer generated display CGR environment, the problem of how to help a user to input a text efficiently and quickly under the conditions of not increasing the memory burden of the user and improving the willingness of the user acceptance degree is not solved.
The patent: CN109126125A discloses a text input method based on a double-rocker controller in a virtual reality environment, which comprises the following steps: positioning a character input position in a virtual reality environment, and popping up a virtual keyboard; rocking the first rocker to locate the area of the target character, and rocking the second rocker to select the target character; and judging whether the holding time of the position of the second rocker is greater than a set threshold, and if so, judging that the target character is continuously input.
This input mode must rely on a dual rocker controller. In a mobile mixed reality environment, carrying of peripheral equipment causes inconvenience in use; secondly, this approach requires the use of both hands to complete the input, and the user cannot use it with one hand. The mode is more obtrusive when used in public places, and the use will of users is reduced.
The patent: CN 102567631a, discloses a method for non-resident text input for mobile virtual reality head-mounted display, comprising: positioning a character input position in a virtual reality environment, and popping up a virtual keyboard; controlling the virtual cursor to move to the area of the target character by using the head rotation; searching a recommended word which is most similar to the input character in a pre-stored dictionary according to the input character, and dynamically displaying the recommended word outside a currently input character area; if the target word exists, the head is used for moving to control the virtual cursor to enter the area where the target word is located to select the word, otherwise, the character is continuously input.
The patent uses head movement to input text, and needs to rotate the head continuously to input the text, so that motion sickness is easy to generate for a user. The user needs to select characters on a virtual keyboard in the virtual environment through head movement, which causes a problem of occlusion in the virtual environment. The head-moving interaction is obtrusive when used in public places, and the use will of users is reduced.
Disclosure of Invention
In view of this, the present invention provides a text input method and system, which can combine gestures and a circular/linear single-line character input interface, reduce the input memory burden of a user, improve the acceptance of the user in public places, improve the input speed, and reduce the input error rate.
In order to achieve the purpose, the technical scheme of the invention is as follows: a text input system includes a computing device, a user gesture detection device, a start signal and end signal detection device, a display device.
The start signal and end signal detection device is used for detecting the receiving trigger signal in real time and sending the receiving trigger signal to the computing device.
And the user gesture detection equipment is used for detecting, receiving and identifying the input gesture of the user and sending the input gesture into the computing equipment.
The display equipment is used for displaying the CGR environment and comprises a virtual character interface, a text input interface and a virtual environment; also for displaying a user part or whole body avatar; the text input interface includes a list of candidate words and text that the user has entered.
The computing device is a computing device with a processor and a memory, and the computing device is provided with the following modules: a user gesture recognizer, a start signal and end signal recognizer, an application controller, and a text input controller.
The user gesture recognizer is used for recognizing input gestures, and the input gestures comprise the following three types: a first gesture, a second gesture, and a third gesture; the first gesture is used for pre-selecting a certain character or a plurality of characters in the character interface as alternative characters; the second gesture is used for confirming and selecting the alternative characters by the user and is used as the input characters; a third gesture is also included for selecting a word from the list of candidate words as the input word.
The start signal and end signal recognizer is used for recognizing a trigger signal, and the trigger signal comprises a text input start signal and a text input end signal; the text input start signal is used to trigger the start of the text input flow, and the text input end signal is used to trigger the end of the text input flow.
And the text input controller predicts the alternative characters, the input characters and the candidate word list input by the user through a prediction algorithm according to the recognition result of the input gesture, and controls a text input interface and a virtual character interface to be displayed in the display.
The application program controller is used for controlling the content related to the application program, and comprises different text input scenes corresponding to different application programs.
Further, the start signal and the end signal identifying device may adopt one of the following devices: an image sensor, a pressure sensor, a touch sensitive device, a microphone, a position sensor, or an orientation sensor. The start signal and end signal recognition device serves as a text input start signal or a text input end signal according to the detected environmental information or user behavior signal.
Further, the user gesture recognition device performs gesture recognition by adopting one of the following devices: an image sensor, a pressure sensor, a touch sensitive device, or a myoelectric detection device.
Further, the text input controller comprises a word predictor, a text input interface controller and a virtual character interface controller; the word predictor predicts input words according to input characters selected by a user at present, predicts the input words by adopting a character position probability model and/or a language model and generating words with the maximum probability in a preset number to form a candidate word list.
The position probability prediction model predicts the alternative character input by the user and the input character through the current position of the character.
The language model judges the possible probability of the input words according to the input content of the user to obtain a candidate word list.
The text input interface controller controls the display device to display a virtual character interface, wherein the virtual character interface comprises an annular display total interface of all characters, an annular display interface of alternative characters and a display interface of input characters.
The text input interface controller controls the display device to display a text input interface including a candidate word list display interface and an input word display interface.
Further, the user gesture detection device, the start signal and end signal detection device and the display device are connected through a communication network, including a wired communication or wireless communication mode, and each device comprises a data sending and receiving component.
Further, a character position probability model and/or a language model are used for prediction, input words are predicted through the character position probability model and/or the language model, words with the maximum probability in the preset number are generated to form a candidate word list, and the method specifically comprises the following steps:
character position probability model determines the position probability P of a character based on an input character selected by a userposition(ii) a The language model is divided into two parts, one part is a word internal prediction model, characters which are possibly input by a user at present are judged according to the probability of the occurrence of words in a word bank and the characters which are already input by the user, and the probability of the characters input at present is Pletter(ii) a The other part of the language model is a prediction model in the sentence, and the probability P of the most probable word input at present is judged according to the probability of the word in the sentence in the language library and the characters input at presentword(ii) a The word stock and the language stock use the existing word stock and language stock at present or carry out dynamic adjustment according to user input; the probability of the input word is P ═ apposition+βPletter+γPword(ii) a Where α is a predetermined character position probability PpositionBeta is a preset character probability PletterGamma is the outline of a predetermined wordRate PwordThe weight value of (2).
Another embodiment of the present invention further provides a text input method, which uses the text input system to input text, and includes the following steps:
s1, a text input start signal is detected and the following text input flow is started.
And S2, displaying a ring-shaped display total interface of all characters by the display device.
S3, detecting a first gesture of the user, wherein the first gesture is used for selecting one or more characters in the input virtual character layout as alternative characters.
And S4, displaying the annular display interface of the alternative characters by the display device.
And S5, detecting a second gesture of the user, wherein the second gesture confirms and selects one or more alternative characters as input characters.
And S6, displaying a display interface of the input characters by the display device.
And S7, performing candidate word prediction according to the current input characters to generate a candidate word list.
And S8, displaying a candidate word list display interface by the display equipment.
S9, detecting a third gesture or other input signal of the user, and selecting a word from the candidate word list as an input word.
And S10, displaying an input word display interface by the display device.
And S11, judging whether a text input ending signal is detected, if so, ending the text input process, and otherwise, returning to S3.
Further, a voice signal, a gesture signal, an eye movement signal, a head movement signal, or a position signal representing the start of text input is set in advance as the text input start signal.
A voice signal, a gesture signal, an eye movement signal, a head movement signal or a position signal for representing the end of text input is preset as a text input end signal.
Further, the annular display interfaces of all characters, the annular display interfaces of the alternative characters, specifically include: the single or multiple characters are arranged in a single row in a circular or oval shape, and all the character rows form a complete ring or a partial ring which is displayed completely or partially when displayed.
Has the advantages that:
1. according to the annular display text input system provided by the embodiment of the invention, the gesture and the annular/linear single-line character input interface are combined, so that the input and memory burden of a user is reduced; the text input speed is improved by small gesture actions and simple sliding gestures; based on the micro-gestures of sliding of the thumb and the index finger and the combination of the single-line character input interface, the hands of the user can be placed at any position and even in a pocket of clothes for operation, so that the use privacy of public places is improved, the obtrusiveness is reduced, the fatigue of the hands used for a long time is reduced, and the use acceptance of the user in the public places is improved; the input text can be predicted by using a non-precise input mode and combining a position probability model and a language model, so that the input speed is increased, and the input error rate is reduced.
2. According to the text input method of the annular display, provided by the embodiment of the invention, at present, text input can be performed through user gestures in a CGR environment, and the method can be used in a mobile scene. In the mode, the specific gesture of the user is coupled with the specific character, the user needs to memorize the specific character, and the memory burden of the user is increased; secondly, the switching speed of the user between gestures can be reduced by large hand motions, so that the text input speed is reduced; the larger hand motion is obtrusive when used in public places, privacy disclosure can be caused by the coupling between the gestures and the characters, and the use will of users is reduced.
In the invention, a user inputs text by combining virtual character layout by using micro gestures of a thumb and an index finger of a hand. The micro gestures are combined with the virtual character layout, the gesture input can be completed through simple relative sliding micro gestures, no specific relation exists between the micro gestures and the characters, and a user does not need to memorize the characters; the user can quickly finish micro-gesture actions, and the text input speed is improved; the micro-gesture has small action amplitude, so that the hand of the user can finish text input at any position, for example, the user finishes input in a pocket, the obtrusiveness of a public place is reduced, meanwhile, a spectator cannot see the virtual environment where the user is located, privacy disclosure is avoided by simply repeating gestures, and the use intention of the user in the public place is improved; the hands of the user can be placed at will, the micro-gesture action amplitude is small, and the hand fatigue of the user during long-time text input can be avoided.
3. In the text input system provided by the invention, a prediction algorithm is used, and the input text is predicted by combining the specific characters of the current virtual character layout at the hand position of the user and the input characters through a character position probability and/or language probability model. The user can realize fast and accurate text input through inaccurate text input.
Drawings
FIG. 1 is a schematic diagram of a text input system according to an embodiment of the present invention;
FIG. 2 is an example of a ring input interface provided by embodiments of the present invention;
FIG. 3 is a diagram of a ring input interface displayed on the back of a hand according to an embodiment of the present invention;
FIG. 4 is a diagram of a multi-character annular input interface provided by an embodiment of the present invention;
FIG. 5 is a diagram of an input interface with a single character presented on the back of the hand according to an embodiment of the present invention;
FIG. 6 is an exemplary diagram of a first gesture with a ring interface at the back of the hand, provided by embodiments of the present invention;
FIG. 7 is an exemplary illustration of a ring interface at an index finger according to an embodiment of the present invention;
FIG. 8 is an exemplary diagram of a second gesture provided by embodiments of the present invention;
FIG. 9 is a diagram of another example of a second gesture provided by embodiments of the present invention;
FIG. 10 is an exemplary diagram of a text input provided by an embodiment of the present invention;
FIG. 11 is a flow chart of text entry according to an embodiment of the present invention.
Detailed Description
The invention is described in detail below by way of example with reference to the accompanying drawings.
The invention provides a text input method and system based on finger micro-gestures, which can be used in a Computer Generated Reality (CGR) environment, particularly a mobile CGR environment, and can improve the text input efficiency and convenience of a user and the use desire of the user in public places.
The text input mode provided by the invention can be used for text input in a Computer Generated Reality (CGR) environment.
A Computer Generated Reality (CGR) environment refers to a fully or partially simulated environment that is perceived by a user through an electronic system. Users can interact with it; the CGR environment may adjust one or more characteristics of one or more virtual objects in the environment based on the user's behavior; for example, the CGR system detects a swipe gesture by the user, and in response, the CGR environment adjusts the image content presented. The user may perceive and interact with the virtual object in the CGR through any sense, including visual, auditory, tactile, olfactory, and the like; for example, a user may perceive and may interact with audio objects in a CGR environment.
Virtual Reality (VR) environments and Mixed Reality (MR) environments are special CGR environments. Mixed reality environments include Augmented Reality (AR) environments and Augmented Virtual (AV) environments. Wherein the CGR environment comprises at least a part of a Virtual Environment (VE). A virtual reality environment refers to a simulated environment that is entirely computer-generated. A mixed reality environment refers to a simulated environment that, in addition to a computer-generated simulated environment, contains one or more inputs from a physical environment or representations thereof; a mixed reality environment refers to conditions from between a purely physical environment and a virtual reality environment, but does not encompass both conditions. Augmented reality environment refers to a simulated environment in which a representation of a physical environment is partially converted by computer-generated information. An augmented virtual environment refers to a simulated environment of a virtual or computationally generated environment in combination with inputs from a physical environment.
A variety of different types of devices may be used to display the CGR environment, enabling a user to perceive and interact with the CGR environment, including but not limited to head mounted displays, smart phones, smart watches, tablets, heads up displays, glasses with display functionality, and the like.
FIG. 1 shows an exemplary system for accomplishing this type of text entry. The system comprises a computing device, a user gesture recognition device, a start signal and end signal recognition device and a display device.
The start signal and end signal recognition devices are used to detect a particular signal or signals for triggering the system to enable a user to begin entering text and end text entry. The device may include, but is not limited to, an image sensor, a pressure sensor, a touch sensitive device, a microphone, a position sensor, an orientation sensor, etc., or other devices that can detect signals such as environmental information or user behavior, etc., as a start signal or an end signal. For example, the touch sensitive device may detect a tap or swipe input as a start signal.
And the user gesture recognition equipment is used for recognizing the input gesture of the user. The input gesture includes at least a first gesture and a second gesture. The first gesture is used for pre-selecting a certain character or a plurality of characters in the character interface; the second gesture is used for confirming and selecting characters by the user; in some cases, a third gesture may be included for selection of a word from the list of candidate words. The gesture recognition device may use a variety of devices capable of recording corresponding data of different gestures of a user to complete gesture recognition by using a computing device, including but not limited to an image sensor, a pressure sensor, a touch-sensitive device, and a myoelectric detection device.
The display device is used for displaying CGR environments including virtual character interfaces, text input interfaces, and other virtual environments. In some cases, a user partial or full body avatar may be displayed.
The text input interface includes a list of candidate words and text that the user has entered.
The computing device is a computing device having a processor and a memory, and includes a user gesture recognizer, a start signal and end signal recognizer, a text input controller, and an application controller.
The user gesture recognizer is used for processing the data acquired by the gesture recognition device and detecting whether the data is a specific gesture. The specific gesture may be a program preset or user defined.
The start signal and end signal recognizer is used for processing the data acquired by the start signal and end signal recognition device and detecting whether the data is a signal for starting or ending text input. The signal may be pre-programmed or user-defined.
The text input controller predicts candidate words for user text input through a prediction algorithm and displays the candidate words on an annular linear character interface or text input.
The word predictor predicts the input word according to the characters selected by the user at present, and a position probability model and/or a language model can be used. And performing input word prediction through a character position probability model and/or a language model, and generating the first three or five or other words with the highest probability to form a candidate word list.
The position probability prediction model predicts the character most probably selected by the user according to the current position of the character. For example, characters that face the user have the highest probability of being entered, and characters that deviate from this direction have a lower probability of being selected.
The language model judges the possible probability of the currently input characters and words according to the characters already input by the user.
The application program controller is used for controlling the content related to the application program, and comprises different text input scenes corresponding to different application programs and the like.
The computing devices may be integral or separate. The user gesture recognition device, the start signal and end signal recognition device and the display device are connected through a communication network, and the devices can be integrated or independent in a wired/wireless mode. Each device contains data transmission and reception components. The processor may be a local processor or a cloud processor. The memory can be various types of local memories including a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk and the like, or a cloud memory. The processor can read information from, and write information to, the storage medium. For example, the computing device, the real device, and the start signal and end signal recognition device of the virtual reality head-mounted display are integrated into the head-mounted display.
The present invention proposes a method for text input in a Computer Generated Reality (CGR) environment. The method comprises the following steps: the start signal triggers text input; displaying a virtual character layout; the user continuously uses the first gesture to select and input one or more characters in the virtual character layout as alternative characters; the user uses the second gesture to make confirmation selection on one or more characters; the system predicts the input text according to the current character; the user completes the input of a word by using a third gesture or other input signals, such as voice; the end signal ends the text input.
The start signal triggers text entry. In the application program, the system judges whether the user wants to input text at this time by detecting the start signal. When the system detects a start signal, the user can use gestures for text input at this time; when the system does not detect a start signal, the user will not be able to perform the text entry function using gestures. The start signal may include, but is not limited to, voice, gesture, eye movement signal, head movement signal, position signal, and may be used as the start signal to trigger the user text input operation. For example, using the speech "start typing" as the start signal, when the application runs, the user says "start typing" which the system detects as the start signal; when the user viewpoint stays for a certain time at the position of the application program where the text can be input, the system detects the user viewpoint as a starting signal; when the user makes an application predefined gesture, the system detects it as a start signal; detecting a hand or a body of a user as a start signal when the hand or the body is located at a specific position in the virtual environment; clicking on a virtual object in the virtual environment by the user can be used as a signal for text input. The start signal triggers the text entry function of the application and the user can begin text entry.
The end signal ends the text input. In the application program, the system judges whether the user has finished text input at this time by detecting a start signal to end the current text input. The ending signal may include, but is not limited to, speech, gestures, eye movement signals, head movement signals, position signals. For example, the user uses "end input" speech as a signal to end the current text input, and when the system detects the speech, the user has finished inputting, and the gesture action will trigger the system input function; detecting a user's hand or body as an end signal when the user's hand or body leaves a particular location in the virtual environment; when the user completes the entry of a sentence, clicking the "send" button in the virtual environment may serve as an end signal.
The virtual character layout can be set to be always displayed, at the moment, the default system is always in a state of inputting texts, and the texts can be input by directly using the first gesture. The virtual character layout is configured to be displayed after the start signal is detected and the system can proceed with text entry. For example, in a head-mounted display, the virtual character layout is located in a three-dimensional virtual environment at the user's back of the hand, palm, fingers, etc.; in a smart watch, a virtual character layout is displayed on a portion of a display screen of the smart watch.
The virtual character interface is a ring-shaped character interface, and the single or a plurality of characters are arranged in a single row in a circular or oval shape or other similar shapes. All character arrangements may form a complete ring or a partial ring, which may be fully or partially displayed when displayed.
An example of a circular input interface is shown in fig. 2, where a circular character interface forms a ring for a single character, but the character is displayed as an incomplete display, or where a circular interface may be displayed, or not, and only the character is displayed. As shown in FIG. 2, the annular character interface may be displayed at the second section of the user's index finger. In addition, the annular character interface can be changed in size and is positioned at other finger joints of the user or at the arm, the back of the hand, the palm of the hand and the like.
FIG. 3 illustrates the manner in which the ring input interface is displayed on the back of the hand.
FIG. 4 is an example of a multi-character presentation.
FIG. 5 is another example of a single character presentation at the back of the hand.
Specifically, when the user is occupied with one hand, the virtual character interface is displayed at the fingers, and text input can be performed using micro-gestures between the thumb and index finger or other fingers. When the virtual character interface is positioned at the positions of the arm, the back of the hand, the palm and the like of the user, the other hand is used for carrying out first gesture operation, and text input is completed.
The user selects one or more characters in the single line character interface using a first gesture. The first gesture is a relative sliding gesture of a user's finger along a circular direction at and around the circular input interface. Fig. 6 is an example of a first gesture of a ring interface at the back of a hand, where a user's index finger slides up and down relative to the back of the hand, and meanwhile, the character interface is updated according to the gesture, and the sliding speed and distance of the character interface may be consistent with the moving speed and distance of the gesture, or may be set separately, and keep a certain proportional relationship with the moving speed and distance of the gesture. The user may continuously update the character interface with the first gesture to pre-select different characters or groups of characters.
FIG. 7 is an example of a ring interface at the index finger, with the sliding of the user's thumb relative to the index finger for preselection of characters.
The user confirms the preselected character or group of characters using the second gesture. The second gesture is a relative sliding gesture of the user's finger at the annular interface with respect to a vertical annular direction at and around the annular input interface, as shown in fig. 8, 9, or other gestures such as a click, double click, etc. The second gesture is distinct from the first gesture.
The user selects from the predicted word list using a third gesture or other signal to complete the entry of a word. The third gesture is distinct from the first gesture. The third gesture may be identical to or distinct from the second gesture. When the third gesture is consistent with the second gesture, the second gesture and the third gesture have a time sequence; the system judges whether the second gesture is finished, if so, the current gesture is a third gesture; if the second gesture is not complete, the current gesture is a third gesture. The third gesture/signal may comprise one or more gestures/signals that may be executed sequentially/in a loop to select a particular word in the word list.
The character position probability model and/or the language probability model make predictions of the input text. Character position probability model determines probability P of each character occurrence using state judgment of character input interfaceposition. The language probability model is divided into two parts and can be calculated by using an N-gram language model; one part is a word internal prediction model, and characters which are possibly input by the user at present are judged according to the probability of the occurrence of the word in the word stock and the characters which are already input by the user, wherein the probability of the characters input at present is Pletter(ii) a (language probability model) a part is a prediction model in a sentence, and the probability P of the most probable word input at present is judged according to the probability of the word in the sentence in a language library and the characters input at presentword. The word stock and the language stock can use the existing word stock and language stock and can be dynamically adjusted according to the input of the user. By the formula: p ═ α Pposition+βPletter+γPwordThe probabilities of different words formed by the characters that have been input at present are obtained, and a list of predicted input words is obtained. Where α, β, γ are weight values, which may be altered in a computing device application. The specific set values are as follows: alpha is larger and is 0.5; beta is 0.2 and gamma is 0.3.
The character position probability can be built in the system in advance, and can also be adjusted to a certain extent by a user.
Each input by the user can be an accurate input or a fuzzy input. In the accurate input, the character or the character group input each time is the only character or the character group, namely the character position probability is 1 or 0; in fuzzy input, the character or character group to be input itself and the surrounding characters or character groups may be input, with a position probability between [0,1 ]. As in fig. 10A, the system is set such that the character in the user direction is the character to be selected, in the precise selection, "t" is the character to be input, and the probability of other characters being input is 0; in fuzzy selection, "t" and its surrounding characters, such as "s" and "u", are all entered with a certain probability, but "t" has the highest probability, while other characters have smaller, different probabilities depending on the degree of deviation from the user's direction.
10A-10D illustrate an example of text input by a user selecting a sequence of characters by a first gesture and a second gesture in succession. In fig. 10A, the user starts inputting, and there is no character already input at this time, and the position probability of the character "t" is the largest, and then "s", "u", then "r", "v", and combines with the language model to obtain the word candidate list at this time. When the user has finished selecting the character "t", the user continues to input the character, and updates the predicted word candidate list in combination with the language model. In 10C, according to P ═ α Pposition+βPletter+γPwordThe calculation is performed with the most probable word being "text" followed by "test", "term" …, at which point the user may choose to perform a third gesture to select a word from the candidate list, or may proceed with the input, e.g., 10D, at which point "term" will no longer be the candidate.
FIG. 11 is an example of a text entry method, including the steps of:
s1, a text input start signal is detected and the following text input flow is started.
A voice signal, a gesture signal, an eye movement signal, a head movement signal or a position signal representing the start of text input is preset as a text input start signal.
And S2, displaying a ring-shaped display total interface of all characters by the display device.
S3, detecting a first gesture of the user, wherein the first gesture is used for selecting one or more characters in the input virtual character layout as alternative characters.
And S4, displaying the annular display interface of the alternative characters by the display device.
And S5, detecting a second gesture of the user, wherein the second gesture confirms and selects one or more alternative characters as input characters.
And S6, displaying a display interface of the input characters by the display device.
And S7, performing candidate word prediction according to the current input characters to generate a candidate word list.
And S8, displaying a candidate word list display interface by the display equipment.
S9, detecting a third gesture or other input signal of the user, and selecting a word from the candidate word list as an input word.
S10, displaying an input word display interface by the display equipment;
and S11, judging whether a text input ending signal is detected, if so, ending the text input process, and otherwise, returning to S3. A voice signal, a gesture signal, an eye movement signal, a head movement signal or a position signal for representing the end of text input is preset as a text input end signal.
The annular display total interface of all characters and the annular display interface of the alternative characters are specifically as follows: the single or multiple characters are arranged in a single row in a circular or oval shape, and all the character rows form a complete ring or a partial ring which is displayed completely or partially when displayed.
In summary, the above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (9)
1. A text input system comprising a computing device, a user gesture detection device, a start signal and end signal detection device, a display device;
the starting signal and ending signal detection equipment is used for detecting and receiving a trigger signal in real time and sending the trigger signal to the computing equipment;
the user gesture detection device is used for detecting, receiving and identifying input gestures of a user and sending the input gestures into the computing device;
the display equipment is used for displaying the CGR environment and comprises a virtual character interface, a text input interface and a virtual environment; also for displaying a user part or whole body avatar; the text input interface comprises a candidate word list and a text input by a user;
the computing device is provided with a processor and a memory, and the computing device is provided with the following modules: a user gesture recognizer, a start signal recognizer, an end signal recognizer, an application program controller and a text input controller;
the user gesture recognizer is used for recognizing the input gesture, and the input gesture comprises the following three types: a first gesture, a second gesture, and a third gesture; the first gesture is used for pre-selecting a certain character or a plurality of characters in the character interface as alternative characters; the second gesture is used for confirming and selecting the alternative characters by the user and is used as the input characters; the gesture recognition device also comprises a third gesture used for selecting words from the candidate word list as input words;
the start signal and end signal recognizer is used for recognizing the trigger signal, and the trigger signal comprises a text input start signal and a text input end signal; the text input starting signal is used for triggering the start of a text input process, and the text input ending signal is used for triggering the end of the text input process;
the text input controller predicts alternative characters, input characters and a candidate word list input by a user through a prediction algorithm according to the recognition result of the input gesture, and controls a text input interface and a virtual character interface to be displayed in the display;
the application program controller is used for controlling contents related to the application programs, and comprises different text input scenes corresponding to different application programs.
2. The system of claim 1, wherein the start signal and end signal identifying devices employ one of:
an image sensor, a pressure sensor, a touch sensitive device, a microphone, a position sensor, or an orientation sensor;
the start signal and end signal recognition device serves as a text input start signal or a text input end signal according to the detected environmental information or user behavior signal.
3. The system of claim 1 or 2, wherein the user gesture recognition device performs gesture recognition using one of the following devices:
an image sensor, a pressure sensor, a touch sensitive device, or a myoelectric detection device.
4. The system of claim 1, wherein the text input controller comprises a word predictor, a text input interface controller, and a virtual character interface controller;
the word predictor predicts input words according to input characters selected by a current user, predicts the input words by adopting a character position probability model and/or a language model and generating words with the maximum probability in a preset number to form a candidate word list;
the position probability prediction model predicts alternative characters and input characters input by a user according to the current positions of the characters;
the language model judges the possible probability of the input words according to the input content of the user to obtain a candidate word list;
the text input interface controller controls the display equipment to display a virtual character interface, wherein the virtual character interface comprises an annular display total interface of all characters, an annular display interface of alternative characters and a display interface of input characters;
the text input interface controller controls the display device to display a text input interface, which includes a candidate word list display interface and an input word display interface.
5. The system of claim 1, 2 or 4, wherein the user gesture detection device, the start signal and end signal detection device and the display device are connected via a communication network, including wired or wireless communication, each device including a data transmission and reception component.
6. The system according to claim 4, wherein the prediction uses a character position probability model and/or a language model, the character position probability model and/or the language model is used for predicting the input words, and a candidate word list is formed by generating words with the highest probability in a preset number, specifically:
the character position probability model determines the position probability P of the character according to the input character selected by the userposition(ii) a The language model is divided into two parts, one part is a word internal prediction model, characters which are possibly input by a user at present are judged according to the probability of the occurrence of words in a word bank and the characters which are already input by the user, and the probability of the characters input at present is Pletter(ii) a The other part of the language model is a prediction model in the sentence, and the probability P of the most probable word input at present is judged according to the probability of the word in the sentence in the language library and the characters input at presentword(ii) a The word stock and the language stock use the existing word stock and language stock at present or carry out dynamic adjustment according to user input; the probability of the input word is P ═ apposition+βPletter+γPword(ii) a Where α is a predetermined character position probability PpositionBeta is a preset character probability PletterGamma is the probability P of a predetermined wordwordThe weight value of (2).
7. A text input method, wherein the text input system according to any one of claims 1 to 7 is used for text input, comprising the steps of:
s1, detecting and receiving a text input starting signal, and starting the following text input process;
s2, displaying a ring-shaped display total interface of all characters by the display equipment;
s3, detecting a first gesture of a user, wherein the first gesture is used for selecting one or more characters in the input virtual character layout as alternative characters;
s4, displaying an annular display interface of the alternative characters by the display device;
s5, detecting a second gesture of the user, wherein the second gesture confirms and selects one or more alternative characters as input characters;
s6, displaying a display interface of the input characters by the display equipment;
s7, performing candidate word prediction according to the current input characters to generate a candidate word list;
s8, displaying a candidate word list display interface by the display equipment;
s9, detecting a third gesture or other input signals of the user, and selecting a word from the candidate word list as an input word;
s10, displaying an input word display interface by the display equipment;
and S11, judging whether a text input ending signal is detected, if so, ending the text input process, and otherwise, returning to S3.
8. The text input method according to claim 7, wherein a voice signal, a gesture signal, an eye movement signal, a head movement signal, or a position signal for representing the start of text input is preset as the text input start signal;
a voice signal, a gesture signal, an eye movement signal, a head movement signal or a position signal for representing the end of text input is preset as a text input end signal.
9. The text input method according to claim 7 or 8, wherein the annular display interfaces of the total interface and the alternative characters of the annular display of all the characters are specifically:
the single or multiple characters are arranged in a single row in a circular or oval shape, and all the character rows form a complete ring or a partial ring which is displayed completely or partially when displayed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111183508.6A CN114047872A (en) | 2021-10-11 | 2021-10-11 | Text input method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111183508.6A CN114047872A (en) | 2021-10-11 | 2021-10-11 | Text input method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114047872A true CN114047872A (en) | 2022-02-15 |
Family
ID=80205256
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111183508.6A Pending CN114047872A (en) | 2021-10-11 | 2021-10-11 | Text input method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114047872A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116069169A (en) * | 2023-03-29 | 2023-05-05 | 深圳市光速时代科技有限公司 | Data processing method and system for inputting virtual text based on intelligent watch |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104718544A (en) * | 2012-10-16 | 2015-06-17 | 谷歌公司 | Partial gesture text entry |
| CN111831112A (en) * | 2020-05-26 | 2020-10-27 | 北京理工大学 | Text input system and method based on eye movement and finger micro-gesture |
| US20210240331A1 (en) * | 2018-05-09 | 2021-08-05 | Apple Inc. | Selecting a text input field using eye gaze |
-
2021
- 2021-10-11 CN CN202111183508.6A patent/CN114047872A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104718544A (en) * | 2012-10-16 | 2015-06-17 | 谷歌公司 | Partial gesture text entry |
| US20210240331A1 (en) * | 2018-05-09 | 2021-08-05 | Apple Inc. | Selecting a text input field using eye gaze |
| CN111831112A (en) * | 2020-05-26 | 2020-10-27 | 北京理工大学 | Text input system and method based on eye movement and finger micro-gesture |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116069169A (en) * | 2023-03-29 | 2023-05-05 | 深圳市光速时代科技有限公司 | Data processing method and system for inputting virtual text based on intelligent watch |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6701066B2 (en) | Dynamic phrase expansion of language input | |
| JP6495376B2 (en) | Handwritten keyboard for screen | |
| US20180349346A1 (en) | Lattice-based techniques for providing spelling corrections | |
| JP6140668B2 (en) | Multi-modal text input system for use with mobile phone touchscreen etc. | |
| US20180322112A1 (en) | Unified framework for text conversion and prediction | |
| US9740399B2 (en) | Text entry using shapewriting on a touch-sensitive input panel | |
| US9547430B2 (en) | Provision of haptic feedback for localization and data input | |
| CN108700996B (en) | System and method for multiple input management | |
| US20180182376A1 (en) | Rank-reduced token representation for automatic speech recognition | |
| US20150261310A1 (en) | One-dimensional input system and method | |
| US20110063231A1 (en) | Method and Device for Data Input | |
| US20150220265A1 (en) | Information processing device, information processing method, and program | |
| US20230244379A1 (en) | Key function execution method and apparatus, device, and storage medium | |
| CN114047872A (en) | Text input method and system | |
| KR20130010252A (en) | Apparatus and method for resizing virtual keyboard | |
| US20240134516A1 (en) | System and method for coarse and fine selection keyboard user interfaces | |
| CN117931335A (en) | System and method for multimodal input and editing on a human-machine interface | |
| CN117931334A (en) | System and method for coarse and fine selection of a keyboard user interface | |
| CN113589958A (en) | Text input method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |